📊 Predictive Benchmark Runner

This directory contains the LLM-based evaluation runner. It consumes ground-truth group files (generated by the data-prep pipeline), prompts an LLM to rank solutions, and produces per-task and cross-task grading reports.

🧐 Functionality

• 📥 Loads ground-truth comparison groups (e.g., groups_<task>_n2.json).
• 📝 Constructs Prompts using task descriptions and data analysis reports.
• 🧠 Queries LLMs to predict the better solution (pairwise) or rank multiple solutions.
• 📈 Computes Metrics: Accuracy (for n=2), Spearman Correlation & Precision@k (for n>2).
• 💾 Saves Reports including full interaction logs and machine-readable JSONs.

Entrypoint: run_bench.py

🏃 Quick Start

python -m skip_bench.grade.run_bench \
  --task-file /path/to/task_name.txt \
  --solutions-dir /path/to/solutions_root \
  --n 2 \
  --model deepseek-v3.2-thinking \
  --temperature 1.0 \
  --parallel 64 \
  --task-parallel 8 \
  --prompt-boost \
  --desc-dir /path/to/description \
  --da-dir /path/to/data_analysis/result \
  --cot

📂 Expected Inputs

1. 📋 Task List (--task-file)

   • A text file with one task name per line.

2. 🗂️ Solutions Root (--solutions-dir)

   • Structure: <solutions_dir>/<task>/ground_truth/groups_<task>_n*.json

3. 📚 Task Resources (Optional but Recommended)

   • Mode 1 (Root): --tasks-root containing description/ and data_analysis/.
   • Mode 2 (Explicit): --desc-dir for descriptions and --da-dir for analysis.
   • Mode 3 (Raw): --raw-data-sample for raw data snippets.

4. 🎯 Ground Truth Override

   • --groundtruth-file: Point to a specific JSON file (supports {task} placeholder).

⚙️ Key Arguments

📤 Outputs

Artifacts are written to <solutions_dir>/<task>/report/:

• 📄 Human-Readable Report: grade_report_<task>_... .txt (Metrics + Logs)
• 🤖 Alignment JSON: alignment_<task>_... .json (Structured predictions)
• 💾 Partial Checkpoint: grade_results_partial.json
• 📊 Global Report: <solutions_dir>/report/grade_report_alltasks_*.txt

Notes

• Group files are created by the data-prep pipeline (see the README in prepare_bench_subset).
• If you pass --groundtruth-file, you must set --n to a non-zero value.
• When using relative paths in group_entry.paths, keep --solutions-dir set so the runner can resolve files correctly.
• The command examples above may not list all tunable flags. For a complete reference, run each script with --help or inspect the corresponding Python file and its argparse comments.