Benchmarks
June 21, 2026 · View on GitHub
Paper-scale, dataset-backed reproductions, grouped by the two LEVI entry points.
New to LEVI? Start with ../quickstart/ instead.
Install uv first (https://docs.astral.sh/uv/getting-started/installation/),
then uv sync (or uv sync --extra examples for all example dependencies).
code/ — evolve_code
The paper-scale circle packing plus the seven ADRS Leaderboard problems (LLM-guided algorithm discovery across systems-optimization tasks).
code/circle_packing/ packs n=26 circles in a unit
square — self-contained, no dataset. run.py defaults to OpenRouter + a local
Qwen at http://localhost:8000/v1; run_claude.py runs it via the Claude Code
CLI.
The ADRS problems share one entrypoint (uv run python run.py) and mostly need
the ADRS dataset:
git clone https://github.com/cmu-db/ADRS-Leaderboard.git
export ADRS_EXAMPLE_DATA_ROOT=/path/to/ADRS-Leaderboard
| Problem | Optimizes | Budget | ADRS data |
|---|---|---|---|
cant_be_late/ | Spot vs on-demand scheduling under deadlines | $4.50 | Yes |
cant_be_late_multi/ | …same, switching across regions | $5.00 | Yes |
cloudcast/ | Broadcast topology across AWS/Azure/GCP | $3.00 | Yes |
eplb/ | Place 64 MoE experts across 288 GPU slots | $4.50 | Yes |
llm_sql/ | Reorder CSV columns for prefix hit rate | $4.50 | Yes¹ |
prism/ | Assign ML models to GPUs to cut KV pressure | $4.50 | No |
txn_scheduling/ | Order 100 transactions to minimize makespan | $13.00 | No |
¹ run problems/llm_sql/download_datasets.sh in your ADRS-Leaderboard checkout.
Extra deps: uv sync --extra example-eplb (torch), uv sync --extra example-llm-sql (pandas).
Suggested picks: prism/ (no dataset, OpenRouter throughout), cloudcast/
(small budget), cant_be_late/ (exercises prompt_opt), cant_be_late_multi/
(richest: custom init, behavior features, punctuated equilibrium, prompt_opt).
prompts/ — evolve_prompts
Larger prompt-evolution setups, close in shape to GEPA's harness. These pull HuggingFace datasets and assume an OpenRouter key for the proposer/judge models.
prompts/hotpotqa/— multi-hop QA (HFhotpot_qa).prompts/hover/— multi-hop fact verification with BM25 retrieval.prompts/ifbench/— instruction-following benchmark.prompts/pupa/— privacy-preserving delegation (PAPILLON).
Model configuration
Examples use LiteLLM model ids. The published configs split paradigm models (creative exploration) from mutation models (cheap iteration):
- Paradigm:
openrouter/google/gemini-3-flash-preview - Mutation:
openrouter/xiaomi/mimo-v2-flash, local Qwen 30B (assumed athttp://localhost:8000/v1)
export OPENROUTER_API_KEY=your-key-here
export OPENAI_API_KEY=your-key-here # if using OpenAI models