ExtractBench

1. The model or API needs to be publicly accessible, either via open weights or a self-serve API that any user can sign up for.
2. The benchmark run needs to finish within a reasonable time (roughly single-digit hours).
3. We can adjust concurrency based on the provider's recommended settings, but providers should not require custom framework changes, so the evaluation stays fair across models.

Costs use each provider's official listed price as of July 1, 2026.

These figures apply the reported mean cost per page to 4,869 pages. Per-system, per-split prices are in leaderboard.csv (Cost_Per_Page, Cost_Short, Cost_Medium, Cost_Long, in dollars per page).

The four self-hosted pipelines need an endpoint you run yourself; see .env.example. Other configurations of the same systems are registered too (reducto_extract, extend_extract, codex_code_extract_gpt_5_5_high, the two-stage parse baselines, and more). Run uv run extract-bench pipelines for the full roster.

All three LlamaExtract tiers return word-level citation boxes, so both grounding metrics are meaningful on each. llamaextract_agentic_plus does it natively. llamaextract_cost_effective and llamaextract_agentic get there by running a parse at their own tier that emits word boxes, which is a second job. llamaextract_cost_effective_standard_bbox and llamaextract_agentic_standard_bbox are those two tiers without that parse pass: one job instead of two, but citations carry only block-level boxes, so word-level grounding scores near zero.

What each task challenge tests:

• T1: long-list completeness. Recover every record of a repeated structure that can span many pages. Typical failures are truncation, duplicated or merged rows, hallucinated records, and values attached to the wrong record.
• T2: needle-in-haystack. Find a small number of requested facts in a long document. T2 has few target records but many plausible mentions, only one of which is canonical; failures are missed targets, wrong occurrences, and unnormalized paraphrases.
• T3: dense documents. Fill many fields from a document dense with labels, blanks, checkboxes, handwriting, and scan artifacts. The characteristic failure is over-extraction, inventing a value for a field that is actually blank, compounded by missed checkboxes and mislabeled fields.

A document can carry more than one task challenge.

The other axes are tagged independently of the task challenge:

• Table structure — S1 merged headers, S2 header not at top / pivoted, S3 cross-page table, S4 enormous table, S5 table within a cell.
• Perception challenge — P1 rotated or image-only capture, P2 scanned page images, and P3 handwriting. 38 documents are degraded re-captures of documents that also appear clean, so capture degradation is a paired measurement on the same documents.
• Business domain — D1 finance (145), D2 energy (98), D3 government (49), D4 automotive (27), D5 supply chain (20), D6 healthcare (15), D7 legal (10), D8 real estate (6).

Sources. All documents come from public records: SEC and regulatory filings, government procurement and customs forms, court and agency exhibits (including tax forms such as W-2, 1040, K-1, and 1099-B), Texas Railroad Commission energy filings, and published business documents. 325 are real; 45 are synthetic long lists rendered from real layouts. PDF metadata has been stripped from every file.

Ground truth uses a method matched to each source: adjudicated agreement across independent extraction systems for real documents, values fixed before rendering for synthetic long lists, and human-verified values and boxes for forms. Each field's ground truth is an evidence list — the expected value plus any alternate acceptable readings, each with its source location — and scoring accepts a match against any listed reading.

• Array alignment. A repeated structure compares as an unordered set of records: records pair by a globally optimal one-to-one assignment (the Hungarian algorithm) that minimizes mismatched cells. Unmatched expected records lower recall, surplus predictions lower precision.
• Normalization. Matching is deterministic and mostly exact: dates in eight written formats canonicalize to ISO form, strings compare exactly after whitespace collapsing, and everything else uses plain equality, with no numeric tolerance and no LLM judge.
• Missing values. An omitted key scores as an explicit null, every scalar field enters both denominators, and a correct null on a blank field is credited. Only repeated records move precision and recall apart, so a gap between them means records were dropped or invented.

For fine-grained control over individual steps:

# Run inference only (call the extraction system, don't evaluate)
uv run extract-bench inference run <pipeline_name> data/short --output_dir output

# Run evaluation only (on existing inference results)
uv run extract-bench evaluation run output/<pipeline_name> --test_cases_dir data

# Generate detailed HTML report from evaluation results
uv run extract-bench analysis generate_report --evaluation_dir ./output/<pipeline_name>

To add a new extraction system, use Claude Code:

/integrate-pipeline <name> <API docs or SDK link>

This creates the provider, registers the pipeline, and updates docs. The skill definition lives in .claude/commands/integrate-pipeline.md and can be adapted for other AI coding agents.

output/
├── _leaderboard.html                       # Cross-pipeline leaderboard
└── <pipeline_name>/
    ├── short/
    │   ├── *.result.json                    # Inference results
    │   ├── _evaluation_report.json          # Evaluation summary
    │   ├── _evaluation_report_detailed.html # Interactive detailed report
    │   ├── _evaluation_results.csv          # Per-example CSV
    │   └── _evaluation_report.md            # Markdown summary
    ├── medium/  (same structure)
    ├── long/    (same structure)
    ├── _errors.json                         # Per-document inference failures
    └── _metadata.json                       # Run metadata

src/extract_bench/
├── cli.py                           # Fire CLI entry point
├── pipeline/cli.py                  # End-to-end pipeline orchestration
├── data/
│   ├── download.py                  # HuggingFace dataset download
│   └── cli.py                       # Data management CLI
├── inference/
│   ├── runner.py                    # Batch inference with concurrency
│   ├── pipelines/                   # Pipeline registry (extract, parse, layout)
│   └── providers/                   # Provider implementations per product type
├── evaluation/
│   ├── runner.py                    # Parallel evaluation + failure penalties
│   ├── evaluators/                  # Product-specific evaluators
│   ├── metrics/extract/             # Unified value F1, grounding, record matching
│   └── reports/                     # CSV, HTML, markdown export
├── analysis/
│   ├── detailed_report.py           # Interactive per-split HTML report
│   └── comparison.py                # Pipeline comparison
├── test_cases/
│   ├── loader.py                    # Load test cases (JSONL or sidecar .test.json)
│   └── schema.py                    # TestCase types (Extract, Parse, LayoutDetection)
└── schemas/
    ├── pipeline_io.py               # InferenceRequest, InferenceResult
    ├── evaluation.py                # EvaluationResult, EvaluationSummary
    └── product.py                   # ProductType enum