Warp-Ingest on ParseBench
July 7, 2026 · View on GitHub
Primary faithful run (2026-07-07, v2.0.1)
Use this section as the primary ParseBench score for Warp. The run uses the
official ParseBench framework and upstream llamaindex/ParseBench data with
the renderer locked to faithful Warp-only mode. All output — including the
table cells — is produced by this repository's own code: table grids come
from Warp's license-clean native table engine
(warp_ingest/ingestor/table_engine.py, pure MIT stack, the default and now
only WARP_TABLE_PROVIDER; the legacy pymupdf4llm ablation path was
removed in PR #6 after the native engine tied it head-to-head).
python -m benchmarks.parsebench.run --full --force \
--pipelines warp_ingest_faithful --max-concurrent 1
| Pipeline | Tables | Charts | Content Faith. | Sem. Format. | Visual Ground. | Overall |
|---|---|---|---|---|---|---|
warp_ingest_faithful (v2.0.1, 2026-07-07) | 57.45 | 7.04 | 70.81 | 45.81 | 20.71 | 40.36 |
warp_ingest_faithful (native tables, 2026-07-02) | 57.42 | 7.13 | 68.27 | 43.46 | 18.57 | 38.97 |
| prior faithful (warp raw tables, 2026-07-01) | 36.45 | 7.10 | 69.28 | 47.14 | 18.57 | 35.71 |
The v2.0.1 gain over the 2026-07-02 baseline (+1.39 Overall) comes from PR #5's
engine/front-end root fixes for the top Content-Faithfulness drivers
(multi-column reading order incl. the OCR path, prose mis-inferred as inferred
table grids, unreadable (cid:NN) text layers): Content Faith. +2.5,
Sem. Format. +2.4, Visual Ground. +2.1, with Tables and Charts flat (±0.1).
A back-to-back rerun after PR #6 (ablation-code removal) reproduced the PR #5
numbers to within ±0.04 per dimension — the removal is score-neutral, as
expected for dead code on the native path. Visual Grounding is scored
in-run again as of PR #6: Python 3.14 switched Linux multiprocessing to
forkserver, which silently dropped the harness's in-process adapter
registrations (VG collapsed to ~10 while the run exited 0); run.py now pins
fork. The VG aggregate is the mean over the 394 layout-detection pages; the
42 image-only inputs (.jpg/.png, warp is PDF-only) fail at inference and
pad only the parse-scoped layout metrics, not the VG headline.
This is above every deterministic local parser measured in this harness — pymupdf4llm 37.5 Overall (and its published-board build 30.9), liteparse 32.8 — by +2.9 and +7.6 respectively, leading on 4 of 5 dimensions (all but Sem. Format., where pymupdf4llm's 52.0 still leads), without any external table parser or content stripping. Tables 57.45 also matches/beats the best number ever recorded here with the removed pymupdf4llm provider (57.3–57.4).
Native-vs-pymupdf4llm head-to-head (same renderer, same harness, same day;
tables dimension, in-process fasteval over all 503 table pages): native
56.93 vs pymupdf4llm 56.92 — page wins 207 vs 213 with 83 ties, i.e. a
statistical tie with the ONNX layout model, deterministically. Tables-only
(no warp fallback): native 56.39 vs pymupdf4llm 56.26; on ruled pages the
native engine is clearly ahead (68.5 vs 62.3), the model keeps an edge on
unruled/rotated layouts. Engine design + measured ablations:
docs/superpowers/specs/2026-07-02-native-table-engine-design.md.
Honest deltas within the 2026-07-02 run (native table engine vs warp raw
tables): Content Faith. −1.0 and Sem. Format. −3.7 — table regions that
previously rendered as bold prose/heading blocks now render as <table> cells
(no ** markup inside cells), a deliberate trade that bought +21 Tables;
PR #5 then recovered (and exceeded) both losses at the root.
Run evidence (2026-07-07):
- Official category reports are under
parsebench_work/output/warp_ingest_faithful/. - Inference deduplicated 2,553 test cases to 2,078 unique files; the only
failures are the 42 image-only layout inputs (
.jpg/.png) — Warp's provider is PDF-only. They score zero on the parse-scoped layout metrics (see above re: the VG headline). - Category totals match ParseBench: Tables 503, Charts 568, Content 506, Semantic Formatting 476, Visual Grounding 500.
The older warp_ingest / "Warp-Quality" numbers in this file are not a clean
faithful score. They used opt-in behavior (WARP_TABLE_PROVIDER=auto, which
delegated table grids to pymupdf4llm when installed, plus render-boundary
content stripping). They remain useful as ablations, but should not be cited as
Warp's primary ParseBench result.
Benchmark: the official LlamaIndex ParseBench — 2,078 human-verified pages (2,553 per-category test cases) from real enterprise documents (insurance, finance, government, legal), scored across five capability dimensions with deterministic, rule-based metrics (GriTS + TableRecordMatch for tables, chart-data-point matching, content-faithfulness, semantic-formatting, layout element pass-rate).
Warp-Ingest is scored by running it through the official framework via a
warp_ingest provider (benchmarks/parsebench/). Nothing about the scoring is
reimplemented — we call ParseBench's own PipelineCLI → inference →
rule-based evaluation → report. A three-part audit (providers, scoring,
dataset/layout — see "Faithful reuse" below) confirmed the harness reuses the
official code and that the numbers sit on the same scale as the published
leaderboard. The strongest proof is the cross-check at the bottom: 6 of the 7
deterministic baselines reproduce their published leaderboard scores exactly.
On "no LLM-as-a-judge." ParseBench's five headline metrics are rule-based. The framework does ship one optional LLM step —
claude-haiku"normalization" of failed chart rules — which defaults on (LLAMACLOUD_BENCH_LLM_NORMALIZATIONdefaults tojudge) but (a) only ever emits separate*_judgecolumns we never read — the headlineavg_rule_pass_rateis always the deterministic parent value — and (b) needs anANTHROPIC_API_KEYto do anything. To make the guarantee literal, our harness forces it off (=off), so this run is fully offline and reproducible with zero API calls.
Historical quality-mode ablation (2026-07-01)
Re-ran the full suite in Warp-Quality mode (tables provider on) after the
olmOCR-bench gap-closure changes
(docs/superpowers/specs/2026-07-01-olmocr-layout-gap-closure-design.md), which
added a within-page header/footer stripper + a CJK/emoji strip to the shared
warp_markdown renderer. Per the project rule — generally-applicable
improvements only, no per-benchmark configs — those apply here too; a small
degradation is the accepted cost.
| Dimension | old Warp-Quality | new engine | Δ |
|---|---|---|---|
| Tables (GTRM) | 57.4 | 57.4 | +0.0 |
| Charts | 7.5 | 7.5 | +0.0 |
| Content Faith. | 68.1 | 67.0 | −1.1 |
| Sem. Format. | 46.4 | 40.7 | −5.7 |
| Visual Ground. | 17.6 | 18.6 | +1.0 |
| Overall | 39.4 | 38.2 | −1.16 |
Within this opt-in quality-mode comparison, Warp-Quality 38.2 remained above
the local deterministic baselines shown here (pymupdf4llm 37.5, LiteParse 32.8).
This is not the faithful primary score because WARP_TABLE_PROVIDER=auto can
delegate table grids to pymupdf4llm when installed and render-boundary content
stripping is enabled. The Sem.Format dip is the within-page H/F stripper removing
a few bold/heading margin blocks (running-head titles) — the same change that
lifts olmOCR-bench headers_footers 48→80 and Overall 33.7→40.7
(benchmarks/olmocr_bench/RESULTS.md). Tables/Charts are unchanged in that
quality-mode run; Visual Ground. is unaffected (computed from raw block geometry).
2036/2078 parsed — the 42 failures are .jpg image-only cases Warp rejects
(PDF-only), same as prior runs. Full-dataset run, mc=1 (~52 min inference),
official harness, WARP_TABLE_PROVIDER=auto.
Recovery pass (2026-07-01) — two generally-applicable fixes, 38.2 → 38.52
A per-rule diagnosis localized the −1.16: 80% is the H/F stripper, split into
(a) a precision bug — it also removed real titles/bold (is_title/is_bold
targets like "LEGAL NOTICES", "CORPORATE OBJECTIVES"), and (b) an irreducible
conflict — ParseBench's is_header/is_footer (PageSectionRule) want
running chrome kept+labeled while olmOCR-bench's absent tests want it removed;
plus the CJK strip (olmOCR-bench baseline disallows CJK; ParseBench scores it as
content). Only the precision bug is recoverable without a per-benchmark config.
Two fixes, both universally correct (no monkey-patch, no per-benchmark toggle):
- Font-gate title-protection (
_strip_page_chrome): a block larger than the body font is a heading/title, never running chrome → never stripped. Recovers the clean false-positives. Sem.Format 40.7 → 41.9. - Blank-recovery (
pdf_plumber_parser._render_page): an unreadable text layer (e.g. a malformed ICC-colorspace stream that crashes pdfminer) now falls back to OCR instead of dropping the page. Content 67.0 → 67.3.
| Dimension | prev | recovered | Δ |
|---|---|---|---|
| Tables / Charts / Visual | 57.4 / 7.5 / 18.6 | 57.3 / 7.5 / 18.6 | ~0 |
| Content Faith. | 67.0 | 67.3 | +0.3 |
| Sem. Format. | 40.7 | 41.9 | +1.2 |
| Overall | 38.2 | 38.52 | +0.32 |
Within quality mode, this remained above the local deterministic baselines. The residual gap to 39.4 is the irreducible olmOCR/ParseBench conflict (chrome + CJK): closing it would require dropping olmOCR-bench below LiteParse or a per-benchmark config, both disallowed. olmOCR-bench held its lead at 40.5 (was 40.7; H/F font-gate −2.2 on headers, blank-recovery +0.4 on baseline).
Historical quality-mode context: olmOCR-bench 40.7 (≥ LiteParse 40.4) and ParseBench-Quality 38.2 (> pymupdf4llm 37.5). Do not cite that as the faithful ParseBench score.
Historical local-parser comparison (pre-faithful audit)
Higher is better; all scores 0–100. Overall = arithmetic mean of the five
dimensions (the official leaderboard aggregation). In this earlier comparison,
every parser scored all 503/568/506/476/500 pages per dimension with 0 parse
errors.
| Parser (version) | Tables | Charts | Content Faith. | Sem. Format. | Visual Ground. | Overall |
|---|---|---|---|---|---|---|
pymupdf4llm 1.28.0 ‡ | 55.7 | 2.1 | 67.2 | 52.0 | 10.7 † | 37.5 |
warp_ingest fdee85a | 35.8 | 6.7 | 68.3 | 46.6 | 17.6 | 35.0 |
liteparse (no OCR) 2.2.1 | 40.2 | 3.5 | 68.6 | 40.8 | 10.7 † | 32.8 |
opendataloader 2.4.7 | 35.2 | 0.9 | 66.1 | 34.1 | 10.8 † | 29.4 |
markitdown 0.1.6 | 15.8 | 2.0 | 64.5 | 0.9 | 9.9 † | 18.6 |
pymupdf (HTML) 1.28.0 | 0.0 | 0.0 | 55.6 | 18.3 | 9.2 † | 16.6 |
pymupdf (Text) 1.28.0 | 0.0 | 0.0 | 68.3 | 0.9 | 10.9 † | 16.0 |
pypdf 6.7.0 | 0.0 | 0.0 | 62.5 | 0.9 | 10.9 † | 14.9 |
† Visual Grounding for the text-only baselines is the published leaderboard
value — they emit no geometry, so the layout category cannot be scored in-run
(only warp_ingest carries real per-block boxes, via the same official
@register_layout_adapter extension point LlamaParse/Docling/Azure use). Every
other cell is from that run.
‡ pymupdf4llm 1.28.0 is newer than the leaderboard's build (published
PyMuPDF4LLM Overall 30.88). Its higher score here — driven by much-improved table
extraction (55.7 vs published 36.7) — reflects that newer version, not a harness
difference. We report the current version of every parser and disclose it
plainly; see the cross-check below.
Where historical warp_ingest stood (rank among the 8 deterministic local parsers)
| Dimension | warp | rank | leader |
|---|---|---|---|
| Tables | 35.8 | 3 / 8 | pymupdf4llm 55.7 |
| Charts | 6.7 | 1 / 8 | warp_ingest |
| Content Faithfulness | 68.3 | 2 / 8 | liteparse 68.6 (≈ tie) |
| Semantic Formatting | 46.6 | 2 / 8 | pymupdf4llm 52.0 |
| Visual Grounding | 17.6 | 1 / 8 | warp_ingest |
| Overall | 35.0 | 2 / 8 | pymupdf4llm 37.5 |
Honest read for that run. Among deterministic, local, no-API parsers, warp_ingest was a top-2 result (2nd of 8) — ahead of liteparse and every simpler baseline, behind only the current pymupdf4llm. Its genuine differentiators: it is the only local parser that carries real visual grounding (17.6 vs the ~10 no-geometry floor of all the others) and it leads on Charts; its Content Faithfulness (68.3) ties the best (liteparse 68.6) and its Semantic Formatting (46.6) is strong. It trails pymupdf4llm specifically on Tables (35.8 vs 55.7). This matters because warp is a layout/RAG/structure parser (its target is the OpenContracts hierarchy), so being top-2 on a markdown-fidelity benchmark while being the sole local parser with real geometry is a strong showing — but it does not "win overall," and the previous version of this file overstated that by omitting pymupdf4llm.
Closing the gap to pymupdf4llm — what was tried (honest)
A page-level investigation (50-page warp-vs-pymupdf4llm isolation set + per-page
GriTS diff over all 503 table pages; helpers in scripts/parsebench_summarize.py)
located the whole gap and its cause:
- Tables (−19.9) is the dominant gap and is engine-level, not a harness fix.
Warp over-segments tables on 266/503 pages (emits N
<table>s for one ground-truth table), with garbled cell/column structure and multi-column prose fused into fake tables. A harness table-coalescing experiment (merge contiguous table fragments into one<table>) was implemented and measured: it reduced over-segmentation (266→113) but regressed Tables 35.8→33.2 — it helped 114 pages (+1145) yet destroyed 189 (−2463), because GriTS pairs each predicted table to ground truth (so fragments win more pairings) and the merge wrongly fused genuinely-separate tables (e.g. Victoria's Secret 10-K p1 1.000→0.222). Reverted. Closing Tables honestly requires engine work on table cell/column segmentation + multi-column reading order (risk-gated by the S-1 suite), not benchmark-harness changes. - Format (−5.4): italic surfacing shipped (genuine, regression-safe). Warp
surfaced bold but no italic (ParseBench
is_italic5.7 vs pymupdf4llm 51). Added a purely-additiveitalic_ratio/italic_maskin the engine (visual_ingestor._annotate_block_emphasis, mirroringbold_maskvia the realline_stylefont-style) and a renderer that wraps real bold/italic/bold-italic runs in**/*/***. This liftsis_italic5.7→35.5 with 0 S-1 / OC / line-parser regressions (152 passed), butis_italicis a minor component of the headline, so Semantic Formatting moves only 46.6→46.7 — genuine, but headline-neutral. - Visual Grounding (warp's win, 17.6) is genuinely capped, not adapter-limited.
The ground-truth layout is 14%
Picture, 3%Page-header, 3%Page-footer— ~20% of elements warp emits nothing for (image detection is deferred, issue #1; warp strips page furniture), so they are unmatchable; plus box-granularity (only 35% of warp boxes reach IoA≥0.5 vs GT regions). Warp's labels do project correctly (SECTION_HEADER→Section,LIST_ITEM→Textin the core view), so this is real capability, not a mapping bug.
Bottom line for that comparison: every meaningful lever to overtake pymupdf4llm (37.5) is genuine engine work — Tables being the highest-ROI and the only one large enough to flip the ranking. The faithful rerun above now locks the renderer to Warp-only output; quality-mode gains should be treated as ablations. The path to improving the faithful score is an engine table/multi-column project (scoped, S-1-baseline-regenerating), not a benchmark tweak.
How warp compares to layout-capable (non-local) parsers — for scale
warp's Visual Grounding (17.6) beats the text-only local floor but is far below the geometry-native VLM/commercial systems on the published board (context only, not this run): Docling-models 66.1, Azure DI (Layout) 73.8, LlamaParse Agentic 80.6. warp wins Visual Grounding within the local-parser class, not against dedicated layout models.
Baseline reproduction cross-check
The earlier local-parser run's deterministic baselines vs the published
leaderboard.csv
(in-run Overall minus published Overall):
| Parser | published Overall | this run | Δ | note |
|---|---|---|---|---|
| liteparse (no OCR) | 32.8 | 32.8 | 0.0 | Sem.Format −3.8 (pip lit 2.2.1 wheel vs board build); other dims exact |
| opendataloader | 29.40 | 29.4 | 0.0 | exact on all five dimensions |
| markitdown | 18.63 | 18.6 | 0.0 | exact on all five dimensions |
| pymupdf (HTML) | 16.62 | 16.6 | 0.0 | exact on all five dimensions |
| pymupdf (Text) | 16.02 | 16.0 | 0.0 | exact on all five dimensions |
| pypdf | 14.87 | 14.9 | 0.0 | exact on all five dimensions |
| pymupdf4llm | 30.88 | 37.5 | +6.6 | newer wheel 1.28.0 (Tables +19) — disclosed, not a scale issue |
Six of seven deterministic baselines reproduce the published numbers within rounding; the seventh differs only by a documented version bump. This is direct evidence that the harness scale matches the leaderboard. It does not make the quality-mode Warp row the primary faithful score.
Faithful reuse of the official framework (audit summary)
Three independent audits of the pinned ParseBench checkout (b74caa14):
- Providers / baselines.
WarpIngestProviderimplements the officialProvidercontract (run_inference→RawInferenceResult,normalize→InferenceResult) exactly like the upstream local providers; all four original baseline pipeline names (liteparse_markdown,markitdown,pymupdf_text,pypdf_baseline) plus the three added here (pymupdf4llm_markdown,opendataloader_markdown,pymupdf_html) are real official pipelines. Thelit-binary redirection points at the pipliteparsewheel — same Rust engine, same--no-ocrflags. Tables are emitted as HTML<table>exactly as the upstream liteparse/markitdown providers do. Zero scoring is reimplemented. - Determinism. 4/5 dimensions are pure rule-based regardless of config; the
charts headline is the deterministic parent value (see the note up top). Overall
is the arithmetic mean of the five dims. Our harness reads exactly the official
aggregate keys (
avg_grits_trm_composite,avg_rule_pass_rate,avg_content_faithfulness,avg_semantic_formatting,avg_layout_element_rule_pass_rate). - Dataset / layout. The full set (test=False) is the same 2,078-page corpus
the leaderboard was computed on;
--testis a 12-page smoke split only. The Visual-Grounding adapter uses the official@register_layout_adapter/@register_layout_label_mapperextension points (same mechanism as LlamaParse/Docling/Azure/Reducto); the passthrough mapper is neutral (warp already emits Canonical-17 labels) and the scorer strictly penalizes missing, extra, and mislabeled elements — it cannot be inflated by emitting clean labels.
Honest caveats
- warp is a layout/RAG parser, not a Markdown-fidelity parser. It surfaces
bold from the engine's own per-word font weight (only real bold is wrapped
in
**…**; no synthetic formatting is invented), which is why it does well on Semantic Formatting, but it carries no strikethrough/super-/sub-script. - Tables (36.5). warp is table-limited; the engine's table boxes are at visual-line granularity and sometimes fuse multi-column bodies, which GriTS penalizes. pymupdf4llm's dedicated markdown-table extraction (55.7) is the local leader.
- Charts are images; warp (like every local parser) extracts no chart data points, so all of them sit near the floor — warp's faithful score is 7.1, still low.
- OCR. Scanned pages are routed to warp's built-in auto-OCR
(
rapidocr-onnxruntime 1.4.4); install the[ocr]extra before--fullso the comparison matches liteparse (which OCRs scanned pages even in "no-OCR" mode). The faithful run parsed all PDF inputs; 42 image-only layout inputs are counted as zero because the Warp provider is PDF-only. - Visual Grounding comparison is within-class. warp leads the local field but is far below geometry-native models (see "for scale" above).
pdf_inspector(published 26.59) is omitted: it needs a Rustpdf2mdbinary (cargo install pdf-inspector) not present in this environment. Its published row is shown for context only.
Published full-dataset leaderboard (local / deterministic rows, for context)
Source: run-llama/ParseBench/leaderboard.csv (pinned b74caa14).
| Provider | Overall | Tables | Charts | Content Faith. | Sem. Format. | Visual Ground. |
|---|---|---|---|---|---|---|
| LiteParse (no OCR) | 32.8 | 40.3 | 3.4 | 68.6 | 44.6 | 10.7 |
| PyMuPDF4LLM (older build) | 30.9 | 36.7 | 1.6 | 60.9 | 44.6 | 10.7 |
| OpenDataLoader | 29.4 | 35.2 | 0.9 | 66.1 | 34.1 | 10.8 |
| pdf-inspector | 26.6 | 26.6 | 5.3 | 56.1 | 35.1 | 9.9 |
| MarkItDown | 18.6 | 15.8 | 2.0 | 64.5 | 0.9 | 9.9 |
| PyMuPDF (HTML) | 16.6 | 0.0 | 0.0 | 55.6 | 18.3 | 9.2 |
| PyMuPDF (Text) | 16.0 | 0.0 | 0.0 | 68.3 | 0.9 | 10.9 |
| pypdf | 14.9 | 0.0 | 0.0 | 62.5 | 0.9 | 10.9 |
| Docling-models (VLM, for scale) | 50.7 | 66.4 | 52.8 | 66.9 | 1.0 | 66.1 |
Reproduce
bash benchmarks/parsebench/setup_parsebench.sh # one-time: pin + install framework + baselines
pip install ".[ocr]" pymupdf4llm opendataloader-pdf # warp OCR + the two extra local baselines
# Fully deterministic, offline run (the harness already forces LLM-normalization off):
python -m benchmarks.parsebench.run --full --force \
--pipelines warp_ingest_faithful,liteparse_markdown,pymupdf4llm_markdown,opendataloader_markdown,markitdown,pymupdf_html,pymupdf_text,pypdf_baseline
# Comprehensive table + faithfulness cross-check vs the published leaderboard:
python scripts/parsebench_summarize.py --output-dir parsebench_work/output