Parse Benchmark

This benchmark compares parse throughput for generated ANTLR parsers and tree-sitter parsers on Kotlin, C#, and Java fixtures, with an explicit Trino SQL fixture set for SQL-dialect Rust vs Go checks.

The harness is intentionally a standalone script instead of cargo bench. cargo bench is useful for in-process Rust-only measurements, but this check has to generate ANTLR parsers, build a Go binary, run Python parsers, and load tree-sitter language libraries. Keeping that orchestration outside Cargo makes the same command usable locally and in CI.

Setup

Use the same ANTLR jar and grammars-v4 checkout described in AGENTS.md. The repository-local directory is ignored and survives operating-system temporary-directory cleanup; cargo clean removes it. The benchmark defaults to:

• ANTLR4_JAR=target/antlr-cleanroom/tools/antlr-4.13.2-complete.jar
• GRAMMARS_V4=target/antlr-cleanroom/grammars-v4

The sparse checkout must include C#, the modern Java grammar, and Trino SQL in addition to Kotlin:

git -C target/antlr-cleanroom/grammars-v4 sparse-checkout set kotlin/kotlin csharp/v7 java/java sql/trino

Install the Python dependencies in the interpreter you will use to run the benchmark:

python3 -m pip install -r tools/parse-bench/requirements.txt

Run

Quick local smoke:

python3 tools/parse-bench/run.py --quick

SQL-only Rust vs Go smoke (with AST parity gate):

python3 tools/parse-bench/run.py \
  --languages trino \
  --runtimes rust-antlr,go-antlr \
  --ast-check \
  --quick

Longer local run with reports:

python3 tools/parse-bench/run.py \
  --iters 20 \
  --warmups 3 \
  --rust-generated-only \
  --json target/parse-bench/results.json \
  --markdown target/parse-bench/results.md

Generator experiments use the same harness through repeatable passthrough arguments. Use separate work directories so generated sources and binaries do not overwrite the baseline:

python3 tools/parse-bench/run.py \
  --languages kotlin,trino \
  --runtimes rust-antlr \
  --rust-generator-arg=--fixed-lookahead \
  --rust-generator-arg=3 \
  --work-dir target/parse-bench-fixed

The script regenerates parsers into target/parse-bench, builds:

• a Rust runner generated directly from the grammar source with this runtime,
• a Python ANTLR runner using antlr4-python3-runtime,
• a Go ANTLR runner using github.com/antlr4-go/antlr/v4,
• a tree-sitter runner using tree-sitter-language-pack.

Rust generation keeps the pinned JavaParser.g4 unchanged, including its JavaParserBase option and both semantic predicates. The generated benchmark parser uses the source compiler's historical assume-true policy for those target-language helpers, matching downstream generation while retaining the predicate metadata that controls runtime routing; since #209 those untranslated coordinates lower as generatable hook-backed templates, so the predicate-bearing rules run generated bodies rather than the ATN interpreter. Python and Go still use the equivalent portable rewrite because that grammars-v4 revision does not provide JavaParserBase implementations for those targets. The issue-174-return-expression.java fixture guards the resulting Java method-body performance path. JSON rows include a benchmark-variant tag, so the comparator skips only method changes such as this legacy-to-predicate transition (or the #209 interpreter-to-generated routing change) and resumes Java regression checks once both reports use the same variant.

Rust generation also keeps the pinned C# grammar unchanged. Its semantic helper names are described by patterns/csharp.toml, while the benchmark-owned Rust support module implements interpolation and the CSharpLexerBase preprocessor state. Generic codegen exposes typed lexer lifecycle hooks, structural channel/mode constants, and reusable parser-predicate lowerings; it does not contain C# grammar or rule names.

The output table reports min and avg parse time per fixture and a relative ratio against rust-antlr for the same fixture. Use --rust-generated-only for Adaptive LL delivery evidence so the Rust generator fails if any parser rule lacks a generated body and the Rust runner fails if a generated parser path falls back to the interpreter.

Rust vs Go AST parity

Pass --ast-check with --phase parse and both rust-antlr and go-antlr selected. Before timing, the harness dumps each fixture's parse tree from both runners (same Rule/Term/Err format as the Kotlin parity dumper) and requires:

• byte-identical dumps, and
• no error nodes on either side (Rust also rejects a non-zero syntax-error count).

Dumps land under <work-dir>/ast-dumps/<language>/.

Kotlin, C#, Java, and Trino fixtures pass this gate. The former C# divergence was caused by the missing CSharpLexerBase.NextToken preprocessing behavior: inactive #if sections were still lexed as ordinary C# by the Rust runner. The Rust support module now evaluates directives and consumes inactive sections before normal lexing resumes.

python3 tools/parse-bench/run.py \
  --languages kotlin,csharp,java,trino \
  --runtimes rust-antlr,go-antlr \
  --ast-check \
  --quick

Lex-only measurements

Use --phase lex to time generated Rust lexing and token buffering without constructing a parser:

python3 tools/parse-bench/run.py \
  --phase lex \
  --languages kotlin,csharp,java,trino \
  --runtimes rust-antlr \
  --iters 20 \
  --warmups 3

The source-derived fixtures cover ordinary Kotlin, C#, Java, and Trino input. Two lex-only Kotlin fixtures add concentrated ASCII coverage for long identifiers, strings, comments, whitespace, and punctuation, plus mixed-script coverage for the Unicode fallback. Two lex-only Java fixtures isolate long ASCII range classes for identifiers, numbers, and whitespace.

Use a detached checkout for same-machine baseline comparisons, and select the compiler-level configurations explicitly:

python3 tools/parse-bench/run.py \
  --phase lex \
  --runtimes rust-antlr \
  --runtime-root /tmp/antlr-runtime-main \
  --rust-native \
  --rust-thin-lto

--rust-native adds -C target-cpu=native. --rust-thin-lto writes lto = "thin" and codegen-units = 1 in the generated benchmark workspace, where Cargo profile settings control the final application and its dependencies.

For profile-guided optimization, first build an instrumented runner and train it on the selected fixtures:

PROFILE_DIR="$(mktemp -d /tmp/antlr-parse-pgo.XXXXXX)"
PROFILE_DATA=/tmp/antlr-parse-pgo.profdata

python3 tools/parse-bench/run.py \
  --languages kotlin,csharp,java,trino \
  --runtimes rust-antlr \
  --rust-generated-only \
  --rust-pgo-generate "$PROFILE_DIR" \
  --work-dir /tmp/antlr-parse-pgo-train

rustup component add llvm-tools-preview
SYSROOT="$(rustc --print sysroot)"
HOST="$(rustc -vV | sed -n 's/^host: //p')"
"$SYSROOT/lib/rustlib/$HOST/bin/llvm-profdata" merge \
  -o "$PROFILE_DATA" "$PROFILE_DIR"

Then rebuild and measure with the merged profile:

python3 tools/parse-bench/run.py \
  --languages kotlin,csharp,java,trino \
  --runtimes rust-antlr \
  --rust-generated-only \
  --rust-pgo-use "$PROFILE_DATA" \
  --work-dir /tmp/antlr-parse-pgo-use

The profile-generation timings are instrumented training data, not comparison results. When testing PGO together with --rust-native or --rust-thin-lto, pass the same options to both commands. Use separate work directories for the ordinary, instrumented, and profile-use runners.

Prediction Memory Counters

Set ANTLR_PERF_DUMP=1 to build the Rust runner with performance counters. Parse runs print prediction and context-store measurements; lex-only runs print lexer direct-ASCII, generic-character, scalar-replay, and bulk-commit counts:

ANTLR_PERF_DUMP=1 python3 tools/parse-bench/run.py \
  --languages csharp \
  --runtimes rust-antlr \
  --iters 10 \
  --warmups 2 \
  --rust-generated-only

The dump includes canonical context counts, pooled and retained bytes, arena and workspace capacities, merge-cache activity, and outer-context cache hits/misses. It also reports learned parser-DFA warm hits/misses, ATN fallbacks, state interning activity, dense/sparse row counts, edge-density histograms, and hot/cold retained bytes. Parser token-set counters report inline, dense, and interval selection; packed bitset bytes; membership hits/misses; and interval probes eliminated or retained. Store statistics are collected in a separate untimed parse after the benchmark loop, so walking the stores does not affect reported timings.

PR Watchdog

For CI, run the benchmark on the base checkout and the PR checkout on the same runner, then compare JSON reports:

python3 tools/parse-bench/compare.py \
  --baseline base-parse-bench.json \
  --current head-parse-bench.json \
  --max-regression 1.15

By default the comparator checks rust-antlr only. Repeat --runtime to add other runtimes.

Fixtures

Fixture metadata lives in fixtures/manifest.json. The fixture files are source-referenced benchmark inputs that point at independent upstream parser stress patterns:

• Kotlin: JetBrains Kotlin, kotlinx.coroutines, Ktor.
• C#: dotnet/wpf, Mono.
• Java: Mojang DataFixerUpper, Bazel, Google Closure Compiler, Trino.
• Trino SQL: Trino benchmark TPC-DS/TPC-H queries with benchmark placeholders normalized to identifiers, including a curated TPC-DS grammar-stress suite selected by CTE/window/UNION/EXISTS/CASE/grouping feature density.