carve-bench

Performance benchmarks for the Carve markup engines. Each engine renders the same documents to HTML in-process, many times, and reports throughput. This is a speed comparison - every engine passes the same conformance corpus, so correctness is not what is being measured here.

Engines covered: carve-js (TypeScript), carve-php (PHP), carve-rs (Rust). The carve-go / carve-py / carve-rb bindings wrap the carve-rs engine, so their core render speed tracks carve-rs plus a thin FFI/IPC layer.

Results

The headline question is the core route: the default conversion API with no opt-in extensions registered, against the fastest same-language peer.

Language	Carve	MB/s	Fastest peer	MB/s	Carve vs peer
Rust	carve-rs	104.46	pulldown-cmark	115.62	0.90x
JavaScript	carve-js	9.98	djot.js	5.78	1.73x
PHP	carve-php	15.69	djot-php	17.82	0.88x

The same core route, Carve engine against Carve engine on the identical document:

Engine	Language	ms/op	MB/s	rel
carve-js	JavaScript	4.7084	9.98	10.47x
carve-php	PHP	2.9938	15.69	6.66x
carve-rs	Rust	0.4498	104.46	1.00x

Every peer row, the capability breadth behind each row, and the method are in COMPARISON.md.

The published numbers are deliberately split into two tracks:

Track	Question	Route	Peers
A: core source-to-HTML	How fast is the normal core conversion API?	Default configuration, borrowed facade where accepted	Same-language Djot/CommonMark libraries
B: full corpus and configured tiers	How does the whole language scale, and what do opt-in extensions cost?	Mixed 1,325-document corpus plus PHP Tier 1/2/3	Carve implementations and internal tiers

Track A and Track B are not interchangeable. A Track-A library may avoid an owned public AST; Track B intentionally measures the cost that facade hides. Competitors are absent from the full Carve corpus because they do not implement equivalent syntax there, so such rows would measure literal/error recovery rather than equal work.

Every table in both documents is regenerated by compare.mjs / run.mjs; no row is transcribed by hand, including the PHP tier table.

Reproducing the Carve rows needs engine checkouts, not the published packages. The npm and Packagist releases lag the engine heads the tables name: measured on 2026-08-21, npm @markup-carve/carve 0.1.4 rendered the comparison document at 1.99 MB/s against 11.81 MB/s for carve-js main, and Packagist markup-carve/carve-php 0.1.5 does not contain the borrowed facade at all. Set CARVE_JS / CARVE_PHP_SRC / the cargo patch at the heads listed in COMPARISON.md before comparing against a published number. Every generated report names the engine each row was produced by, so a run against the published packages cannot be mistaken for a run against the heads.

Measured hotspots and optimization candidates are in FINDINGS.md. The comparison's auditable workload scoring is in FEATURES.md. The follow-up architecture prototypes and costed recommendations are in ARCHITECTURE.md. Numbers are machine- and version-specific - run them yourself; treat them as relative, not absolute.

Documents

corpus/ holds three sizes built from the spec corpus: small (~1 KB), medium (~13 KB, the whole corpus concatenated) and large (~100 KB, the corpus repeated). Regenerate from a local carve checkout:

CARVE_REPO=../carve node scripts/gen-corpus.mjs

Running

Each engine has a small harness under engines/ that takes <doc> <iters> and prints one JSON line (ms_per_op, mb_per_s). run.mjs runs every engine over every document and writes RESULTS.md.

# 1. Build the Rust harness (release):
(cd engines/rs && cargo build --release)

# 2. Make the JS and PHP engines resolvable (see "Engine resolution").

# 3. Run:
node run.mjs              # full run
node run.mjs --quick      # few iterations, to smoke-test the harness

For a publication run, use a clean PHP INI so a globally loaded coverage or debug extension cannot disable JIT:

CARVE_PHP_INI='-n -d extension=ctype -d extension=mbstring' \
CARVE_RUN_META='YYYY-MM-DD on HOST; Node X, PHP Y tracing JIT, rustc Z.' \
CARVE_CORPUS_SNAPSHOT='carve `REV` (N documents).' \
node run.mjs

The engine line is not among those: each harness reports the engine it resolved and the report is written from what came back, so it cannot name a revision the run did not use. See "Engine pinning and provenance".

Accept the PHP rows only when every harness line reports jit=true.

For the same-language comparison, install the locked dependencies in each engine directory, build both Rust binaries, generate the comparison corpus, then run:

node scripts/gen-comparison-corpus.mjs
(cd engines/js && npm ci)
(cd engines/php && composer install)
(cd engines/rs && cargo build --release)
CARVE_JS=../carve-js/dist/index.js CARVE_PHP_SRC=../carve-php/src node compare.mjs
node scripts/gen-charts.mjs

compare.mjs runs PHP with -n, so it loads ctype and mbstring explicitly; carve-php calls ctype_alnum, and a build where ctype is a shared extension fatals without that flag.

compare.mjs implements the documented 48 KiB, warm, min-of-five method. It uses equivalent native syntax for each markup family rather than feeding Carve syntax to a Markdown parser. Competitor-facing Carve measurements always use Tier 1/core; Tier 2 and Tier 3 are separate internal diagnostics. Environment overrides use the same variables as the cross-Carve run, plus CARVE_RS_COMPARE_BIN for the comparison binary.

Engine resolution

The harnesses resolve each engine via environment variables, so you can point at a published package or a local checkout:

Engine	Env var	Default
carve-js	CARVE_JS	@markup-carve/carve (the npm package)
carve-php	CARVE_PHP_AUTOLOAD	engines/php/vendor/autoload.php
carve-php	CARVE_PHP_SRC	unset - a checkout's src/, prepended
carve-rs	CARVE_RS_BIN	engines/rs/target/release/carve-bench-rs

Every harness reports what it resolved as carve_source in its JSON line, and run.mjs / compare.mjs write the report's engine line from those values.

Prefer CARVE_PHP_SRC for a checkout. CARVE_PHP_AUTOLOAD alone cannot beat the benchmark's own vendored carve-php: Composer prepends that loader, so it resolves MarkupCarve\Carve\* first and the checkout never runs. CARVE_PHP_SRC rewrites the PSR-4 prefix instead and works against a bare worktree with no vendor/.

Example, all three from local checkouts beside this repo:

export CARVE_JS=../carve-js/dist/index.js
export CARVE_PHP_SRC=../carve-php/src
# engines/rs deps on the published carve-lang crate; for a local checkout instead:
(cd engines/rs && cargo build --release \
  --config 'patch.crates-io.carve-lang.path="../../../carve-rs"')
node run.mjs

Engine pinning and provenance

Comparability is what this repo produces, so every lane names one exact published release of its engine and none of them can move without a tracked manifest line changing:

Lane	Manifest	Requirement
carve-js	engines/js/package.json	0.1.4 - npm is exact without a range operator
carve-php	engines/php/composer.json	0.1.5 - Composer is exact without a range operator
carve-rs	engines/rs/Cargo.toml	=0.1.3 - the = matters, a bare version is a caret range in Cargo

To move a lane onto a newer engine release, edit that requirement and refresh the lockfile beside it (npm install, composer update markup-carve/carve-php, cargo update -p carve-lang), then re-run the benchmarks: a table mixing engine revisions is not a comparison. Measuring an unreleased engine is an override at run time, not an edit to these manifests.

The report says which of the two happened. Each harness resolves its engine, reports it as carve_source, and the generated documents name it - a published release by version and package checksum or reference, a checkout by path and revision. An engine that reports nothing is written as unreported rather than omitted, and one that resolves two different sources within a single run is written as a MISMATCH, because neither run is comparable and the report is where that has to be visible.

For the PHP extension-stack measurement, use a clean INI so a loaded coverage extension cannot disable JIT silently:

for profile in tier1 tier2 tier3; do
  php -n -d extension=mbstring -d opcache.enable_cli=1 \
    -d opcache.jit=tracing -d opcache.jit_buffer_size=128M \
    engines/php/tiers.php "$profile" corpus/comparison/carve.crv 5 5
done

Here tier3 means Tier 1 + every Tier-2 extension + the reproducible zero-configuration Tier-3 bundle listed in FINDINGS.md; the specification has no canonical all-Tier-3 profile because app extensions can require host data or callbacks.

COMPARISON.md reports both portable-workload points and the substantially broader core capability points enabled in each exact configuration. The latter make parser scope visible but are not a speed-normalization divisor; see FEATURES.md.

Method notes

• Timing is in-process (no per-render process startup), so it measures render throughput, not CLI launch cost.
• Each harness warms up before timing (JIT for JS, opcode cache for PHP).
• rel in the results is relative to the fastest engine per document.
• The comparison is parse + render, not parse-only. Feature sets and generated HTML differ, so it is a throughput comparison over equivalent representative inputs, not an output-equivalence claim.
• The PHP engine is benchmarked with opcache.enable_cli=1 and opcache.jit=tracing so it reflects production PHP performance. Coverage/debug extensions (xdebug, pcov) must be disabled before benchmarking PHP - they override zend_execute_ex, which disables JIT and inflates timings by roughly 2x. The harness warns on stderr if either is detected or JIT is not active.
• Every harness's JSON includes carve_source, the engine it actually resolved. Verify it before accepting a comparison. In PHP this also guards against Composer autoloader precedence silently benchmarking the vendored release instead of the checkout CARVE_PHP_SRC names.