datalogic-bench

August 25, 2026 · View on GitHub

Dev-only benchmark harness for datalogic-rs. For the latest captured matrix and headline numbers, see BENCHMARK.md — link to that file from other docs rather than re-quoting cells inline.

Four binaries share a common suite loader and reporter (src/lib.rs):

BinaryPurpose
selfTimes datalogic-rs alone using the fast arena path (compile once, persistent input arena, eval-arena reset). Use this to track regressions in our own engine.
compareCross-library matrix: runs every suite against every available subject (datalogic-rs API tiers, gated Rust crates, JS/WASM via Node) and prints a markdown table of avg ns/op.
boundary_coreThe rust-core runner for the per-binding boundary benchmark under boundary/; emits the same JSON-lines schema as the other runtimes' runners.
profile_macroSampling-profiler feeder (samply / Instruments): hammers one macro suite in a hot loop so the profile shows only that suite's evaluation path.

A third area, boundary/, measures the opposite of the matrix: per-binding boundary cost — what a real caller pays per evaluation through each language binding (C ABI, Node, Python, WASM, Go, JVM, .NET, PHP) on the three workloads from BINDINGS-OVERHEAD.md. One runner per runtime, one shared discipline (warmup, ~250 ms samples, median of 5), JSON-lines output, and a renderer for that document's tables:

cd boundary && ./run.sh && python3 render.py

See boundary/README.md for runner status: all nine runners are verified and produced the 2026-07-03 v2 capture; go / dotnet / jvm / php sit in the extended set only because they need their language toolchains installed. The boundary_core bin target in this crate is the rust-core runner.

Both read JSON suites from crates/datalogic-rs/tests/suites/, and both accept --macro to swap those for the synthesized macro suites instead. Both write JSON reports to tools/benchmark/output/ (gitignored): report-self-*.json for --all runs of self, report-compare-*.json for compare, report-compare-macro-*.json for compare --macro.

self — regression baseline

# Single suite (compatible.json by default)
cargo run --release -p datalogic-bench --bin self

# All suites
cargo run --release -p datalogic-bench --bin self -- --all

# Specific suite
cargo run --release -p datalogic-bench --bin self -- arithmetic/plus.json

# Macro tier: synthesized large-payload suites (1k/10k arrays, 128-key
# objects, 48-level nesting, 10 KB strings, one eligibility rule)
cargo run --release -p datalogic-bench --bin self -- --macro

Each suite is timed three ways with the same discipline (median of 3 reps, black_box, session reset per iteration, pre-sized arena): the whole suite (the headline, comparable with older reports), just the rules the compiler constant-folded to a literal (Logic::is_constant), and the rest. The per-suite line shows the split (folded 23/32 @ 2.94 ns, rest @ 75.75 ns) and the summary reports overall / folded-only / non-folded-only geomeans, so constant-folded rules can't flatter the data-dependent number. The macro tier scales its per-suite iteration count from a pilot pass so one timed rep lands near ~250 ms; see BENCHMARK.md for the suite list.

compare — cross-library matrix

The matrix has one row per suite and one column per subject. Cells are the median ns/op of three timed samples, each sized to hit a ~200ms wall budget. Two aggregation rows at the bottom show the arithmetic mean (familiar) and geometric mean (the right average for cross-library comparison — one slow suite doesn't dominate).

Subjects

The matrix shows one column per library / API tier that takes a precompile-once approach — apples-to-apples cells. Convenience-API tiers (Engine::eval_str, Session::eval_borrowed, raw evaluate(ruleStr, dataStr, false) on the WASM) are intentionally not in the matrix because their numbers measure API-shape costs (parse cost, session reset cost, WASM string marshalling) rather than engine cost. For per-API-tier numbers on datalogic-rs alone, see bin/self.rs.

Always compiled in:

ColumnWhat it exercises
dlrs:enginePre-compiled Logic + caller-owned Bump, batch-style reset between iterations. The native baseline.

Behind a Cargo feature:

ColumnFeature flagCrate
jsonlogic-rssubject-jsonlogic-rsbestowinc/json-logic-rs 0.5 — apply(&Value, &Value), no compile API

Auto-detected at runtime (require Node + an npm install in runners/):

ColumnAPI exercised
dlrs:wasm:compiled@goplasmatic/datalogic-wasm new CompiledRule(ruleStr, false) once per rule, then .evaluate(dataStr) per call. WASM analog of dlrs:engine; remaining per-call cost is data marshall + parse + result stringify across the V8↔WASM boundary.
json-logic-jsjson-logic-js (jwadhams) — apply(rule, data), interpreted, no compile API.
json-logic-enginejson-logic-engine (TotalTechGeek) — interpreted (engine.run(rule, data)).
json-logic-engine:compiledjson-logic-engine — pre-compiled (engine.build(rule), "12.5–20× hot path" per the library's README).

json-logic-engine and json-logic-engine:compiled share their npm package but exercise different APIs (interpreter vs build-then-call).

One-time setup for Node subjects

# Build the WASM that the dlrs:wasm column points at:
cd bindings/wasm && ./build.sh

# Install the runner deps (json-logic-js + a file: link to the wasm pkg):
cd tools/benchmark/runners && npm install

If node isn't on PATH or runners/node_modules/ is missing, the matrix runner hard-fails by default (the surprise of "complete-looking matrix with silently-empty columns" is worse than an explicit error). Pass --allow-missing-subjects to render the matrix without the unavailable columns.

Run

# Single suite (compatible.json by default)
cargo run --release -p datalogic-bench --bin compare

# Specific suite
cargo run --release -p datalogic-bench --bin compare -- arithmetic/plus.json

# Every suite from tests/suites/index.json
cargo run --release -p datalogic-bench --bin compare -- --all

# Synthesized macro suites (large payloads) across all subjects; report
# lands in output/report-compare-macro-<timestamp>.json
cargo run --release -p datalogic-bench --bin compare -- --macro

# With the gated Rust competitor
cargo run --release -p datalogic-bench --bin compare \
  --features subject-jsonlogic-rs -- --all

# Allow rendering even when Node subjects aren't installed
cargo run --release -p datalogic-bench --bin compare -- --all --allow-missing-subjects

Reading the output

=== Cross-Library Matrix — avg ns/op (median of 3, ~200ms target/cell, 50 suites) ===

| Suite                | dlrs:engine | jsonlogic-rs | dlrs:wasm:compiled | json-logic-js | json-logic-engine | json-logic-engine:compiled |
|----------------------|------------:|-------------:|-------------------:|--------------:|------------------:|---------------------------:|
| arithmetic/plus.json |         2.8 |       224.4* |              518.6 |        393.4* |              73.0 |                       22.6 |
...
| arithmetic mean      |         ... |          ... |                ... |           ... |               ... |                        ... |
| geometric mean       |         ... |          ... |                ... |           ... |               ... |                        ... |

* partial coverage — subject errored on some cases in this suite.
  • Numbers are nanoseconds per evaluation (lower is better).
  • — = subject unavailable for this run (feature off, runtime missing, or precompile failed for the suite).
  • ERR = subject ran but errored on >50% of cases in the suite.
  • A trailing * on a number = subject errored on some cases in the suite but completed enough that ns/op is still meaningful.
  • Negative-test cases (entries with error: {...} instead of result) are filtered out of compare runs — engines disagree on what "errors" and how expensive their error path is, so including them would unfairly penalise verbose-error subjects.

After the matrix, a pairwise ratio table is printed:

=== Pairwise shared-suite ratios ===

  json-logic-engine:compiled      9.4x slower than dlrs:engine  over  3 shared suites
  ...

Each line is the geomean of per-suite ns/op ratios computed only over suites where both subjects have finite cells. The per-column mean rows in the matrix cover different suite subsets when subjects ERR on different suites, so quotients of column geomeans mix incomparable sets; the pairwise ratios never do. Matrix cells, per-column means, and these ratios are also written to output/report-compare-<timestamp>.json (output/report-compare-macro-<timestamp>.json for --macro runs).

Native-CPU build (optional, host-only numbers)

A .cargo/config.toml inside tools/benchmark/ adds -C target-cpu=native. Cargo only picks this up when the cwd is at or below the benchmark crate, so it's opt-in by location:

cd tools/benchmark
cargo run --release --bin compare -- --all

Numbers from a native build are not portable across machines — keep them as a relative baseline, not an absolute publishable figure. Builds invoked from the repo root remain portable.

Adding more subjects

Native Rust crate

  1. Add an optional dep + a Cargo feature in tools/benchmark/Cargo.toml: 
    [dependencies]
my-jsonlogic = { version = "X.Y", optional = true }

[features]
subject-my-jsonlogic = ["dep:my-jsonlogic"]
  2. Add a Subject impl inside bin/compare.rs, gated by #[cfg(feature = "subject-my-jsonlogic")]. Mirror the pattern of JsonLogicRs — pre-parse rule and data once, time apply() only.
  3. Push the subject into build_subjects() (also gated).
  4. Run with --features subject-my-jsonlogic.

JS / WASM library (via Node subprocess)

  1. cd tools/benchmark/runners && npm install <pkg>.
  2. Add a LIBS entry in runners/node-runner.js — one async setup that returns a callable apply(case).
  3. In build_subjects() inside bin/compare.rs, push a new NodeSubject::new("display-name", "<npm-pkg>") (gated on node_dep_installed("<npm-pkg>")).

That's the entire recipe — three files each, no harness changes.

Platform support

Linux and macOS. The Node runner uses POSIX path conventions in file:../../../bindings/wasm/pkg and the runners/ setup is shell-coded; Windows isn't tested.

CI

Don't run compare in CI. WASM build + npm install + 3+ minutes of matrix work makes for flaky CI runs. self is the regression-tracking target — keep CI on cargo test --workspace --all-features plus a single-suite self invocation if you want a perf signal.