obscura-benchmark
September 12, 2026 ยท View on GitHub
Conformance, capability, and performance benchmarks for
Obscura, a headless browser engine
written in Rust for web scraping and AI agent automation. WPT coverage here is
intentionally limited to the classic DOM/JavaScript standards path. Rendering
regressions belong to Obscura's deterministic render-repros/ suite in the main
repository, where changes can be reviewed against engine-owned expectations.
Benchmark tracks
| track | question it answers | where |
|---|---|---|
| WPT conformance | how much of the web platform does Obscura implement correctly | crates/wpt-runner, crates/triage |
| Obstacle course | does it handle the modern web (React/SPA/async/web APIs), and how fast | obstacle-course/ |
| vs headless Chrome | how does its speed and memory compare to the standard headless engine | compare/ |
| Real-world corpus | does it render real public pages, including SPAs | realworld/ |
| Stealth bench | does it present a consistent, undetectable Chrome fingerprint | stealth-bench/ |
| Perf bench | per-page fetch / scrape latency on a small URL set | crates/perf-bench |
| Reliability | does it crash, panic, or hang on a large corpus of real pages | reliability/ |
Results
The latest full pass ran on 2026-09-11 against Obscura commit
01e1caa33360f6c02643457307894ec885e82eef and pinned WPT commit
03f14d4780c4d981bc84c65679b18e9327a1affe. The complete compressed JSON and
triage report are under results/published/2026-09-11-wpt-no-render/.
Web Platform Tests (conformance)
Subtest pass rate by tier (a tier is a capability scope, defined in
crates/triage/src/tiers.list, not a cherry-picked subset). A subtest is one
assertion; a single test file holds many, so subtest pass rate is the standard
conformance measure:
| tier | subtests passing | role |
|---|---|---|
| Core | 327,094 / 392,547 (83.3%) | the DOM/HTML/URL/fetch scraping contract |
| Relevant | 521,393 / 600,709 (86.8%) | Core plus broader JS-observable correctness |
| Full | 603,997 / 893,328 (67.6%) | the whole testharness suite, for transparency |
Core subtest pass rate over time:
| date | obscura state | Core subtests |
|---|---|---|
| 2026-06-03 | baseline | 8.0% |
| 2026-06-04 | round 2 | 15.4% |
| 2026-06-04 | + charset/URL encoding | 72.7% |
| 2026-06-04 | + IDL reflection, attr folding, storage | 81.6% |
| 2026-07-03 | historical main | 83.3% |
| 2026-09-11 | 01e1caa3 | 83.3% |
The "Full" tier includes large subtrees outside Obscura's classic profile scope
(layout, rendering, media, and hardware), so it is reported only for
transparency. Core and Relevant remain the headline because they exclude those
areas by capability, not by outcome. See crates/triage/src/tiers.list for the
exact rules. For cross-engine context, the same WPT areas for
Chrome/Firefox/Safari are published on wpt.fyi.
Obstacle course (capability + speed)
33 / 33 stages pass, median ~44 ms per stage (cold obscura fetch, including
process startup). Covers client-side React/Preact/Vue, SSR hydration, ES modules
and dynamic import, IntersectionObserver/MutationObserver, fetch + pushState
SPAs, the URL/TextDecoder/FileAPI/Range/Selection/custom-element/dialog web-API
surface, the --dump extraction modes, charset decoding, cookies, and stealth
fingerprint consistency.
vs headless Chrome (speed + memory)
Cold process per page, both fully rendering the same client-side app (verified:
obscura serializes the post-JavaScript DOM, e.g. 100 React <li> elements that
are not in the shipped HTML):
| page | obscura | headless Chrome | obscura advantage |
|---|---|---|---|
| react | 81 ms, 30 MB | 815 ms, 190 MB | 10x faster, 6x less memory |
| preact | 45 ms, 30 MB | 745 ms, 191 MB | 16x faster, 6x less memory |
| vue | 86 ms, 31 MB | 796 ms, 190 MB | 9x faster, 6x less memory |
Across all 33 obstacle-course fixtures the median is ~21x faster and ~7x less memory (obscura ~27 MB vs Chrome ~190 MB per process). The framework rows above are the heaviest-render, most conservative cases; lighter pages widen the gap because Chrome pays the same fixed startup regardless of the page.
Throughput and memory as concurrency rises (24 React-app loads, idle host):
| engine | 1 worker | 4 workers | 8 workers |
|---|---|---|---|
| obscura | 12 pg/s, 30 MB | 40 pg/s, 112 MB | 18 pg/s, 159 MB |
| headless Chrome | 1.2 pg/s, 1.1 GB | 3.0 pg/s, 4.2 GB | 3.0 pg/s, 8.1 GB |
Obscura sustains far higher throughput at a fraction of the memory. Chrome pays a large fixed startup (process + browser stack) on every page; under concurrency its RAM climbs into the gigabytes while Obscura stays in the low hundreds of MB. This is the no-render scraping profile; production Chrome would reuse one browser across tabs (the cold-process numbers are Chrome's worst case).
Real-world corpus (vs headless Chrome)
98 live public pages, fetched with obscura and headless Chrome side by side:
| engine | rendered | median latency | median peak RSS |
|---|---|---|---|
| obscura | 94 / 98 (95.9%) | 5.2 s | 64.2 MB |
| headless Chrome | 85 / 98 (86.7%) | 2.1 s | 201.2 MB |
Obscura renders more of the corpus than headless Chrome (94 vs 85), and at ~3x
less memory. Chrome's misses are mostly sites that serve a datacenter IP an
anti-bot or CAPTCHA wall, which is not an engine failure on either side. Latency
is mixed: obscura is much faster on static content pages and slower on heavy
client-rendered SPAs, where its post-load settle wait (--wait) dominates the
median. The live web drifts, so these are a snapshot; see realworld/sites.txt.
Reliability (crash / hang sweep)
Conformance and speed do not matter if the engine crashes or hangs on a live page. A 1500-URL corpus (a one-level crawl from the real-world seed list) is rendered through obscura, classifying each outcome from the exit code and stderr:
| outcome | count |
|---|---|
| rendered | 1432 / 1500 (95.5%) |
| thin / blocked | 67 |
| bounded hang (deadline) | 1 |
| crash (signal) | 0 |
| panic | 0 |
Zero crashes and zero panics across 1500 diverse pages. Any page that does not
finish in its budget is terminated deterministically (a V8 termination watchdog
plus a process-level hard deadline), so no page can wedge a worker. The single
bounded hang (kayak.com) is a heavy page that exceeds the tight per-page budget
under concurrency; it renders normally when run on its own. Run it with
OBSCURA_BIN=<bin> python3 reliability/sweep.py.
1. WPT conformance
crates/wpt-runner runs every supported WPT testharness file through a separate
obscura fetch process and reads the result left by
wpt-overlay/resources/testharnessreport.js. WPT reftests are not run here:
rendering regression ownership stays with the deterministic fixtures in the
main Obscura repository.
Tests requiring unimplemented runner automation (testdriver, PAC, or HTTP/3
serving) are reported as unsupported, not as engine
failures. crates/triage reports that count separately and groups real failures
into deduplicated root causes.
# one-time setup: check out the revision in wpt-config/WPT_COMMIT,
# install the report overlay, and build the manifest
scripts/setup-wpt.sh
# then add the WPT hostnames once (needs sudo), as printed by setup-wpt.sh:
# ( cd wpt && ./wpt make-hosts-file ) | sudo tee -a /etc/hosts
# full classic WPT pass
OBSCURA_BIN=/path/to/obscura-no-render \
scripts/run-wpt.sh --concurrency 32 --wait-secs 15
# a subset, by one path substring
OBSCURA_BIN=/path/to/obscura-no-render \
scripts/run-wpt.sh FileAPI/url/
The suite is roughly 38k test variants at the current pin. Use a concurrency
that does not oversubscribe the host: oversubscription makes per-test timeouts
fire and depresses the pass rate. Outputs are written under results/, including
the JSON report and a triage Markdown report.
Build the classic Obscura binary without optional features:
BENCH_BIN_DIR=/path/to/obscura-benchmark/.bench-bin
mkdir -p "$BENCH_BIN_DIR"
CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=2 cargo build --release \
-p obscura-cli --bins --no-default-features
cp target/release/obscura "$BENCH_BIN_DIR/obscura-no-render"
Run those build commands in the Obscura repository. Record its exact commit with the result commit; the WPT side is reproducibly pinned by this repository.
2. Obstacle course
A curated set of small self-contained modern-web pages that a no-DOM-engine
browser cannot handle. Each fixture runs its JavaScript and asserts a
deterministic result; the runner also times each page. See
obstacle-course/README.md for the full stage list.
OBSCURA_BIN=/path/to/obscura scripts/run-obstacle-course.sh
OBSCURA_BIN=/path/to/obscura scripts/run-obstacle-course.sh --filter react --runs 10
It exits non-zero if any stage's result does not match, so it doubles as a correctness regression check.
3. Head-to-head vs Chrome
compare/ runs Obscura and headless Chrome as cold processes over the same
pages and reports latency, peak memory, and throughput under concurrency. See
compare/README.md.
OBSCURA_BIN=/path/to/obscura CHROME_BIN=google-chrome scripts/run-compare.sh
Run it on an idle host; CPU contention skews throughput (memory is unaffected).
4. Real-world corpus
realworld/ fetches a list of live public pages and reports render-success rate,
latency, and memory. See realworld/README.md.
OBSCURA_BIN=/path/to/obscura scripts/run-realworld.sh
5. Stealth bench
stealth-bench/ evaluates Obscura's browser fingerprint, anti-detection measures,
and cross-page consistency when running in --stealth mode. It runs locally
and does not require external network access.
OBSCURA_BIN=/path/to/obscura-stealth python3 stealth-bench/run.py --stealth
6. Perf bench
crates/perf-bench times obscura fetch / obscura scrape on a small default
URL set (override by passing URLs).
OBSCURA_BIN=/path/to/obscura scripts/run-bench.sh
OBSCURA_BIN=/path/to/obscura scripts/run-bench.sh https://example.com https://news.ycombinator.com
Repo layout
crates/
wpt-runner/ classic WPT testharness runner
triage/ groups WPT failures into root causes; tiers.list defines the tiers
perf-bench/ times `obscura fetch` / `obscura scrape`
obstacle-course/ modern-web capability + speed fixtures (+ run.py, manifest.json)
compare/ obscura vs headless Chrome: head-to-head.py, scale.py
realworld/ live-page render-success corpus: sites.txt, run.py
stealth-bench/ stealth fingerprint, anti-detection, and consistency test suite
wpt-overlay/ the custom WPT report script installed into the WPT checkout
scripts/ setup and run wrappers
results/ generated run artifacts (gitignored)
wpt/ the WPT checkout (gitignored; created by setup-wpt.sh)
Setup and requirements
- Rust toolchain (
cargo build --releasebuilds the runner, triage, and perf-bench). - An Obscura binary, passed via
OBSCURA_BIN. - Python 3 for the obstacle-course, compare, and realworld runners.
compare/scale.pyusespsutilfor memory sampling (pip install psutil), and the compare/realworld harnesses read peak RSS from GNUtime -v. - For the head-to-head, a Chrome or Chromium build (
CHROME_BIN). - For WPT, the one-time
scripts/setup-wpt.sh(checks out the pinned WPT commit, builds the manifest, and installs the report overlay) plus the WPT hostnames in/etc/hosts.
results/ and the wpt/ checkout are gitignored; run artifacts are regenerated
by the scripts.