Lexbench-Headless-Browser

English · 中文

1,928 tasks · 13 browser automation tools including Playwright, Puppeteer and Selenium

Agents are taking over the web chores that used to be done by hand: searching, comparing prices, filling forms, placing orders. The page is no longer rendered for a person to look at: the agent reads state, performs operations and collects results inside it, and the browser has become the agent's runtime.

An agent never touches the browser directly. A whole control chain sits in between: the tool layer, the driver, the control protocol, and only then the engine. That chain has no de facto standard, and different agent frameworks picked different control paths: browser-use speaks CDP through its own cdp-use client, hermes-agent drives sessions through agent-browser, and Google's chrome-devtools-mcp sits on Puppeteer. Whether an engine is usable is decided by every link on these paths holding up.

Meanwhile, a batch of new engines has appeared, among them Moli, Lightpanda and Obscura. Their pitch is lightness, and their goal is to replace Chrome in agent workloads. For the replacement to hold, the drivers of the Chrome ecosystem must keep working when pointed at the new engine, and once the protocol connects, the page semantics seen through it must be right.

That premise has never had a systematic test. WPT (web-platform-tests) is the cross-vendor suite for web-platform specifications and interoperability, and it does include tests for standardized automation protocols such as WebDriver, but its goal is not to verify that real client stacks like Playwright, Puppeteer, Selenium or agent-browser can drive a candidate engine end to end, and it offers no compatibility or resource comparison of those paths under one common standard. What engine projects publish about their own supported APIs cannot give a cross-engine, reproducible comparison either.

Lexbench-Headless-Browser turns that premise into something testable: every task starts from a real control path (raw CDP, or one of 13 pinned drivers) and walks the whole control chain, then checks whether the operation completed and whether the page semantics behind it came out right. The results also tell engine authors which interfaces they are missing and on which driver path. Chrome runs the same tasks as a reference column: it takes no part in the replacement comparison, confirms that the tasks themselves pass on a mature engine, and serves as the baseline for the resource comparison, answering how much memory and CPU each task saves once Chrome is swapped out.

Results

Five-engine results at a glance (with Kitesurf, on a 1,308-task comparable subset)

Engine	Comparable subset (1,308 tasks)
Chrome for Testing	1,306 / 1,308 · 99.85%
Moli	1,071 / 1,308 · 81.88%
Kitesurf (remote)	812 / 1,308 · 62.08%
Lightpanda	697 / 1,308 · 53.29%
Obscura	587 / 1,308 · 44.88%

This comparison uses a subset of the same 1,928-task set: tasks whose failures cannot be attributed when the engine is a remote endpoint are removed, and the removal applies to all five engines alike; removing a task does not mean Kitesurf would pass it once a local binary ships. The subset definition and the full report are on the kitesurf-eval branch.

Run parameters: run id four_engine_full_20260812, bench tag 2026.08.02-v0_4.1, seed official20260709, k=3, 23,136 result rows. A task counts as passed only when all three attempts pass. Each engine is scored on its own attempts (--score-mode independent); Chrome is the reference column. Full report: docs/reports/four-engine-report-20260812.md.

Resource Cost

The lightweight engines' biggest selling point is low resource use, and whether the replacement case holds also depends on how much they actually save, so resources are measured with the same rigor as compatibility. The task set is l1.raw_cdp (375 tasks) plus L2's l2.web_platform subset (182 tasks), 557 in total; the resource round runs strictly serial (--jobs 1) so engines never compete for the machine. Every resource statistic is computed only on the task-attempt intersection that all four engines passed (1,045 attempts): comparing cost makes sense only when everyone finished the same work, and a task an engine failed never enters its resource account.

Because observing a process can itself distort the numbers, the measurement is a two-round A/B design: the same tasks run once with the profiler off as a baseline, then once with it on, and the two rounds are compared to quantify the disturbance of observing. Numbers are recorded only when that disturbance clears the calibration gate (resource_comparison_eligible: true).

Median peak process-tree memory per task: Lightpanda 34 MiB, Obscura 39 MiB, Moli 92 MiB, Chrome 697 MiB. Median engine CPU time per task on the same set: 36 ms, 38 ms, 101 ms, 687 ms. Details and the calibration record are in the resource card.

Quick Start

Prerequisites: Linux with cgroup v2, Python 3.11+, Node 20, and one engine binary in place (where to get and put it: docs/RUNNING.md). Then two commands:

npm ci
python3 -m runner.run run --subset l1.raw_cdp --tag purpose.smoke \
  --engines chrome --score-mode independent --seed smoke

A smoke round finishes in minutes, with per-task pass/fail in runs/<run-id>/results.jsonl (a single-engine run verifies the environment and is never formally scored). In a hurry? Hand docs/RUNNING.md to your coding agent and let it set up the environment and run the bench for you.

Documentation

Benchmark Composition

Current dataset version 0.4.2: 1,928 tasks in 18 subsets, on two layers. The published run above remains pinned to 0.4.1 (recorded in its manifest as 2026.08.02-v0_4.1, the dated label used before the version scheme was unified). The difference is a PATCH: task descriptions and non-executable metadata, not task membership, drivers, graders, fixtures, or capability assignments. How the dataset and harness versions are bumped: Versioning.

L1 measures protocol and driver compatibility (1,740 tasks) across two paths:

• Raw CDP (l1.raw_cdp, 375 tasks), speaking the protocol directly on the websocket with no driver library.
• The 13 pinned drivers in the table below. 1,233 of the tasks are expanded from 116 scenario specs across the 13 drivers, so one behavior is tested once in every ecosystem; not every behavior is expressible on every driver, and a spec must record an explicit skip reason for each driver it does not bind, which scenarios --check verifies.

L2 measures web-platform semantics (188 tasks, 182 of which form the l2.web_platform subset):

• It covers DOM, storage, network, workers and CSSOM.
• It judges by what the page finally does rather than by protocol echo: a call that returns success while nothing happens on the page does not pass.
• The rows fold through a capability map into 72 capabilities; capabilities with semantic probes are scored per attempt (k=3), giving the 192 scoring units in the report's L2 row.

The 13 drivers span five language ecosystems. Every version is pinned in harness_pins.json and verified by doctor before each run:

The task set is fixed by manifest.json: task content is frozen inside a bench version, and any change requires a version bump. The set will keep growing as engines and the driver ecosystem evolve, with new subsets and tasks shipped under new bench versions.

Methodology

Every attempt passes two identity checks: an HTTP /json/version probe before connecting, and a second check on the live transport after. A failed check is recorded as infra and scores nothing. A candidate engine quietly answering with Chrome underneath is exactly what this step blocks.

Every result row carries a status (pass, fail, unsupported, timeout, crash, infra); a failing row also carries failure.class and failure.origin.

Reports under docs/reports/ come from generators that read a run's results.jsonl. Running a generator again on the same run produces the same bytes.

Functional and resource rounds never share a run. Resource figures come from the A/B protocol above and are published only when resource_comparison_eligible is true.

run_manifest.json records digests of the runner source tree, the fixture tree and the compiled adapter binaries, so anyone can check which code produced a run. Timestamps are UTC, result rows carry no absolute host paths, and --provenance-level minimal keeps hardware facts while dropping deployment fingerprints.

Repository Layout

License

Apache-2.0. See LICENSE. Upstream task and fixture attributions are retained in THIRD_PARTY_NOTICES.md.