WindTunnel compares WebMCP with other ways browser agents interact with websites. It runs the same tasks on the same sites and measures success rate, execution time, token usage, and cost.

Quick start

Just want the results? Jump to them.

Run the harness for free (Node 20.11+, no key, no Docker):

npm ci
WT_FAKE_LIFECYCLE=1 npm run bench    # no LLM — scores 0/7 by design, just proves it runs

Run the real benchmark — needs Docker, Linux or macOS, and a key (setup):

npx playwright install chromium
export ANTHROPIC_API_KEY=sk-ant-...
npm run bench -- --arms wm-claude,cu-claude --budget 2

Runs the 7 tasks on the three lightweight sites once each, same model on two interfaces — WebMCP against screenshots. 14 attempts, well under $1; --budget stops launching further attempts once accumulated spend exceeds it.

Background: Four ways to operate a website

A browser agent can operate a website through four main interfaces:

1. Screenshots (computer use) — it reads rendered images of the page and acts by coordinate.
2. Page structure — it reads the page's DOM and accessibility tree.
3. Code execution — the model writes browser-automation code (Playwright) that reads and drives the page; OpenAI's recommended mode for GPT-6 Astra.
4. WebMCP — the website exposes direct actions (add_to_cart(id), book_slot(time)) for the agent to call. (What is WebMCP?)

Results

Canonical board v1.2: 2026-09-18 — 21 configurations × 49 tasks across 8 sites × 3 attempts = 3,087 attempt rows and 1,029 majority verdicts, with a 600s per-attempt cap. What changed since v1.1: CHANGELOG.md.

Ten configurations solve 49/49 tasks: all nine WebMCP configurations and GPT-6 Astra on code execution, OpenAI's recommended computer-use mode. Raw task-solve rate therefore does not separate WebMCP from the best screen-driving configuration; cost and time do.

New: Jev + Mercury 2.5 solves 49/49 tasks with WebMCP and 25/49 with ultrafast DOM controls, without WebMCP. WebMCP leads the composite score and is 1.7× faster for this pair. The page setup costs less per median attempt ($0.0008 vs. $0.0011), but succeeds less often.

Five configurations pass every attempt (147/147): Gemini 3.6 Flash, Sonnet 5 (native and Stagehand v4) and Claude Opus 5 via WebMCP, and GPT-6 Astra via code execution. For the same model, Astra's code-execution attempts cost $35.53 total vs. $3.33 for native WebMCP (10.7×), while median agent time was 16.4s vs. 6.3s.

_{Regenerate with node scripts/readme-charts.mjs — it reads results/canonical and fails if any label would overflow its column.}

Canonical artifacts — CSV, run JSON, provenance, changelog, and interactive explorer.

Configuration	Interface	Solved	Attempts	Turn-cap	Median cost	Median tokens	Median s
Jev + Mercury 2.5	WebMCP	49/49	141/147	0	$0.0011	9,793	3.2
GPT-5.6 Luna · native	WebMCP	49/49	146/147	0	$0.002	2,596	5.7
Gemini 3.6 Flash · Stagehand v4	WebMCP	49/49	146/147	0	$0.004	4,371	8.0
Gemini 3.6 Flash · native	WebMCP	49/49	147/147	0	$0.004	4,453	7.2
Sonnet 5 · native	WebMCP	49/49	147/147	0	$0.009	5,172	6.8
Sonnet 5 · Stagehand v4	WebMCP	49/49	147/147	0	$0.010	5,161	8.1
GPT-5.6 SOL · native	WebMCP	49/49	145/147	0	$0.012	2,573	9.3
Claude Opus 5 · native	WebMCP	49/49	147/147	0	$0.014	4,770	9.8
GPT-6 Astra · native	WebMCP	49/49	146/147	0	$0.017	2,575	6.3
GPT-6 Astra	code execution	49/49	147/147	0	$0.119	10,982	16.4
Sonnet 5	DOM + vision	48/49	145/147	1	$0.210	64,424	29.3
GPT-5.6 SOL	computer use	46/49	134/147	12	$0.063	16,235	27.3
GPT-5.6 Luna	computer use	45/49	134/147	11	$0.017	20,914	18.3
Claude Opus 5	computer use	45/49	134/147	27	$0.139	47,141	50.4
GPT-6 Astra	computer use	45/49	135/147	11	$0.261	20,560	20.8
Gemini 3.6 Flash	computer use	43/49	130/147	22	$0.020	23,857	33.7
GPT-5.6 Luna	DOM + vision	43/49	130/147	0	$0.033	29,561	19.8
Sonnet 5	a11y tree	42/49	128/147	29	$0.038	10,762	37.5
GPT-5.6 Luna	a11y tree	40/49	119/147	38	$0.020	18,517	16.0
Sonnet 5	computer use	39/49	119/147	41	$0.070	57,701	31.7
Jev + Mercury 2.5	DOM (ultrafast)	25/49	76/147	19	$0.0008	13,892	5.4

_{Attempts are successful attempts out of 147; Turn-cap counts attempts that used every turn of their model-turn budget (whether or not the final turn produced an answer). Medians are rounded by the chart generator (JavaScript toFixed; an exact 6.85 s prints as 6.8 s). Median tokens are total processed: uncached input + cache reads + cache writes + output. The table reports tasks solved by a majority of three attempts; infrastructure rows are excluded.}

_{Headline ranges compare each WebMCP configuration with the median of all non-WebMCP configurations: 24.03s, $0.05024, and a composite score of 64.4. Each configuration has equal weight; time and cost include failed attempts.}

Turn budgets are a material separator. None of the 1,323 WebMCP attempts hit the turn budget, versus 211 of 1,764 screen-driving attempts. Matched like-for-like — each model's native WebMCP run against its own computer-use run, 882 attempts per side — the count is 0 vs. 124. The screen-driving arms are given roughly 3× larger budgets because a screenshot agent needs about three model turns per journey step.

Model snapshot reporting has a harness limitation. Stagehand and browser-use cannot report the exact model snapshot they were served, so those rows use snapshot_source: unavailable:<harness>. The configured model is confirmed out of band with scripts/verify-model.mjs.

How the advantage scales with journey length

The following table covers the six native WebMCP/screenshot model pairs; it excludes Jev's page-control setup.

Tier	Tasks	WebMCP solved	Computer use solved	Cheaper	Faster	Lighter
Answer (1–2 steps)	21	126/126	107/126	6.0×	3.5×	6.0×
Action, short (3–5)	16	96/96	87/96	6.4×	3.3×	5.6×
Action, long (6–10)	8	48/48	47/48	11.5×	5.2×	11.2×
Sensitive action (8–15)	4	24/24	22/24	12.7×	7.8×	12.7×

On long journeys the efficiency gap widens sharply. Across the 12 long and sensitive-action tasks, WebMCP solved 72/72 model-task cells vs. computer use's 69/72 (96%), using 12.3× lower cost, 5.8× less agent time, and 13.0× fewer tokens. Until board v1.1 the sensitive-action row was a deliberate coverage boundary: the store's WebMCP tools handed guest checkout back to the page before the final purchase (task md-8), so no WebMCP configuration could complete it while screen-driving agents could. v1.1 added a complete_checkout tool and re-measured that cell — see CHANGELOG.md.

_{How these numbers are computed: docs/SPEC.md.}

How it works

Every task runs on the same site, from the same seeded state, scored by the same check, for every method — so a difference in outcome isn't the site or the data. What differs is the whole method configuration: interface, model, framework, and turn budget.

• Real, self-hosted applications. Production open-source apps, not synthetic pages. Each runs locally in Docker, pinned to a fixed version.
• Outcome-based scoring. No human or model judges a run: WindTunnel inspects the resulting application state — is the item in the cart, does the appointment exist — and records pass or fail, plus what the attempt cost.
• Integrity. Checked values are generated fresh from a seed, state-changing tasks are scored by inspecting the application, and published final answers are redacted. The Jev release includes redacted transcripts; the combined result rows stay compact. Memorization risk and its limits: docs/SPEC.md.
• Scorer correction. Independent review found that the canonical merge had been built before corrected predicates were applied. Re-scoring fixed 19 false negatives — 19 promotions and 0 demotions — and the corrected rows are marked in the canonical CSV.

The sites

Eight applications across public and authenticated pages and different stacks — six live targets, two read-only controls.

Site	Type	Pulled from (upstream)	Exercises
nextjs-starter-medusa	online store	medusajs/nextjs-starter-medusa	browse → cart → checkout
hi-events	events / ticketing	HiEventsDev/Hi.Events	browse → ticket checkout
easyappointments	appointment booking	alextselegidis/easyappointments	booking; admin (auth)
learnhouse	course platform	learnhouse/learnhouse	catalog; authoring (auth)
idurar-erp-crm	B2B CRM	idurar/idurar-erp-crm	record CRUD (auth)
directory-9d8	business directory	9d8dev/directory	search / filter
tailwind-nextjs-blog	blog (control)	timlrx/tailwind-nextjs-starter-blog	read-only content
bulletproof-react	web app (control)	alan2207/bulletproof-react	read-only; auth

The tasks

49 benchmark tasks across the eight sites, plus 10 calibration tasks for cost measurement. Four difficulty tiers by journey length:

Tier	Steps	Example
Answer	1–2	"What is the price of X?"
Action (short)	3–5	"Add two of X to the cart."
Action (long)	6–10	"File a ticket, assign it, set its priority from the report."
Sensitive action	8–15	"Book the cheapest available slot and confirm."

Each task is attempted N times (3 by default). A task is solved when a majority of attempts pass; the headline is solved ÷ total, reported next to median time, tokens, and cost.

Each task also carries a per-interface turn budget — a screenshot agent needs ~3 turns per step, a tool-calling agent ~1 (defaults: docs/SPEC.md).

The methods

The board spans 13 implementations and 21 configurations: nine with WebMCP and twelve using screenshots, page structure, or code execution. Models include Sonnet 5, Opus 5, GPT-5.6 Luna, GPT-5.6 SOL, GPT-6 Astra, Gemini 3.6 Flash, and Jev + Mercury 2.5. Every configuration uses the same 49 tasks and scoring.

Jev makes decisions; Mercury writes arguments or field values and the final answer. The WebMCP and ultrafast page setups use separate frozen harnesses, so this is a comparison of complete setups. Their runner code and run instructions are published separately from the standard CLI. Versions, accounting and redacted transcripts.

Cost

Where the tokens go. How much an agent reads each turn is set by the interface:

• Screenshots (computer use) — a full page image every turn: thousands of tokens each, refreshed nearly every step.
• Page structure (DOM / accessibility tree) — the page's text every turn, the whole page re-read each step. DOM + vision sends the text and a screenshot, and is often the heaviest configuration of all.
• WebMCP — a short list of tool schemas plus small JSON results. No page text, no screenshots. Usually less context.

Task length multiplies it. Page-reading interfaces grow fastest because they re-read the page each step. Across the native model pairs, WebMCP's median cost advantage grows from about 6× on short tasks to 11–13× on longer journeys.

Per-task medians across the current 21-configuration leaderboard (tokens include cache reads and cache writes; pricing detail in docs/SPEC.md):

Interface	Configurations	Median tokens / task	Median cost / task
WebMCP	9	2,573–9,793	$0.0011–$0.017
DOM (ultrafast)	1	13,892	$0.0008
Computer use	6	16,235–57,701	$0.017–$0.261
Code execution	1	10,982	$0.119
Accessibility tree	2	10,762–18,517	$0.020–$0.038
DOM + vision	2	29,561–64,424	$0.033–$0.210

PROJECTED cost per 1,000 task attempts — extrapolated from the 147 observed attempts in each configuration, not an observed 1,000-run experiment:

Interface	Range	Median
WebMCP	$1.37–$22.67	$10.65
Screen-driving	$1.67–$453.68	$120.01

These ranges overlap. Costs use reported usage; unknown usage and excluded infrastructure spend for the new setups are disclosed in their provenance.

What a run costs:

Run	Scope	Ballpark
quick check	--preset smoke --sites lite — 3 light sites, 1 attempt each	under $1
small	--preset lite --sites lite — the lite task set × 3 attempts	$5–10
full paired model (measured additions)	all 8 sites, 49 tasks × 3 attempts × WebMCP + computer use	~$6–70
full canonical leaderboard (measured)	all 8 sites, 49 tasks × 3 attempts × 21 configurations	$281.80

In the former 2026-07-27 reference flight, the three WebMCP methods were ~9% of the bill; historical breakdown: docs/CALIBRATION.md.

Spending less. The levers, cheapest first:

• Fewer sites — --sites lite (3 lightweight sites, no databases).
• Fewer / cheaper methods — WebMCP medians are $0.0011–$0.017/task; computer use and DOM + vision carry most of the cost.
• Fewer attempts — --preset smoke or --n 1 instead of the default 3 (you lose majority voting, so one run decides each task).

Running it yourself

Setup. You need Docker, Node 20.11+, and an API key for the model under test:

npm install
npx playwright install chromium                       # browser for the agents (or set WT_CHROME)
python3 -m venv .venv-browseruse \
  && .venv-browseruse/bin/pip install browser-use==0.12.7   # only for the dom-browseruse method

Real site boots use Linux-oriented capsule tooling. Native Linux is the lowest-overhead host; macOS works through Docker Desktop when GNU tar and coreutils are installed and placed first on PATH (brew install gnu-tar coreutils). Docker Desktop's VM and filesystem layer add boot/reset and wall-clock overhead, so report the host and avoid comparing absolute latency with bare-metal Linux. You can also dry-run the pipeline (WT_FAKE_LIFECYCLE=1) or point it at a site you booted yourself (WT_MANUAL_BASEURL=http://localhost:PORT).

Site profiles size the run to the machine at hand:

profile	sites	needs	runs on
lite	3 lightweight sites, no databases	Node	laptop, CI
core	lite + the online store	+ Postgres	8 GB laptop
categories	one site per category	mixed	16 GB machine
full	all 8 sites	all stacks	16 GB+ worker

Commands. --arms picks the methods; the default (scripted) is a free, no-LLM baseline that only exercises the pipeline:

$\text{bash} \text{npm} \text{run} \text{bench} -- --\text{preset} \text{smoke} --\text{sites} \text{lite} --\text{arms} \text{scripted} # \text{free} \text{pipeline} \text{check}, \text{no} \text{API} \text{key} \text{npm} \text{run} \text{bench} -- --\text{preset} \text{smoke} --\text{sites} \text{lite} --\text{arms} \text{wm}-\text{claude},\text{cu}-\text{claude} # \text{cheapest} \text{paid} \text{run}, 1 \text{attempt} \text{per} \text{task} \text{npm} \text{run} \text{bench} -- --\text{preset} \text{lite} --\text{sites} \text{lite} --\text{arms} \text{wm}-\text{claude},\text{cu}-\text{claude},\text{dom}-\text{browseruse},\text{a11y}-\text{stagehand} # \text{every} \text{task} \text{on} \text{the} \text{three} \text{lite} \text{sites} \times 3 \text{attempts} \text{npm} \text{run} \text{bench} -- --\text{preset} \text{smoke} --\text{sites} \text{lite} --\text{arms} \text{wm}-\text{claude} --\text{seed} 7 --\text{budget} 2 --\text{label} \text{check} \text{npm} \text{run} \text{bench} -- --\text{preset} \text{smoke} --\text{sites} \text{idurar}-\text{erp}-\text{crm} --\text{arms} \text{cu}-\text{openai} --\text{task}-\text{ids} \text{id}-6,\text{id}-6,\text{id}-8 --\text{n} 1 $

Keys. Runs read the provider key straight from your environment:

export ANTHROPIC_API_KEY=sk-ant-...   # cu-claude, dom-browseruse, a11y-stagehand, wm-claude, wm-stagehand
export OPENAI_API_KEY=sk-...          # cu-openai, wm-gpt, code-openai
export GEMINI_API_KEY=...             # cu-gemini, wm-gemini
export WT_SECRET=...                  # optional capsule secret
export WT_SECRET_KEY=...              # optional key used to protect it

A method whose key is absent is skipped with a notice, never an error. --task-ids selects exact task IDs; repeated IDs intentionally repeat a task, which is useful for targeted diagnostics.

Switching models. --model <method>=<model> overrides the model for one method:

npm run bench -- --preset lite --sites lite --arms wm-claude --model wm-claude=claude-opus-5
npm run bench -- --preset smoke --sites lite --arms cu-openai,wm-gpt,code-openai --model cu-openai=gpt-6-astra --model wm-gpt=gpt-6-astra --model code-openai=gpt-6-astra

Computer use needs a computer-use-capable model. Switching provider takes that provider's own method and key, not just a model name.

Repo layout

docs/       design spec and methodology
sites/      sites under test + subset configuration
capsules/   self-contained boot recipes per site (Docker, pinned commits, WebMCP tools)
fixtures/   per-site seed data and runtime patches
goldens/    the reference WebMCP tool implementations, one patch per site
tasks/      task definitions
harness/    the runner (bin/ has the site-boot CLIs)
scoring/    per-task checks and result formats
arms/       interface implementations
results/    finished runs and reports
tests/      harness test suite (npm test)

License

WindTunnel's own code — harness, boot recipes, and WebMCP patches — is Apache-2.0. It does not vendor any site's source tree: each site is cloned from its upstream at a pinned commit and patched locally at run time, so the copyleft (AGPL/GPL) sites run locally only, and Hi.Events' required "Powered by Hi.Events" footer is preserved. A few reference patches modify upstream files; the upstream lines those hunks carry remain under the upstream project's license. Upstreams, licenses, and pinned commits: ATTRIBUTION.md (canonical) and each capsules/<site>/capsule.yaml.