Eval framework guide

VT Code ships a first-class evaluation framework (vtcode-eval) that measures how reliably an agent completes tasks: each task runs autonomously in an isolated git worktree, the claimed outcome is verified with environment probes (the agent's own report never counts as success), and results aggregate into pass@k / pass^k metrics split by capability and regression categories.

This is the "verifiable harness" wedge: because suites are plain JSON and reports carry run metadata, VT Code can publish reproducible benchmark numbers for any model route — including local models — something closed competitors structurally cannot match.

One-command usage

# Prerequisites: [automation.full_auto] enabled and workspace trust granted.
vtcode eval --suite my-suite.json                     # markdown report to stdout
vtcode eval --suite my-suite.json --format json --output report.json
vtcode exec eval --suite my-suite.json                # long form, identical behavior

What --format json emits

A reproducible-run envelope (schema_version 1):

Field	Purpose
harness / harness_version	Which VT Code produced the numbers.
provider / model	The model route under test.
suite_path / suite_sha256	Pins the exact task definitions to the metrics.
report	Per-task and aggregate pass@k / pass^k, cost, duration, trace summaries.

Publishing "verified on these models with these scores" means attaching this envelope plus the suite file; the SHA-256 digest ties the numbers to the tasks. Per-attempt traces (turn/tool/error counts, latency, token usage — no transcript content) land in .vtcode/eval/traces/ for auditing.

Authoring suites

Suites are JSON: tasks with a prompt, category (capability / regression), optional timeout_secs, and verify_commands. Each verify command runs as raw argv in the worktree (no shell): write them as space-separated tokens without quotes, pipes, or &&; every command must exit zero for the attempt to pass.

Two suite families live in crates/codegen/vtcode-eval/evals/ (see its README):

• regression suites that pin VT Code's own documented behaviors, and
• smoke-workspace-basics.json, a repository-agnostic capability baseline.

For repo-specific credibility, author a suite against your own repository's hot paths and run it per model — the harness, not the model vendor, owns the numbers.

Related

• docs/development/preview-budget-blocked-replan.md — an example of a suite-derived regression doc.
• vtcode benchmark — the separate SWE-bench-style runner for single tasks with JSON reports.