Benchmarks
April 30, 2026 · View on GitHub
HarnessX benchmark adapters — each sub-directory is a self-contained adapter for one evaluation suite.
| Directory | Benchmark | Status |
|---|---|---|
terminal_bench_2/ | Terminal Bench 2.0 | Available |
gaia/ | GAIA | Available |
swebench/ | SWE-bench | Available |
tau2/ | TAU2-Bench | Available |
evoclaw/ | EvoClAW | Ongoing |
locomo/ | LoCoMo | Ongoing |
osworld/ | OSWorld | Ongoing |
Benchmark descriptions
Terminal Bench 2.0
Terminal Bench 2.0 tests agents on 89 bash/file-system tasks executed inside Harbor-managed containers. Tasks cover compilation, debugging, system administration, cryptography, and ML workloads. Use it to evaluate low-level terminal proficiency and tool-use reliability.
GAIA
GAIA tests general agent capability across reasoning, tool use, and web information retrieval. Tasks range from simple factual lookups to multi-step questions requiring file parsing, web search, and cross-source synthesis. Use it to evaluate how well a harness handles open-ended, real-world assistant tasks.
SWE-bench
SWE-bench tests software engineering capability using real GitHub issues from popular Python repositories. Each task asks the agent to produce a patch that resolves the issue and passes the associated test suite. Use it to evaluate coding and debugging ability in a realistic, large-codebase setting.
TAU2-Bench
TAU2-Bench tests tool-augmented agents in user-simulation scenarios. Tasks involve multi-turn interactions with simulated users and structured tool APIs (retail, airline, etc.), evaluating whether the agent can complete user requests correctly while following domain-specific constraints.
Results
Terminal Bench 2.0
89 terminal/coding tasks inside Harbor-managed containers.
| Agent | Model | Tasks | Pass | Score | Runs per task | Notes |
|---|---|---|---|---|---|---|
| HarnessX | claude-opus-4-6 | 89 | 56 | 63.0% | 1 (k=1) | OpenSandbox |
| Claude Code | claude-opus-4-6 | 89 | — | 58.0% ± 2.9 | 5 (k=5) | tbench.ai |
| HarnessX | claude-haiku-4-5 | 89 | 28 | 31.5% | 1 (k=1) | OpenSandbox |
| Claude Code 2.0.31 | claude-haiku-4-5 | 89 | — | 27.5% ± 2.8 | 5 (k=5) | tbench.ai |
HarnessX was run once per task (k=1); the official leaderboard uses k=5 and reports mean ± confidence interval. Direct comparison is not strictly valid, but the scores are closely aligned.
TAU2-Bench
Multi-turn customer-service simulation across three domains. Agent: Sonnet 4.5 + Extended Thinking (budget: 62976 tokens). User simulator: GPT-5.2. The w/o HarnessX column is the published tau2 leaderboard score for the same model.
| Domain | Tasks | Pass | Pass rate | w/o HarnessX | Notes |
|---|---|---|---|---|---|
| telecom | 114 | 98 | 86.0% | 84.9% | base split |
| airline | 50 | 37 | 74.0% | 72.0% | |
| retail | 114 | 81 | 71.1% | 72.4% | StopGuard enabled |
StopGuard strips premature
###STOP###tokens from GPT user-simulator confirmation messages, recovering tasks that would otherwise fail before the agent executes the write tool (~8 pp on retail without it).