HarnessLab ๐ฌ
April 4, 2026 ยท View on GitHub
A benchmark for evaluating AI agent harness designs โ the first framework that treats the harness as the independent variable.
"When we evaluate 'an agent,' we're evaluating the harness and the model working together." โ Anthropic, Demystifying Evals for AI Agents (2026)
The Problem
Every existing agent benchmark evaluates the combined model+harness system. When LangChain changed only the harness (same GPT-5.2-Codex model) and jumped from 52.8%โ66.5% on Terminal-Bench 2.0, or when fixing harness bugs alone took Opus 4.5 from 42%โ95% on CORE-Bench, it became clear: the harness is a first-order variable that no benchmark isolates.
What HarnessLab Does
HarnessLab holds the model and task fixed and systematically measures the causal contribution of each harness module.
Three Novel Metrics
| Metric | What It Measures |
|---|---|
| Harness Contribution Matrix (HCM) | ฮทยฒ effect size: what % of performance variance is explained by each module? |
| Harness Generalization Score (HGS) | Cross-domain rank correlation: does a harness that works for coding also work for web tasks? |
| Durability Curve | Performance by trajectory depth (steps 1โ100+): where does the "cliff" occur? |
Architecture
6 Harness Modules ร 4-5 Implementations = 30 Variants
| Module | Implementations |
|---|---|
| Context Manager | NullContext, SlidingWindow, LLMSummarizer, ObservationMask, HybridAdaptive |
| Retry Policy | NoRetry, SimpleRetry, ExponentialBackoff, InformedBacktrack, MultiPathSearch |
| Tool Selector | FullToolset, ProgressiveDisclosure, SemanticRouter, MinimalToolset, HarnessVerified |
| Memory System | Stateless, FilesystemState, EpisodicMemory, SemanticSearch, WorkspaceManifest |
| Output Validator | NullValidator, SchemaValidator, SelfCritique, SeparateVerifier, DSAPVerifier |
| Orchestration | SingleAgent, FixedPipeline, DynamicOrchestration, SelfEvolution, SubAgentDelegation |
120 Tasks ร 4 Domains ร 3 Difficulty Levels
| Domain | Easy (10-20 steps) | Medium (20-50 steps) | Hard (50+ steps) |
|---|---|---|---|
| Coding | 10 | 10 | 10 |
| Web | 10 | 10 | 10 |
| Research | 10 | 10 | 10 |
| Computation | 10 | 10 | 10 |
Fractional Factorial Experimental Design (Resolution V)
- 64 core configurations (covers all main effects + 2-factor interactions)
-
- 32 targeted follow-up runs
- Total: 96 harness configurations evaluated
Quick Start
pip install harnesslab
# Run a mini experiment (mock executor, no API key needed)
harnesslab run --design mini --tasks-per-domain 2
# Launch the interactive dashboard
harnesslab dashboard
# Analyze results
harnesslab analyze results/exp_001/results.jsonl
# List available tasks
harnesslab list --tasks --domain coding
Expected Key Findings
Based on the research literature (NLAH, AutoHarness, Meta-Harness, Efficient Benchmarking):
- Context Manager + Memory System explain >60% of performance variance
- Durability cliff at ~step 50-60 for most configurations; ObservationMask + FilesystemState flatten it dramatically
- ObservationMask + FilesystemState is super-additive (combined effect > sum of individual effects)
- InformedBacktrack + SeparateVerifier is sub-additive (redundant recovery mechanisms)
- 5-10 "universal" harnesses generalize across domains (HGS > 0.6)
- Minimal Viable Harness (3 modules) achieves ~85% of full benefit at ~40% of cost
Project Structure
harnesslab/
โโโ harnesslab/
โ โโโ modules/ # 6 module families ร 5 implementations
โ โ โโโ context/ # NullContext, SlidingWindow, LLMSummarizer, ObservationMask, HybridAdaptive
โ โ โโโ retry/ # NoRetry, SimpleRetry, ExponentialBackoff, InformedBacktrack, MultiPathSearch
โ โ โโโ tools/ # FullToolset, ProgressiveDisclosure, SemanticRouter, MinimalToolset, HarnessVerified
โ โ โโโ memory/ # Stateless, FilesystemState, EpisodicMemory, SemanticSearch, WorkspaceManifest
โ โ โโโ validation/ # NullValidator, SchemaValidator, SelfCritique, SeparateVerifier, DSAPVerifier
โ โ โโโ orchestration/ # SingleAgent, FixedPipeline, DynamicOrchestration, SelfEvolution, SubAgentDelegation
โ โโโ tasks/ # 120 benchmark tasks + graders
โ โโโ experiments/ # Fractional factorial design + mock/real executor
โ โโโ analysis/ # HCM, HGS, Durability Curve
โ โโโ dashboard/ # Plotly Dash interactive leaderboard
โ โโโ cli.py # Click CLI
โ โโโ config.py # YAML config system
โโโ scripts/ # run_mini_experiment.py, run_dashboard.py
โโโ tests/ # Full test suite
โโโ results/ # Experiment outputs (gitignored)
โโโ pyproject.toml
Related Work
| Paper | What It Does | HarnessLab's Contribution |
|---|---|---|
| NLAH (Pan et al., 2026) | Ablates 6 modules on SWE-bench | Extends to 4 domains, adds interaction effects, cost model, durability |
| AutoHarness (Lou et al., 2026) | Synthesizes harnesses for games | Provides evaluation target for synthesized harnesses |
| Meta-Harness (Lee et al., 2026) | Black-box harness optimization | Explains why optimized harnesses work (module attribution) |
| Efficient Benchmarking (Ndzomga et al., 2026) | Shows scaffold = major confound | Quantifies and attributes the confound |
Citation
@software{harnesslab2026,
title={HarnessLab: A Benchmark for Evaluating Agent Harness Designs},
year={2026},
note={First benchmark treating harness as independent variable},
url={https://github.com/harnesslab/harnesslab}
}
License
MIT