SIP-Bench

May 7, 2026 · View on GitHub

SIP-Bench Figure

SIP-Bench turns existing benchmarks into self-improvement benchmarks.

It adds a shared longitudinal protocol on top of existing benchmarks so you can measure:

whether the agent improved on held-out tasks,
whether it retained performance on tasks it already knew,
what interaction, compute, and runtime cost that improvement required.

Most benchmark reports stop at a single score. SIP-Bench is for the questions that come after that score:

Did improvement on new tasks hurt replay performance?
Did the gain persist at a later checkpoint?
Was the gain operationally expensive?
Did the run fail for infrastructure reasons rather than capability reasons?

Protocol surface:

T0 / T1 / T2 checkpoints,
replay / adapt / heldout / drift splits,
normalized runs.jsonl and summary.jsonl records,
protocol metrics such as FG, BR, PDS, IE, and NIS,
first-class tracking of failed executions and retry provenance.

Adapter-First

SIP-Bench is designed around one idea: benchmark-specific logic belongs in adapters, while SI evaluation logic stays shared.

That gives you:

benchmark-specific logic stays inside an adapter,
the protocol contract stays benchmark-agnostic,
imported results are normalized into shared runs.jsonl and summary.jsonl schemas,
the same T0 / T1 / T2 and replay / adapt / heldout / drift logic can be reused across very different task worlds.

If you can write an adapter for a benchmark, you can evaluate that benchmark under the SIP protocol.

What You Get

Current repository surface:

an adapter layer for wrapping new benchmarks into the same SI protocol,
first-class support for SkillsBench, EvoAgentBench, and tau-bench,
split planning and suite orchestration,
normalized runs.jsonl and summary.jsonl outputs,
aggregation, schema validation, and evidence-gate tooling,
tracked dry-run and real-artifact examples.

Benchmarks

Benchmark path	Status	Notes
`SkillsBench oracle`	primary	real execution and tracked suite artifacts
`EvoAgentBench`	primary design target	demonstrates the protocol on self-improvement-heavy agent workflows and motivates the adapter-first architecture
`tau-bench historical`	supplementary	import-only path with tracked suite artifacts
`tau-bench live`	optional	requires provider credentials
`SkillsBench prepared external`	experimental	useful for task-preparation and path validation

Why Clone This

If you already have a benchmark and want SI evaluation instead of a single-shot score, this repository gives you the reusable pieces:

write an adapter,
map the benchmark into replay / adapt / heldout / drift,
import normalized run records,
compute SI-focused metrics under the same protocol as every other adapter.

That is the real product surface of SIP-Bench.

Quickstart

Minimum local checks:

python3 -m venv .venv
. .venv/bin/activate
python -m pip install -e .
python3 scripts/run_release_checks.py
python3 -m unittest discover -s tests -p "test_*.py"

Build a protocol summary from tracked sample runs:

python3 scripts/aggregate_metrics.py \
  --runs results/dryrun/sample_runs.jsonl \
  --out /tmp/sip_summary.jsonl

Import a fixture-format SkillsBench Harbor job:

python3 scripts/run_eval.py import-skillsbench-job \
  --job-dir tests/fixtures/skillsbench_harbor_job_sample \
  --out /tmp/skillsbench_job_runs.jsonl \
  --benchmark-split smoke \
  --phase T0 \
  --path-type oracle \
  --seed 21 \
  --registry tests/fixtures/skillsbench_registry_sample.json \
  --agent-version fixture-import \
  --benchmark-version skillsbench-harbor-fixture

Validate the imported records:

python3 scripts/validate_records.py \
  --data /tmp/skillsbench_job_runs.jsonl \
  --schema runs

Artifacts

If you want concrete outputs before reading code, start here:

Minimal Proof

This repository only needs to prove one thing: protocol-level evaluation reveals information a single post-adaptation score hides.

The smallest tracked proof is results/dryrun/summary.jsonl:

held-out performance improves from T0 to T1,
replay performance regresses at the same time,
the gain softens again by T2,
the improvement has explicit interaction cost.

That is enough to justify the project.

Execution-backed proof:

Design proof:

SkillsBench shows a real execution-backed path,
EvoAgentBench shows that the same protocol ideas can target a very different self-improvement workflow,
tau-bench shows that the same normalization layer also works for import-oriented external environments.