Study 2026-08
August 24, 2026 · View on GitHub
Tested Second Brain against the seven published Harness-Bench harnesses, on three models
x two modes, 106 tasks x 1 trial per cell. Everything here is derived from the
read-only snapshot in ../raw-snapshot/ plus the archived published data in
published/, and can be rebuilt with build_comparison.py.
The entire Second Brain framework can be found here.
Files
| file | what it is |
|---|---|
by_harness.csv | primary table — one row per harness x model (Second Brain split by mode) |
by_model.csv | same rows, grouped so harnesses compare within each model |
second_brain_trials.csv | per-trial detail, both scoring variants (1,272 rows) |
second_brain_cost.csv | cost, Second Brain only |
oracle_variants.json | 012/013/014 dual-scored, upstream vs PR #7 |
regraded/ | the 100 re-graded trials, one JSON each, plus regrade_summary.json |
manual_overrides.json | per-trial overrides, each carrying its evidence |
published/ | the fetched rival data + SOURCE.md provenance |
harness_bench.sqlite | full export: trials, model_calls, tool_calls, oracle_checks |
Headline
Ranked by combined score, Second Brain in YOLO mode is 2nd of 7 on DeepSeek-V4 Flash, 2nd of 7 on Kimi-K2.5, and 3rd of 7 on Qwen-3.6 Plus — behind nanobot everywhere, and behind hermes on Qwen only. Lockdown mode costs 8–12 points and lands mid-field.
| model | Second Brain (yolo) | rank | leader |
|---|---|---|---|
| deepseek-v4-flash | 73.2 | 2 / 7 | nanobot 74.8 |
| kimi-k2.5 | 74.0 | 2 / 7 | nanobot 76.3 |
| qwen3.6-plus | 74.3 | 3 / 7 | nanobot 79.2 |
Codex (80.4) is gpt-5.4 only and is not a like-for-like comparison; it is included as an indirect reference.
The gap to nanobot is process, not work done. Completion is within 1.4 points on all three models (80.4 vs 81.8, 80.9 vs 81.8, 82.9 vs 83.5); process is 1.3–6.4 points behind. Second Brain also spends far more tokens per task than nanobot (1.8x–2.9x), though prompt caching makes it cheap in dollars where the provider supports it.
Two columns for the combined score
combined is each harness's number as published / as measured.
combined_judge_adjusted exists because a failed LLM judge is scored as a
perfect process grade — upstream falls back to process_effective = 1.0 and
security = 1.0, collapsing combined to the raw outcome.
Second Brain's judge failures were re-graded away (100 trials, see below), so its
failure rate is now ~0%, while the rivals' published numbers still contain that
free credit at rates of 0.9–13.2%. Leaving it there would bias the comparison
in our favour. The adjusted column imputes each pair's own mean observed
process onto its unjudged trials — never ours — so no behaviour of ours leaks
into a rival's number. It moves rivals down 0.1–1.4 points and leaves Second
Brain essentially unchanged. No ranking changes. Prefer the adjusted column
for like-for-like claims; report combined when quoting the paper.
What was repaired (Part I)
100 judge failures re-graded, 99 recovered. Upstream truncates the judge
payload with a raw string slice, json.dumps(trace)[:24000], which cuts
mid-string inside the JSON. One failing trial's payload was 94,010 chars cut to
25.5%, ending mid-token. Handed a malformed blob, the judge continues the
transcript instead of grading; the verdict then either fails to parse or parses
into a stray object with no scores key — the silent case, which nothing flags.
Failed gradings had 4.4x the median transcript of successful ones (1,085 KB vs
249 KB), and lockdown failed far more than yolo (74 vs 26).
The fix trims the trace data before serialization instead of slicing the serialized output, at upstream's unchanged 24,000-char budget. Result: all 57 silent, all 16 parse errors and 26 of 27 timeouts recovered. Mean new process score 0.788 where every one had been credited 1.0. Mean combined delta -0.1263, maximum +0.0000 — not one trial improved.
All 100 re-grades resolved the identical rubric source as the original in-container runs, so the only procedural difference is the payload trim.
Two hard oracle crashes fixed.
010-office-docs(qwen/lockdown) re-run; it had crashed on a corrupt.docx. Now 0.4444 outcome / 0.75 process. The run's completion score moved 0.737484 -> 0.741677, exactly 0.4444/106, confirming nothing else changed.028-email-thread-merge(deepseek/yolo) reconciled against a prior offline re-score. Verified rather than trusted: re-running the patched oracle on the preserved workspace returns 0.8182, reproducing the recorded value exactly.
The split benchmark commit is a non-issue. The two DeepSeek runs used
1025086, the other four 2274ee0. The only difference is the 028 timeline fix,
and it is score-preserving: run 119ae5 stores 0.7273 from the unpatched
oracle, and the patched oracle returns 0.7273 on the same workspace.
Scoring variants
paper-comparable is the headline and the only variant that may be set against a
published figure. pr7-corrected is research interest only.
Tasks 012/013/014 all resolve ground_truth.json as w.parent.parent, which is
never the task directory, so gt silently becomes {}. Upstream is unfixed
(PR #7 open), so the paper was produced with the bugs present.
| task | uncorrected effect | headline uses |
|---|---|---|
| 012-doc-synthesis | pinned at exactly 0.75 regardless of output | upstream (0.75) |
| 013-image-edit | none — defaults duplicate ground truth exactly | either (identical) |
| 014-task-decomposition | topic coverage never scored | upstream |
Both validation predictions hold: upstream 012 returns exactly 0.75 in all six runs, and 013 is byte-identical across variants everywhere. The two variants differ by only 0.1–0.3 combined points, and uncorrected 012 slightly favours Second Brain — the deviation never inflated us in the other direction.
Protocol deviations — disclose these with any quoted figure
- 012 oracle normalization — our runs corrected 012; the headline puts it back to upstream, so this no longer applies to the reported number.
- localhost mock tasks — 5 tasks pointed at loopback rather than a public address. Residual uncertainty ~1 point.
- The vision quality LLM never ran. Tasks 008 and 013 carry
outcome_llm_weight: 0.9, but it is hard-disabled in our runs (HARNESSBENCH_SKIP_ORACLE_QUALITY_LLM=1), verified: 0 of 634 trials have a non-nulloracle_quality. The published harnesses did run it, and it cost them a mean 0.275 combined on each of those two tasks. Worth +0.52 combined points to us across the 106-task mean. This is the largest remaining deviation and it favours Second Brain. - Mixed grading procedure. 534 trials were graded from upstream's malformed payload slice; 100 were re-graded from a structurally-trimmed payload. Scoping the fix to the failures only was a deliberate choice; the corpus therefore carries two procedures.
Netting 3 against the field, a conservative reading of the DeepSeek result is Second Brain ~72.7 vs nanobot 74.2 — still 2nd, still a statistical near-tie on completion.
Cost
Cost is Second Brain only: rival prices are unknown, and these Atlas Cloud rates measure the vendor rather than the harness.
| model | mode | cache hit | $/task | $/score point |
|---|---|---|---|---|
| deepseek-v4-flash | yolo | 81.9% | 0.0116 | 0.0168 |
| deepseek-v4-flash | lockdown | 81.5% | 0.0209 | 0.0340 |
| kimi-k2.5 | yolo | 77.9% | 0.0467 | 0.0668 |
| kimi-k2.5 | lockdown | 72.8% | 0.0942 | 0.1514 |
| qwen3.6-plus | yolo | none | 0.0918 | 0.1310 |
| qwen3.6-plus | lockdown | none | 0.1207 | 0.2045 |
Qwen reports no cache at all through Atlas Cloud — cached_input_tokens is
NULL, not zero — which is why it costs 8x DeepSeek for a similar score. The same
shows in the published data, where most qwen pairs have cache_read_tokens of 0.
Cache-% is therefore not comparable across models.
models.json has no cache-write price field, so Qwen's $0.406/M cache-out is
unmodelled. Harmless while Qwen caches nothing, but it would understate cost if
that changes.
Caveats on the token columns
Token accounting differs by harness. For every harness except codex,
total = input + output + cache_read with input_tokens uncached-only; for
codex, input already includes cache reads. Second Brain follows the codex
convention. The CSVs normalize all of it to billed input + output, which
reproduces each rival's published total exactly.
turns and tool_calls are populated for Second Brain only — the published data
contains no turn or tool-call counts for any rival, so those cells are blank
rather than estimated.