OB-1 Build Progress

June 25, 2026 · View on GitHub

Roadmap from ob1-plan.html §10. Ship the Solo workhorse first; add the multi-mind modes only once the foundation + eval harness exist (so each mode can be proven, not assumed — R5).

Phase 0 — Solo core · 🟡 in progress

Project scaffold (TS + Bun, tsconfig, gitignore)
CLI/REPL with slash-commands + mode/phase prompt
Provider gateway — multi-provider: Anthropic + OpenAI-compatible (OpenRouter/OpenAI/local) — R5/R7
Gated ReAct loop (Plan/Act + approval gate)
Tool/action system: read/list/write/edit/bash + memory tools
Diff edit engine (search/replace, uniqueness check) — R4
End-to-end verified against a live model (Qwen 3.6 Plus via OpenRouter): tool-calling, approval gate, Plan-mode blocking, and memory persistence all confirmed working
Streaming output — SSE for both providers, live token output, idle-timeout + retry (see Phase 10)
Live token/cost meter in the statusline — R1 (Ink TUI <StatusBar>: live in/out/cached tokens + ~$cost, see Phase 11)

Phase 1 — Context & memory engine · 🟡 in progress

† sqlite-vec / ‡ tree-sitter — now ACTIVE on Bun (2026-06-21). Both libraries the research recommends are wired in and run on Bun (and Node); the pure-TS implementations remain as an automatic fallback so a host without the prerequisites never breaks. The earlier "neither loads on Bun" note was a version/config problem, not a hard limit (re-probed). ‡ tree-sitter: web-tree-sitter@0.20.8 + tree-sitter-wasms grammars (the matched 0.20 ABI — newer web-tree-sitter broke the wasm ABI/API); src/context/treesitter.ts parses each file and falls back to the regex extractor when a grammar is missing or a parse fails (OB1_TREESITTER=0 disables). † sqlite-vec: Bun's bundled sqlite omits dynamic extension loading, so Database.setCustomSQLite() repoints bun:sqlite at a capable libsqlite3 (Homebrew on macOS, a system lib on Linux), then sqlite-vec runs a vec0 cosine KNN; falls back to the pure-TS brute-force cosine when no capable lib is found (OB1_VEC=0 disables). The embedder is pluggable: set OPENAI_API_KEY for real semantic embeddings; otherwise a local hashed-subword baseline keeps it fully offline.

Phase 2 — Tools, skills & safety · 🟡 in progress

MCP client (stdio + http (Streamable HTTP) + legacy SSE) JSON-RPC, server manager + namespaced tools — verified — R6/R7
On-demand Skills — descriptions in prompt, bodies lazy-loaded via use_skill; /skills, /skill — R1
OS sandbox: Seatbelt (macOS) + bubblewrap (Linux) for run_bash, /sandbox modes — verified — R7

Phase 3 — Multi-mind runtime · 🟢 core done

Parallel worker spawner with isolated context windows (read-only tools) — R5
Result compression back to orchestrator (synthesizer pass)
fanout orchestrator (best-of-N skeleton) + /fanout — verified live (3 workers + synthesis)
Concurrency-capped runParallel, ordered results, per-worker token accounting

Phase 4 — Fusion · 🟢 done

Best-of-N fan-out with optional multi-model routing (OB1_FUSION_MODELS) — R5
Auto-scoring of candidates: in-process TS/JS syntax (Bun.Transpiler), Python py_compile, or a custom objective check (OB1_FUSION_CHECK, $OB1_FILE)
Score-grounded synthesis (synthesizer told which candidates PASS/FAIL)
/mode fusion routes every task through Fusion — verified live (single + multi-model)

Phase 5 — Council · 🟢 done

Author drafts → lens-specialized critics (correctness / safety / simplicity) review in parallel, each returning a VERDICT: BLOCK|OK — R5
Revise rounds: author revises against blocking critiques; early-stop once a round raises no blockers; round budget via OB1_COUNCIL_ROUNDS (default 2)
Arbiter ships the final draft with an ACCEPT|REVISE verdict, grounded in outstanding critiques
Configurable lenses via OB1_COUNCIL_LENSES; /mode council + one-shot /council <task>
Deterministic smoke (council-smoke.ts): verdict parsers + full flow via injected runner (early-stop, round cap, accept/revise) — no API key

Phase 6 — Personas · 🟢 done

Persona Former dynamically picks the expert roles a task needs (lead role first); it may pick exactly ONE — collapsing to Solo — when a panel isn't warranted (the R5/R6 honesty caveat)
Shared blackboard (durable state, not message-passing — R5): personas post findings / constraints / open questions; later rounds read the whole board and refine (OB1_PERSONAS_ROUNDS, default 2)
Lead synthesis: the first-formed persona integrates the board into one final plan/answer
/mode personas + one-shot /personas <task>; panel size capped via OB1_PERSONAS_MAX
Deterministic smoke (personas-smoke.ts): persona parsing + blackboard + full former→panel→board→lead flow, collapse-to-Solo, and board-awareness — no API key

All three opt-in multi-mind modes (Fusion · Council · Personas) now run on the one Phase 3 runtime.

Phase 7 — Eval harness · 🟢 done

SWE-bench-shaped tasks with objective checks (exit 0 = PASS via $OB1_FILE; no model-judge); built-in sum-evens + slugify, extensible from eval/tasks/*.json — R5
runEval runs every (task × mode × trial), extracts code, and grades it with the task's check
Compute-matched verdict (computeMatched): each mode's pass rate vs Solo@k, where k = mode tokens / Solo tokens and Solo@k = 1-(1-p_solo)^k (pass@k) — a mode is "justified" ONLY if it beats that equal-token baseline; otherwise default to Solo (the R5 honesty gate)
/eval [modes…] in the REPL + standalone scripts/eval.ts (OB1_EVAL_TRIALS for real Solo@k)
Plain-text report with per-mode pass% · tokens · k× · solo@k% · verdict
Deterministic smoke (eval-smoke.ts): the built-in checks really discriminate correct vs wrong code through the real grader, and the Solo@k math is exact on hand-computed values — no API key

Phase 8 — Research-backed mode improvements · 🟢 done

Grounded in a literature review (see commit messages for citations); the consistent finding — and our own eval's — is that the real levers are objective verification and difficulty-aware allocation, not more agents.

Fusion selection layer: pairwise knockout tournament with a position-debiased judge (LLM-Blender) · agreement/Universal-Self-Consistency fallback · F4 verify-the-synthesis + revert-on-regression · optional Mixture-of-Agents refine layer (OB1_FUSION_MOA)
Council grounding/debiasing: objective check feeds the critique/revise/arbiter loop as oracle feedback (OB1_COUNCIL_CHECK — intrinsic self-correction degrades without it, Huang et al.) · distinct arbiter model (OB1_COUNCIL_ARBITER_MODEL) + anti-verbosity (LLM-judge bias) · optional symmetric debate (OB1_COUNCIL_DEBATE, Du et al.)
Adaptive router (/mode adaptive, /route): Solo-first, escalate to Fusion/Council only when the objective check fails (Snell et al. — match effort to difficulty)
Deterministic smokes for every piece (fusion-smoke, council-smoke, router-smoke) — 11/11 green

Honest note: these are wired and unit-tested, not yet proven to beat Solo at equal tokens — that's what the eval is for. Run /eval with a real OB1_*_CHECK to see if any now clears the bar.

Phase 9 — Capability frame + hard-task suite · 🟢 done

Goal reframe: the question that matters is "does a mode SOLVE the hard problems Solo fails?" (raw capability — token cost is not a gate), not just "beat Solo at equal tokens."

computeCapability / renderCapability: per-mode raw pass% · pass% on hard tasks (those Solo doesn't ace) · lift vs Solo · # recovered · # Solo never solved. /eval now prints Capability first, then the compute-matched efficiency table.
12 adversarially-verified hard tasks (eval/tasks/*.json) generated by a 12-agent fan-out workflow — date math, byte humanization, strict roman/base parsing, int→words, interval merge, CSV, version compare, glob, RLE, expr eval, duration. Every check was verified to discriminate a reference-correct impl from a plausible-wrong one (checks base64-embedded for quote-safety).

Phase 10 — spec-gap closure · 🟢 done (2026-06-20)

A spec audit (9 subsystem auditors + adversarial verify) found features in ob1-plan.html not yet built. Closed the "missing entirely" set — 12 of 13 (Ink TUI deferred):

Phase 11 — Ink (React) TUI · 🟢 done (2026-06-20)

The final spec-gap item: replaced the plan's recommended Ink front-end (was a plain readline REPL).

All former spec "partial" sub-items are now closed: MCP http/SSE transports (transport-abstracted JsonRpcMcpClient + Streamable-HTTP/legacy-SSE clients, factory dispatch) · Linux bubblewrap sandbox (bwrap backend, capability-probed, loud degradation) · Fusion git-worktree real-test scoring (per- candidate worktree at HEAD, real tests under the sandbox, diff fed to the judge). Each is covered by a deterministic smoke (sandbox-smoke, mcp-http-smoke, fusion-worktree-smoke) — 17/17 smokes green, typecheck clean. Hardened against a 13-bug adversarial review (string JSON-RPC ids, SSE connect timeout/ leak, bwrap --unshare-pid, worktree path-traversal/symlink-escape, shared-.git writes, new-file diffs).

Both verification gaps now closed with LIVE tests (2026-06-20):

scripts/mcp-interop.ts — drives all three transports (stdio, Streamable HTTP, legacy SSE) against the OFFICIAL @modelcontextprotocol/server-everything reference server (13 real tools, echo round-trips). ✓
scripts/bwrap-enforce.ts — runs the REAL wrapCommand() under bubblewrap on actual Linux (via colima Docker) and proves enforcement: network denied + writes confined per mode; also confirmed the probe loudly degrades (sandboxAvailable=false) when a host blocks unprivileged user namespaces. ✓ These two are opt-in live tests (need npx+network / a Linux+bwrap host), not part of the deterministic glob.

CI + the remaining verification gaps closed (2026-06-21):

.github/workflows/ci.yml — three jobs on push/PR: (1) typecheck + the deterministic suite (scripts/ci-smokes.ts, 18 smokes) + the real-PTY TUI test; (2) the two live tests — mcp-interop (npx reference server) and bwrap-enforce (apt bubblewrap; re-enables unprivileged user namespaces on Ubuntu 24.04 so the sandbox really enforces instead of self-skipping); (3) opt-in github-mcp (manual workflow_dispatch, gated on a GH_MCP_TOKEN secret, self-skips otherwise).
Authenticated MCP verified two ways: scripts/mcp-auth-smoke.ts (deterministic — a bearer token + arbitrary custom headers are sent on every request, a missing token is 401-rejected, and the Mcp-Session-Id is captured + replayed — on both Streamable HTTP and legacy SSE) and scripts/mcp-github-live.ts (LIVE — a real authenticated handshake with GitHub's remote MCP server at api.githubcopilot.com/mcp/: 27 tools listed, get_me round-trips). The authenticated-cloud-server gap is now closed against a real third-party endpoint, not just a local mock.

The full agent loop is verified live on Qwen via OpenRouter; the Anthropic path is code-complete but unexercised (streaming + cache_control unit-tested, not live — deferred per user).

Phase 12 — PLAN-V2 gap-closure · 🟡 in progress

Closing the 12 research-vs-codebase gaps from PLAN-V2.md (gitignored). Each item is web-researched for best practices first, then built read-only/opt-in per the standing conventions. Phase 1 (quick wins / foundation):

#13 Persistent usage analytics (/usage) — src/usage/log.ts appends one JSON line per model call to <dataDir>/usage.jsonl (the central accrue() chokepoint, so Solo + every multi-mind worker step is captured), then rolls it up by day / model / mode. Keeps the four token counters separate (input · output · cache-read · cache-write) and precomputes cache-aware cost (read 0.1× / write 1.25× of input, reusing the models.ts price table — no second copy of rates). /usage prints the report; added to HELP + the slash-menu. Research: ccusage field set / dedup / bucket model. Guard: usage-smoke (append/reload round-trip, corrupt-line tolerance, four-counter aggregation, cache-cost ratios, sum-reconciliation). 31 smokes.
#1 Subagents saved report (SUBAGENTS-PLAN Phase B) — after a spawn_subagents batch, writeSubagentReport saves .ob1/subagents/<ts>.md: parent task + run-metadata header (ok/failed counts, total tokens, failures called out up front), a one-row-per-subagent summary table whose tokens reconcile to the total, then one delimited section per subagent carrying the exact dispatched sub-task + context + full findings (the durable artifact the bounded tool_result loses) — failures kept visible, never dropped. Default ON (env opt-out OB1_SUBAGENTS_REPORT=0); best-effort so a write failure never breaks the turn; pointer logged to scrollback. Research: Claude Code summary-only return + Cline per-subagent cost rollup. Guard: subagents-smoke (formatter content + reconciling table + failure visibility + sanitized-ts file write + env toggle + end-to-end report saved through runTurn).
#8 Memory graph export (/memory export [dot|html]) — src/memory/export.ts writes the relationship graph to .ob1/memory-graph.<ext>: DOT (Graphviz, zero-dep, default — quoted/escaped ids, labeled edges) and a fully self-contained HTML (inline SVG, deterministic circular layout, no external src= — opens offline). Bi-temporal honesty: invalidated/expired edges render dashed + dimmed, never dropped (both formats), fed by listRelationships(true) + a new store.listEntities(). Research: DOT vs Cypher vs HTML (skip Cypher — needs a live DB); self-contained-HTML pattern. Guard: memory-export-smoke (DOT structure/escaping/dashed edges · self-contained HTML · determinism · real store path). 32 deterministic smokes, typecheck clean.

Phase 2 (memory quality — the largest verified gap):

#5 Weighted retrieval (recency + importance + relevance) — retrieval ranked by cosine alone; now a two-stage retrieve: (1) prefilter top-N by cosine, (2) re-rank by the Generative-Agents weighted score in src/memory/rank.ts — w_rel·rel + w_rec·recency + w_imp·importance, each min-max normalized across the candidate pool (the paper's method; a flat map kills cosine discrimination), with a relative-epsilon collapse so ms-apart facts don't get their recency spread amplified. New importance column (1–10, default 5) + idempotent migration + setImportance; recency = 0.5^(age/7d). Relevance-led defaults 1/0.5/0.3 (override OB1_MEM_WEIGHTS); 1,0,0 reduces exactly to cosine order (back-compat, pinned). Research: Park et al. Generative Agents §4 (formula, weights, decay) + Mem0 production posture. Guard: memory-rank-smoke (recency decay · min-max incl. near-constant collapse · recency/importance flips · back-compat · store re-rank); existing memory smokes regression-clean. 33 smokes.
#4 LLM-managed memory evolution (consolidate / dedup / contradiction) — remember() no longer blindly appends: when ON it retrieves the 10 nearest same-scope facts and asks a cheap LLM (src/memory/evolve.ts) to pick ONE op — ADD (new) · UPDATE (merge in place, same id, revision recorded) · DELETE (supersede a contradicted fact — archived, recoverable) · NOOP (drop a duplicate) — and to score the fact's importance 1–10 (feeds #5). Two safeguards the published systems lack: an UPDATE/DELETE/NOOP id outside the retrieved set is coerced to ADD (no hallucinated ids), and any parse failure / model error falls back to ADD (never lose info; immutable revisions make even a wrong op recoverable). Opt-in (cfg.memEvolve default OFF, env OB1_MEM_EVOLVE, persisted, /memory evolve on|off, cheap model via OB1_MEM_MODEL) since it costs one LLM call per write ([[no-auto-escalation-to-expensive-modes]]); every op surfaces a visible 🧠 note ([[visible-progress-no-silent-work]]). Research: Mem0 (4 ops, s=10, ids-from-input-only) + Vertex Memory Bank. Guard: memory-evolve-smoke (parse/validate/safeguards + all four ops through a real store with injected LLM + revision preservation + fail-safe + live toggle). 34 smokes.
#6 Reflection / consolidation trees — once accumulated fact importance crosses a threshold (150 on the 1–10 scale, per Park et al.), the agent distils its recent window into higher-level reflection facts (src/memory/reflect.ts), each linked to the sources it generalizes from via a reflection_sources table (the derived_from edges). New kind + reflection_level columns (+migration); a reflection sits one level above its sources and the depth cap (MAX 3) bounds reflection-of-reflection; the accumulator resets before distilling and reflections are created off the remember path, so they can't re-trigger themselves (loop guard). Ungrounded insights (citing no valid source id) are dropped (anti-hallucination). Opt-in (cfg.memReflect default OFF, OB1_MEM_REFLECT, persisted, /memory reflect on|off); best-effort. Research: Generative Agents §4.2 (threshold, citations, recursion). Guard: memory-reflect-smoke (threshold trip · grounded distillation through a real store · derived_from links · level math · depth cap · reset · OFF). 35 smokes.
#7 Agentic auto-linking (Zettelkasten) — folded into the #4 evolution call (one LLM round-trip): the model also proposes up to 3 related-memory links from a closed vocabulary (related_to / refines / contradicts), applied as idempotent fact↔fact edges in a fact_links table (distinct from the named-entity graph — facts are id'd, not named). Validated against the retrieved ids (no hallucinated targets), deduped, capped, and clamp-logged (requested vs kept), no self-links — the gap between the 10-candidate window and the 3-link cap is what stops a fully-connected graph (A-Mem finding). Opt-in (cfg.memAutolink default OFF, OB1_MEM_AUTOLINK, /memory autolink on|off); rides evolution (warns if evolve is off). Research: A-Mem (retrieve→ LLM-confirm, K-cap, closed relation set). Guard: memory-evolve-smoke extended (link parse/validate/ dedup/cap + real-store edges + idempotency + no-self-link + visible note). 35 smokes; typecheck clean.

Phase 2 complete. The memory engine now forgets, merges, prioritizes, reflects, and links — not just accretes. All four levers are opt-in/default-OFF (each costs LLM calls) and individually toggled.

Phase 3 (orchestration depth):

#11 Dual-ledger + re-plan / stall recovery — the fixed-round modes had no error recovery; added the Magentic-One orchestration pattern as an optional controller (src/multimind/ledger.ts): a Task Ledger (facts/guesses/plan) + a Progress Ledger (satisfied/progress/inLoop/nextStep) assessed each round, a stall counter that resets on progress and trips a bounded re-plan at the threshold (3, matching AutoGen) capped at MAX_REPLANS, hard-stopped at 20 rounds — gives up gracefully (stalledOut) instead of looping. Adds a deterministic loop backstop (identical worker output two rounds running → forced inLoop, no LLM trust needed). Pure state machine + injected assess/replan seams (live model-backed defaults for production). Opt-in: OB1_ORCH_LEDGER=1 routes /fanout through the act→assess→re-plan loop instead of single-shot. Research: Magentic-One (arxiv 2411.04468) + AutoGen defaults. Guard: ledger-smoke (parse · stall transition · replan trigger · satisfied-stop · stall-out-at-bound · loop backstop · clean pass-through). 36 smokes.
#2 Worktree write-capable subagents (SUBAGENTS-PLAN Phase C) — the riskiest feature, built loud and refusing-by-default (src/multimind/subagents-write.ts). Opt-in OB1_SUBAGENTS_WRITE exposes a spawn_write_subagents tool: each agent declares an explicit disjoint file lane and edits in its own git worktree (reuses worktree.ts); the safety model is (1) a pre-spawn partition check (overlapping declared lanes → refuse before any work), (2) post-work path-overlap detection (an agent that wrote out of lane → conflict even if git could auto-merge — the "clean-but-wrong" case), (3) abort-on-ANY-conflict leaving the real tree untouched, and (4) a single approval-gated sequential merge. Worktrees always cleaned up (try/finally). Forced off on apply turns. Research: Cline agent-teams + the git merge-tree/path-set preflight pattern (path-set is primary — the dangerous case merges cleanly). Guard: subagents-write-smoke — real temp git repo: partition refusal · disjoint worktree merge · post-work overlap abort (tree untouched) · gated apply (+ denial) · non-git guard · runTurn e2e (model → tool → gated apply lands files; inert when gated off). 37 smokes; typecheck clean.

Phase 3 complete. Orchestration gained bounded error-recovery (dual-ledger) and safe parallel writes (worktree-isolated, conflict-refusing, gated) — both opt-in/default-OFF.

Phase 4 (edit/execution paradigms — ⚠️ benchmark-claim-only, gated on /eval before being trusted):

#9 Code-as-action (CodeAct) — opt-in /codeact <task> runs the CodeAct paradigm: the model acts by emitting ONE fenced python/bash block (src/agent/codeact.ts), which runs in the existing OS sandbox (network-off) approval-gated, and the [observation exit_code=N] (head/tail-clipped) feeds back; it loops until the model answers with no code block (= done). Refinements from research: last-block-wins (models narrate then act), exit-code observations, a repetition/loop guard (identical block re-emitted → steer once, then stop as looping), 15-step cap. Pure parse/build/format core + injected model/exec/approve seams. Honest ⚠️: an unproven paradigm shift for our setup — kept a separate opt-in path (JSON tools stay the default) to be measured on /eval. Research: CodeAct (arxiv 2402.01030) + OpenHands. Guard: codeact-smoke (parse · observation/clip · loop · gate · abort · loop-guard · step cap). 38 smokes.
#10 Architect/Editor two-model edits — opt-in OB1_EDIT_ARCHITECT adds an architect_edit tool: a strong architect model describes the change in prose (no diff syntax — frontier models reason well but mangle structured diffs) and a cheaper editor model emits machine-appliable <<<<<<< SEARCH / ======= / >>>>>>> REPLACE blocks (with a whole-file fallback), applied through OB-1's existing flexible edit engine (src/agent/architect.ts is the pure prompts+parsing core; the tool applies — no import cycle). Pin the pair with OB1_ARCHITECT_MODEL / OB1_EDITOR_MODEL. Honest ⚠️: ~2× model calls, worth it only on complex edits — opt-in and to be measured on /eval. Research: Aider architect/editor (~85% edit-bench with a strong+cheap pair). Guard: architect-smoke (prose/mechanical prompts · search-replace + whole-file parse · injected pipeline · real-engine apply · throw-on-empty · conditional registration). 39 smokes; typecheck clean.

Phase 4 complete. Both edit paradigms shipped behind their own flags as separate opt-in paths (the JSON-tool loop stays the default). Per the plan they remain gated on /eval — wired + unit-tested, NOT yet proven to beat the baseline at equal tokens; that's the next validation step, not a claim made here.

Eval validation pass (2026-06-22) — the gate ran; verdict: default to Solo

Wired CodeAct as an eval mode (runCodeActMode in src/eval/runners.ts: develops+verifies via sandboxed execution in a throwaway scratch dir, then a final extraction call yields the graded block; selectable via bun run scripts/eval.ts solo codeact). Ran live on the baked auto/Qwen model:

run	tasks	result
Solo vs CodeAct	6 (2 simple + 4 hard)	both 100% pass; CodeAct 2.4–5.5× tokens → ✗ not justified
Solo vs Fusion vs Council	3 (partial)	all 100% pass; Fusion ~3.4–5× · Council ~3–3.7× tokens → ✗ not justified

Honest verdict: on the current 16-task suite this model aces every task as Solo (0 "hard" tasks), so the suite is saturated and cannot discriminate any mode — CodeAct/Fusion/Council all match Solo's correctness at 2.4–5.5× the cost and none clears the compute-matched bar. Per R5, default to Solo; the ⚠️ Phase-4 paradigms (and the heavy modes) correctly stay opt-in/flagged, credited to nobody on faith — which is exactly what the gate is for. #10 Architect/Editor was NOT measured: the suite is write-from-scratch, but architect_edit is an edit tool needing a starting file — a fair eval needs edit/fix-shaped tasks (buggy stub → repair), which the suite doesn't yet have. Next to make the eval meaningful: add genuinely hard tasks Solo fails + edit-shaped tasks.