LOGIC.md × Archon

May 7, 2026 · View on GitHub

A 60-trial controlled experiment measuring whether LOGIC.md adds something to Archon's review-style nodes that YAML-only orchestration can't already express.

Date: 2026-04-19 Model: Claude Sonnet (via Claude Agent SDK, default temperature) Scope: 3 fixtures × 2 cases × 10 runs = 60 trials Headline: structural-consistency improvement (70% → 87% overall hash agreement); auth-sql-injection fixture goes from 50% to 100% identical structured outputs across 10 runs. Runtime cost is real (2.6×). Full report.

What this evaluates

Archon's YAML layer orchestrates between nodes — what runs, in what order, with what dependencies. Excellent at that.

LOGIC.md declares reasoning inside a node — the multi-step cognition, output schemas, quality gates, and tool permissions an agent applies at one step.

The two compose orthogonally. This experiment tests whether composing them produces measurable benefit on review-style nodes (where structured, auditable, modifiable reasoning matters most).

The two cases

Case A — stock prompt (control). Archon node configured with a prose prompt: and output_format: (Zod schema for the verdict). No LOGIC.md.

Case B — LOGIC.md spec (treatment). Same Archon node. The reasoning step is delegated to LOGIC.md's MCP server (@logic-md/mcp) via the existing mcp: field on the node. The LOGIC.md spec defines: 4 reasoning steps (assess_correctness, assess_style, assess_security, aggregate_verdict), output contracts, and a deterministic precedence rule for verdict aggregation.

Zero patches to Archon. The integration uses mcp: + prompt: that Archon already supports. The pitch (if there is one) is a logic_spec: ergonomic field, not an architectural change.

Fixtures

Three deliberately calibrated PR diffs, each represented as a diff.patch:

Fixture	What	Expected verdict
`auth-sql-injection`	Three security regressions: SQL string interpolation, hardcoded JWT secret, plaintext password compare	`REQUEST_CHANGES` (CRITICAL)
`feature-weak-tests`	New Stripe charge feature, single happy-path test, no failure-mode coverage	Borderline — `REQUEST_CHANGES` or `NEEDS_DISCUSSION`
`clean-refactor`	Extract pure utility function, add tests, no behaviour change	`APPROVE`

Diffs at fixtures/<name>/diff.patch.

Headline results

                         Case A (prose)        Case B (LOGIC.md)      Δ
Verdict-label agreement   100%                  87%                   −13%
Hash agreement (overall)  70%                   87%                   +17%
Avg runtime               34.5 s                91.0 s                +56.5 s (+164%)

Per-fixture hash agreement (sha256({verdict, critical_count, high_count})):

Fixture	Case A	Case B
auth-sql-injection	50% (5 different tuples in 10 runs)	100% (identical)
feature-weak-tests	60%	60%
clean-refactor	100%	100%

Full per-trial breakdown. Full report with methodology, traces, and analysis.

The single most important result

auth-sql-injection, Case A: 10 runs, 5 different (verdict, critical_count, high_count) tuples. Modal output appeared 5/10 times. The model's choice between e.g. "1 critical / 2 high" vs "2 critical / 1 high" is essentially a coin flip on identical input.

auth-sql-injection, Case B: 10 runs, 1 unique tuple. 10/10 identical structured output.

Same model, same prompt content, same fixture. The difference is whether the reasoning is declared as a structured contract (Case B) or expressed in prose (Case A). LOGIC.md doesn't change the verdict — both reach REQUEST_CHANGES — but it eliminates the structural variance in the supporting metadata that downstream consumers depend on.

The honest disagreement on `feature-weak-tests`

Case B's verdict-label agreement on this fixture is 60% (split between REQUEST_CHANGES:4 and NEEDS_DISCUSSION:6). On its face that looks worse than Case A's 100% REQUEST_CHANGES:10.

It isn't. Case B's verdict aggregation rule is explicit: >= 2 HIGH → REQUEST_CHANGES, exactly 1 HIGH → NEEDS_DISCUSSION. The 60/40 split reflects deterministic application of that rule against the model's stochastic count of HIGH issues (1 or 2 across runs). Case A masked the same underlying variance behind a uniform REQUEST_CHANGES because its prose didn't constrain the verdict-from-counts mapping.

Case B's verdict variance is a property of the LLM (count instability), made visible by the rule. Case A's verdict consistency is a side effect of prose loosely mapping any non-empty issue list to REQUEST_CHANGES. The two are not comparable as "consistency."

Modifiability

Adding a new rule — "any HIGH-severity issue that came from the security-assessment step blocks the PR" — was added to both cases.

Case A (diff): 8-line prose edit inside the prompt. No type or contract validation. Whether the LLM honours the new rule depends on prose comprehension at runtime.
Case B (diff): 9-line structured edit to the precedence list inside the .logic.md spec, with @logic-md/cli validate as a CLI contract check. The rule becomes a typed, validated contract.

Both diffs are roughly the same size. The Case B diff has type-checked semantics. The Case A diff is a textual hint.

Runtime cost is real

Case B averages 2.6× the wall-clock of Case A (~91 s vs ~34 s). The cost is the MCP tool call: subprocess spawn, tool roundtrip, structured-output parse on the LOGIC.md side.

This trade is worth making for audit-heavy review nodes. It is not worth making for fast classification gates. The pitch would be: use LOGIC.md selectively on the steps where structured reasoning, audit traces, and modifiability matter; keep prose prompts elsewhere.

Honest limitations

N=3 fixtures is a pilot, not a benchmark. Generalising would require 8-10 fixtures spanning architectural reviews, dependency bumps, perf changes, and security work.
Same model across both cases. Both cases inherit identical LLM non-determinism (Archon passes no temperature to the SDK in either case).
One rate-limit hit during the original run. 29 of 60 trials failed initially and were rerun after the API rate limit reset. Final numbers reflect the rerun.
Archon's own primitives could narrow the gap. Archon already supports output_format: (Zod) and trigger_rule:. A future Archon version that pushed rule application to the DAG level might deliver similar gains without LOGIC.md. This experiment did not benchmark against a hypothetical "Case A++."
Auditability and modifiability are demonstrated, not measured. They're properties of the runtime architecture (workflow event traces from Archon's logs, structured diffs from the spec), not numerical metrics.

Repo structure

.
├── README.md                   This file
├── REPORT.md                   Full report (28 KB) — methodology, traces, per-fixture analysis
├── fixtures/                   3 PR diffs used as test inputs
│   ├── auth-sql-injection/diff.patch
│   ├── feature-weak-tests/diff.patch
│   └── clean-refactor/diff.patch
├── harness/                    Experiment driver (Bun + TypeScript)
│   ├── run.ts                  Spawns 60 trials, captures verdicts + traces
│   └── analyze.ts              Aggregates results, computes hashes, writes SUMMARY
├── modifiability/              Modifiability test artifacts
│   ├── case-a-{original,modified}.yaml
│   ├── case-a.diff
│   ├── case-b-{original,modified}.logic.md
│   └── case-b.diff
├── results/                    Raw output from 60 trials
│   ├── SUMMARY.json
│   ├── SUMMARY.md              Per-fixture × case statistics
│   ├── harness.log             Original run log
│   ├── harness-rerun.log       Rerun log (after rate-limit recovery)
│   └── <fixture>/case-{A,B}/
│       ├── run-NN.json         Verdict + hashes
│       └── run-NN-trace.jsonl  Workflow event trace from Archon
└── .planning/                  Methodology / requirements docs (REQUIREMENTS.md, STATE.md)

Reproducing

This repo is the experiment artifact. To reproduce, you also need Archon and LOGIC.md cloned alongside.

# 1. Clone everything into the same parent directory
mkdir -p ~/eval && cd ~/eval
git clone https://github.com/SingularityAI-Dev/logic-md-archon-eval.git
cd logic-md-archon-eval
git clone https://github.com/coleam00/Archon.git archon
git clone https://github.com/SingularityAI-Dev/logic-md.git logic-md

# 2. Set up Archon and LOGIC.md per their respective READMEs
cd archon && bun install && bun run build
cd ../logic-md && npm install && npm run build
cd ..

# 3. Configure the experiment workflows in archon/specs/
#    (case A: prose; case B: mcp invocation of @logic-md/mcp)
#    See REPORT.md §3 for the workflow YAML.

# 4. Set ANTHROPIC_API_KEY (Claude Sonnet via Agent SDK)
export ANTHROPIC_API_KEY=sk-ant-...

# 5. Run the harness
bun run harness/run.ts                  # default: 10 runs × 3 fixtures × 2 cases
RUNS=2 bun run harness/run.ts           # quick smoke (12 trials)

# 6. Analyse
bun run harness/analyze.ts              # writes results/SUMMARY.{json,md}

The harness expects Archon's CLI at ./archon and writes to ./results. It reads workflow event traces from ~/.archon/workspaces/coleam00/Archon/logs/ (the Archon default).

Methodology details

For full per-trial breakdown, scoring rubric, hash construction, trace excerpts, and the complete REPORT including discussion sections, see REPORT.md (28 KB).

For the underlying methodology requirements, see .planning/REQUIREMENTS.md.

What this is not

Not a quality benchmark of LOGIC.md generally. Cross-model quality benchmarks for LOGIC.md are at SingularityAI-Dev/logic-md benchmarks/published/ and currently show no measurable quality lift on Sonnet 4.6 or Llama 3.1 70B at n=10 per condition. The value LOGIC.md provides is structural — consistency, auditability, modifiability — not generative.
Not a critique of Archon. Archon's design is sound; this experiment shows where LOGIC.md complements rather than replaces it.
Not a finished benchmark. N=3 fixtures, single model, one experimental session. Treat the directional findings as robust, the precise percentages as wide-CI estimates.

License

MIT. The fixtures, harness, and results in this repo are released under MIT. Archon and LOGIC.md retain their own licenses (both MIT at time of writing).

Contact

Issues, PRs, questions: SingularityAI-Dev/logic-md.