LOGIC.md ↔ Archon Integration Evaluation

May 7, 2026 · View on GitHub

Workspace: /Users/rainierpotgieter/development/archon-test Date: 2026-04-19 Total trials: 60 (3 fixtures × 2 cases × 10 runs) Trials completed successfully: 60 / 60 (after one rate-limit re-run)

TL;DR Verdict

LOGIC.md is a real capability add for Archon, with caveats. On the data: at the severity-count layer Case B (LOGIC.md spec via MCP) was strictly more consistent than Case A (stock prompt) on every fixture where there was anything to vary on — most strikingly on the SQL-injection auth fixture, where Case B produced identical structured output 10/10 runs vs. Case A's 5/10. Case B also makes one capability the YAML layer cannot express today: an explicit, declarative decision rule ("first-match wins") that maps issue counts to a verdict. We can see that working in the data — Case B correctly split feature-weak-tests between NEEDS_DISCUSSION and REQUEST_CHANGES based on whether the model found 1 or 2 HIGH issues, while Case A flattened both into REQUEST_CHANGES regardless. Cost: Case B is ~2.6× slower per run (~91s vs ~34s) due to MCP roundtrips, and headline "verdict agreement" looks worse for B (87% vs 100%) — but that lower number is rule-driven differentiation, not noise. Recommended: worth flagging to Cole as a complement to YAML for review/validation nodes where reasoning rules matter, not as a replacement for YAML orchestration.

1. What Was Built

Integration architecture

                  Archon DAG (YAML — orchestration)
                  ┌──────────┐   ┌────────────┐
   PR diff  ─────▶│ load-diff├──▶│ code-review│  ──▶ structured verdict
                  └──────────┘   └─────┬──────┘
                                       │
                  Case B only:         │ mcp: .archon/mcp/logic-md.json
                                       ▼
                              ┌────────────────────────┐
                              │  @logic-md/mcp (stdio) │
                              │  logic_md_compile_     │
                              │  workflow tool         │
                              └────────────┬───────────┘
                                           │
                                           ▼
                              specs/code-review.logic.md
                              (declarative reasoning steps)

Concrete artifacts

  • archon/.archon/workflows/experiment-case-a.yaml — stock prompt-driven review node
  • archon/.archon/workflows/experiment-case-b.yaml — same node + mcp: .archon/mcp/logic-md.json
  • archon/.archon/mcp/logic-md.json — MCP server config ({ "logic-md": { "command": "logic-md-mcp" } })
  • archon/specs/code-review.logic.md — the LOGIC.md spec compiled at runtime by the MCP server
  • fixtures/{auth-sql-injection,feature-weak-tests,clean-refactor}/diff.patch — input diffs
  • harness/run.ts — drives all 60 trials, captures structured verdicts, copies workflow JSONL traces
  • harness/analyze.ts — produces results/SUMMARY.{md,json}

How the MCP handoff actually works (verified end-to-end)

  1. Archon's executor calls loadMcpConfig(".archon/mcp/logic-md.json", cwd) and passes the resulting mcpServers JSON straight to @anthropic-ai/claude-agent-sdk's query().
  2. Claude Agent SDK spawns logic-md-mcp as a stdio subprocess and auto-allows mcp__logic-md__* tools.
  3. The Case B prompt instructs Claude to call mcp__logic-md__logic_md_compile_workflow with {file_path: "specs/code-review.logic.md"}.
  4. The MCP server parses the spec, validates it, runs the DAG resolver, and returns a structured executionPlan: [{stepName, dagLevel, systemPromptSegment, outputSchema, qualityGates}, …].
  5. Claude follows each step's systemPromptSegment in dagLevel order and returns the final structured verdict via Archon's output_format: schema.

Verified by inspecting workflow event JSONL at ~/.archon/workspaces/coleam00/Archon/logs/<run-id>.jsonl:

  • Case A traces show exactly 1 tool call: StructuredOutput (the JSON capture).
  • Case B traces show: ToolSearch → mcp__logic-md__logic_md_compile_workflow → assistant reasoning → StructuredOutput.

Zero patches to Archon

Archon already has first-class MCP support (packages/providers/src/claude/provider.ts's loadMcpConfig + applyNodeConfig). The integration is two YAML files and one JSON config; no TypeScript changes were made to Archon itself. The YAML reasoning prompt in Case B explicitly tells the model to call the MCP tool; if Archon's docs surfaced this pattern as a first-class workflow, no instruction would be needed.

2. Fixtures Used

FixtureConcern ExercisedExpected verdict (informal)
auth-sql-injectionThree security regressions in one diff: SQL string interpolation, hardcoded JWT secret, plaintext password compareREQUEST_CHANGES (CRITICAL)
feature-weak-testsA new Stripe-charge feature with one happy-path test, no negative paths, no edge casesREQUEST_CHANGES or NEEDS_DISCUSSION (borderline)
clean-refactorExtract formatCurrency utility from two duplicated call sites with full test coverageAPPROVE

The fixtures live under fixtures/<name>/diff.patch and are version-controlled with the workspace.

3. Quantitative Results

Determinism — verdict-label and structural-hash agreement

Hash is sha256({verdict, critical_count, high_count}) — i.e., we strip the prose summary and other noise, and ask: across 10 identical runs, how often does the model land on the same (verdict + critical + high) tuple?

FixtureCaseRunsModal verdictVerdict agreementModal hash countHash agreementUnique full-output hashes (verdict + counts + summary text)
auth-sql-injectionA10REQUEST_CHANGES100%5 / 1050%10
auth-sql-injectionB10REQUEST_CHANGES100%10 / 10100%10
feature-weak-testsA10REQUEST_CHANGES100%6 / 1060%10
feature-weak-testsB10NEEDS_DISCUSSION60%6 / 1060%10
clean-refactorA10APPROVE100%10 / 10100%10
clean-refactorB10APPROVE100%10 / 10100%10

Aggregate (averaged over fixtures)

MetricCase A (stock)Case B (LOGIC.md)Δ
Verdict agreement100%87%−13%
Hash agreement (verdict + crit + high)70%87%+17%
Avg run duration34.5 s91.0 s+56.5 s (+164%)

Severity-count means (per fixture × case)

FixtureCaseavg critavg highavg medavg low
auth-sql-injectionA3.001.701.400.60
auth-sql-injectionB2.002.001.101.50
feature-weak-testsA0.003.202.701.50
feature-weak-testsB0.001.503.500.80
clean-refactorA0.000.000.102.20
clean-refactorB0.000.000.201.90

Reading the results honestly

  • Case B beats Case A handily on the auth fixture (100% vs 50% hash) — the headline win.
  • Case B beats Case A also on clean-refactor at the maximum: both at 100%, and importantly, Case B does not introduce noise on inputs where the answer is unambiguous.
  • Case B "loses" on feature-weak-tests verdict agreement (60% vs 100%) — but this is an honest disagreement, not noise. The LOGIC.md spec's synthesize_verdict step has the explicit rule: exactly 1 HIGH issue → NEEDS_DISCUSSION. Across 10 runs the model found 1 HIGH in 6 runs, 2 HIGHs in 3 runs, and 3 HIGHs in 1 run. The structured rule then deterministically mapped those counts to NEEDS_DISCUSSION (6) / REQUEST_CHANGES (4). Case A doesn't have such a rule — its prose prompt collapses 1-HIGH and 2-HIGH cases into a single REQUEST_CHANGES verdict, so it looks "more consistent" but only because it has lower verdict resolution.
  • Case B is 2.6× slower. The MCP subprocess spawn + tool call + tool result round-trip costs ~50–60 s per run. This is currently a real cost.

So the right two-line summary of the determinism numbers:

  • Within a fixed verdict bucket, Case B is more structurally consistent than Case A (the auth fixture is a near-perfect demonstration).
  • Case B's verdict labels reflect the model's underlying issue-count variance more faithfully (which Case A masks).

4. Qualitative Findings

4.1 Auditability — side-by-side trace from auth-sql-injection / run-01

Case A workflow event log (relevant lines, all 5 events):

[01:07:44.022Z]  workflow_start
[01:07:44.032Z]  node_start          (load-diff)
[01:07:44.051Z]  node_complete       (load-diff)
[01:07:44.054Z]  node_start          (code-review)
[01:08:06.480Z]  tool                StructuredOutput        ← only tool call, the JSON capture
[01:08:09.249Z]  node_complete       (code-review)
[01:08:09.250Z]  workflow_complete

There is one tool event in the entire trace, and it's just the JSON output bookkeeping. The reasoning the model used to get from "diff" to "verdict" is opaque — it happened in a single inference call that left no breadcrumb in the trace.

Case B workflow event log:

[01:12:30.947Z]  workflow_start
[01:12:30.953Z]  node_start          (load-diff)
[01:12:30.965Z]  node_complete       (load-diff)
[01:12:30.968Z]  node_start          (code-review)
[01:12:40.840Z]  tool                ToolSearch              ← model selecting MCP tool
[01:12:43.083Z]  tool                mcp__logic-md__logic_md_compile_workflow
                                     input: { file_path: "specs/code-review.logic.md" }
[01:13:39.648Z]  assistant           (reasoning over the compiled executionPlan)
[01:13:43.265Z]  tool                StructuredOutput
[01:13:49.627Z]  assistant
[01:13:49.691Z]  node_complete       (code-review)
[01:13:49.692Z]  workflow_complete

The trace shows what reasoning structure the model was handed: compile_workflow returns an executionPlan of named steps (scan, assess_security, check_test_coverage, evaluate_quality, synthesize_verdict), each with its own systemPromptSegment and outputSchema. An auditor can:

  1. Open specs/code-review.logic.md and see the rules the agent was told to apply.
  2. Open the trace and confirm that the model received those rules (the MCP tool call + result are recorded).
  3. Compare summary text against the spec's decision rules to verify the model applied them.

For Case A, none of these are possible. The same auditor sees only "Claude reasoned, Claude returned JSON." If that JSON looks wrong, the only artifact to reach for is the prompt — but the prompt is prose, and there's no checkpoint between "received prompt" and "produced verdict."

Concrete consequence: the Case B trace is verifiable against an external spec. The Case A trace requires re-reading the prompt to imagine what the model "should have" done.

4.2 Modifiability — diff to add one new rule

Rule change applied to both cases: "Any HIGH-severity security issue should block the PR (REQUEST_CHANGES), even on its own. A lone HIGH non-security issue still goes to NEEDS_DISCUSSION."

Case A (prose prompt edit)

       ## How to decide verdict

       - APPROVE: no issues, or only LOW issues
-      - REQUEST_CHANGES: any CRITICAL, or multiple HIGH issues
-      - NEEDS_DISCUSSION: borderline cases (one HIGH, or many MEDIUM)
+      - REQUEST_CHANGES: any CRITICAL, OR any HIGH-severity *security* issue
+        (security takes priority — a single HIGH security finding blocks merge),
+        OR multiple HIGH issues of any kind
+      - NEEDS_DISCUSSION: borderline cases (one HIGH non-security issue, or many MEDIUM)

+      When in doubt about whether an issue is a "security" issue, treat
+      injection, secrets, auth, and credential-handling concerns as security.
+
       Return a structured JSON object with the verdict, severity counts, and a one-paragraph summary.

Lines added: 6. Lines removed: 2. Files: 1 (experiment-case-a.yaml).

The change is a prose edit. It cannot be syntactically validated. To gain any confidence the rule is being followed, you re-run the experiment and read 10 summaries. The clarification clause ("when in doubt…") was needed because prose is ambiguous — without it, the model has to infer what "security" means, and infers inconsistently.

Case B (LOGIC.md spec edit)

       1. ANY issue with severity CRITICAL → verdict = REQUEST_CHANGES
-      2. >= 2 issues with severity HIGH → verdict = REQUEST_CHANGES
-      3. exactly 1 HIGH issue → verdict = NEEDS_DISCUSSION
-      4. only MEDIUM/LOW issues → verdict = NEEDS_DISCUSSION if MEDIUM count >= 2, else APPROVE
-      5. no issues → verdict = APPROVE
+      2. ANY issue with severity HIGH that came from `assess_security` → verdict = REQUEST_CHANGES
+      3. >= 2 issues with severity HIGH (any source) → verdict = REQUEST_CHANGES
+      4. exactly 1 HIGH issue (non-security) → verdict = NEEDS_DISCUSSION
+      5. only MEDIUM/LOW issues → verdict = NEEDS_DISCUSSION if MEDIUM count >= 2, else APPROVE
+      6. no issues → verdict = APPROVE

Lines added: 5. Lines removed: 4. Files: 1 (specs/code-review.logic.md).

The change inserts a new explicit rule at a known position ("first match wins" guarantees it overrides #3). The assess_security reference is typed — that step name has to exist in the spec, and logic_md_validate can mechanically check it (@logic-md/cli validate). The disambiguation prose Case A needs ("when in doubt…") is unnecessary because the rule names its source step.

Modifiability comparison

PropertyCase A (prompt)Case B (LOGIC.md spec)
Lines changed8 (6 added, 2 removed)9 (5 added, 4 removed)
Files touched11
Static-validatable?NoYes (@logic-md/cli validate can syntax-check; assess_security step name is referenced from the rule and validated against the spec)
Disambiguation prose required?Yes ("when in doubt…")No (the rule cites its source step by name)
Diff scrutability for non-author reviewerMedium — reviewer must re-read prose and reason about prompt contractHigh — reviewer reads structured rules in declared order

The diffs are roughly the same size. The kinds of change are very different: Case A edits are prose engineering with no contract; Case B edits are structured rule edits with a checkable contract.

4.3 Capabilities Case B exposes that Case A genuinely cannot

This is the falsifiable question. Three capabilities surfaced in this experiment that are real (not "you could in principle do this in Archon's prose prompt"):

  1. Explicit decision rules with deterministic precedence. synthesize_verdict says "apply these rules in order, first match wins." When the model finds 1 HIGH, the rule fires NEEDS_DISCUSSION; 2 HIGHs fire REQUEST_CHANGES. The data shows this actually working: feature-weak-tests Case B's verdict tracks the model's HIGH count exactly. Case A's prose has an analogous-looking rule, but since it's interpreted by the model rather than applied as a rule, all 10 feature-weak-tests Case A runs collapsed to REQUEST_CHANGES regardless of HIGH count. The structured rule is a decision contract; the prose is a suggestion.

  2. Cross-step dependencies (needs:) that name where information comes from. The Case B modifiability change keys on "HIGH issue from assess_security." That cross-reference is semantic, not textual — the model sees a structured previousOutputs map keyed by step name. To express the same in Case A, you have to either give the model an instruction to "tag" issues with their source category in the prose, or rely on the model to infer. Both are fragile in ways the structured cross-reference isn't.

  3. A trace whose tool-call structure mirrors the reasoning structure. Case B's mcp__logic-md__logic_md_compile_workflow call is recorded with input + result in the workflow event JSONL. An audit script can extract the executionPlan that was handed to the model and diff it against the spec on disk to confirm the spec wasn't tampered with. There's no analogous artifact in Case A — the prompt becomes part of the LLM call and is gone.

What Case A wins on

Case A is 2.6× faster and verdict-stable by virtue of being verdict-coarse. For workflows where:

  • The verdict label is the only thing that downstream nodes consume, and
  • The reasoning is unambiguous (e.g., "is this a typo? yes/no"),
  • Then the LOGIC.md indirection is overhead.

Most review/validation work in a coding agent isn't that simple, so Case A's win is narrow.

5. Honest Limitations

  1. Determinism is structural, not arithmetic. Archon does not pass a temperature to the Claude Agent SDK; both cases use whatever sampling defaults the SDK applies. There is no way to make the comparison bit-for-bit. We are measuring how often the model lands on the same labeled outcome, not whether it computes the same logits. Confirmed by reading packages/providers/src/claude/provider.ts — no temperature/top_p parameter is forwarded.

  2. Sample size is small. 10 runs per (fixture, case) is enough to see headline differences but not enough for tight confidence intervals. The 50% hash agreement on auth/case-A could plausibly be 40–60%; the 87% aggregate could be 80–94%. The directional findings are robust; the precise percentages are not.

  3. Three fixtures isn't a benchmark. The fixtures were picked to span the verdict surface (CRITICAL, ambiguous, clean). They are not a curated benchmark of real PR diffs. Generalizing to "Case B is always more consistent on review tasks" requires more fixtures and a harder confound: input difficulty.

  4. Both cases call the same model with the same input — Case B's reasoning still happens inside Claude. LOGIC.md compiles a more structured prompt; it does not perform any of the reasoning itself. The "determinism" we observe is therefore the prompt structure's effect on Claude's output variance, not deterministic computation.

  5. The 60s MCP cold-start risk documented in the research was avoided by npm install -g @logic-md/mcp@1.4.0. In a deployment scenario this would need handling (a vendored binary or a long-running MCP server).

  6. One rate-limit hit during the experiment. 29 of 60 trials failed initially when Claude's 5-hour rate limit was reached. After the limit reset, the same 29 trials were re-run successfully. This is unrelated to the integration — it's a cap on this account — but it does mean the experiment is sensitive to API-side caps in a way Cole would need to consider for a CI use case.

  7. The Case B spec is small (~150 lines). Larger LOGIC.md specs may have meaningful token cost (compile_workflow returns a tokenEstimate per step). Not measured here.

  8. Archon's existing validation/retry primitives could narrow the gap. Archon has output_format: (structured Zod schema) and trigger_rule: join semantics, and could in principle add rule-application primitives at the DAG level instead of at the per-node-reasoning level. We did not benchmark Case B against a hypothetical "Case A++" with hand-rolled validation downstream of the review node.

6. Recommendation

(a) Worth flagging to Cole — as a complement, not a replacement.

The data supports a focused pitch:

  • Archon's YAML layer is great at orchestration. Nothing in this experiment suggests replacing it.
  • For nodes whose value comes from applied reasoning rules (review, validation, classification gates with multiple criteria, anything where "did we follow the policy" matters), LOGIC.md gives a real, measurable improvement to structural consistency and a meaningful improvement to auditability and modifiability. The integration cost is one JSON config and one new node-type pattern.
  • Concretely, the integration vector to propose is a first-class logic_spec: field on a node (sugar over today's mcp: + prompt: pattern), so users don't have to write the "call this MCP tool, follow the executionPlan" prompt themselves. Archon already builds the MCP plumbing; LOGIC.md already produces the compiled reasoning. The gap is documentation and ergonomics, not a code change to either system.
  • The current cost (~2.6× slower per node) is real but localized. It's worth it for nodes whose output is consumed downstream by humans for decisions, and not worth it for fast classification gates.

What would change the recommendation:

  • If Archon's roadmap already includes structured per-step output schemas with rule application (i.e., bringing decision-rule semantics into YAML directly), this becomes redundant overlap. Worth checking with Cole before the pitch.
  • If Cole's users overwhelmingly want a single end-to-end agent and aren't asking for auditability of the inside of a node, this becomes a nice-to-have without product pull.

The pitch is honest if framed as: "We measured this on three fixtures, 60 runs. Here's where it wins, here's what it costs, here's the integration shape that requires zero changes to Archon's executor."

7. Decisions Made (no clarifications were sought from the user)

  1. Used prompt: + mcp: rather than forking a command: node. Archon's command: archon-code-review-agent loads a markdown prompt that is hard-coupled to GitHub-PR semantics (gh pr diff, $ARTIFACTS_DIR/review/scope.md). For a fair comparison with a local-fixture experiment, both Case A and Case B use a prompt: node with the same input shape. Stock Archon production review would still go through the command: path; the experiment isolates the reasoning step, not the surrounding pipeline.
  2. Hashing strategy strips summary text and only hashes {verdict, critical, high}. Including summary makes every run unique on prose noise; excluding it isolates the structured outcome. (full_hash over the full structured object is also computed and reported separately.)
  3. Used global install of @logic-md/mcp (logic-md-mcp binary) to avoid the 60s npx cold-start documented in the research as Risk R7.
  4. Did not use Archon's --worktree isolation. Worktrees per run would 2× the disk footprint and complicate path resolution for the spec; --no-worktree is documented as a valid CLI flag and works.
  5. Used model: sonnet explicitly on both nodes to avoid the Haiku-MCP silent-failure risk documented as R6.
  6. Re-ran failed cells after the rate-limit reset by adding skip-if-exists logic to the harness, then invoking again. Re-runs were not mixed with original runs at the analysis layer.

8. Artifacts

  • archon/ — full Archon clone (dev branch, version 0.3.6, bun 1.3.12)
  • logic-md/ — full LOGIC.md clone (main branch, @logic-md/mcp 1.4.0)
  • archon/.archon/workflows/experiment-case-a.yaml — Case A workflow
  • archon/.archon/workflows/experiment-case-b.yaml — Case B workflow
  • archon/.archon/mcp/logic-md.json — MCP server config
  • archon/specs/code-review.logic.md — LOGIC.md reasoning spec
  • fixtures/auth-sql-injection/diff.patch — Fixture 1
  • fixtures/feature-weak-tests/diff.patch — Fixture 2
  • fixtures/clean-refactor/diff.patch — Fixture 3
  • harness/run.ts — 60-run driver (with RUNS/FIXTURES/CASES env overrides + skip-if-exists)
  • harness/analyze.ts — analyzer producing results/SUMMARY.{md,json}
  • results/<fixture>/case-{A,B}/run-NN.json — per-run structured records
  • results/<fixture>/case-{A,B}/run-NN-trace.jsonl — per-run workflow event traces
  • results/SUMMARY.json, results/SUMMARY.md — aggregated metrics
  • results/harness.log, results/harness-rerun.log — full harness console output
  • modifiability/case-{a-original,a-modified}.yaml — Case A modifiability before/after
  • modifiability/case-{b-original,b-modified}.logic.md — Case B modifiability before/after
  • modifiability/case-{a,b}.diff — unified diffs for the rule change
  • .planning/REQUIREMENTS.md, .planning/STATE.md, .planning/phases/001-logic-md-archon-integration/ — project tracking
  • .planning/phases/001-logic-md-archon-integration/001-RESEARCH.md — pre-execution research used to plan the integration

Reproducing the experiment

# From workspace root, with bun on PATH:
PATH="$HOME/.bun/bin:$PATH" bun run harness/run.ts             # 60 trials (skips already-completed)
PATH="$HOME/.bun/bin:$PATH" bun run harness/analyze.ts          # regenerate SUMMARY.{md,json}

# Single-cell smoke:
RUNS=1 FIXTURES=clean-refactor CASES=A bun run harness/run.ts