Maturity Model & Self-Assessment

On-demand reference (not auto-loaded). The evaluation yardstick for this skill's convergence-loop completeness: how to grade it, where it sits, and how to keep a change from lowering the grade. This is the epidemiology / grading layer.

Self-contained. This page carries the full maturity framework inline — the long-run degradation effects, the L1–L4 ladder, the orthogonal verification axis, the calibration anchors — so it reads correctly on its own, with no dependency on the private upstream. (It does cross-reference companion docs in this skill — convergent-loop.md, intent-blueprint.md, arch-contracts.md — which ship with the skill; that is not an external dependency.) The framework originates from a private internal wiki (ai-coding-agent-maturity); it is acknowledged here as provenance, not a citation the reader can follow. Maintainer note: this page is a publication port — re-sync it from the private upstream when that changes.

It is orthogonal to two adjacent layers — do not duplicate them here:

• The therapeutics (how the loop is implemented) live in convergent-loop.md (dual-ring), intent-blueprint.md (goal anchor), and arch-contracts.md (deterministic gate).
• The pathology (the catalog of failure modes) is out of scope here — this page is the ruler (epidemiology / grading), not the symptom catalog.

This page only provides the ruler that measures the therapeutics' completeness.

Why the ceiling is set by flow control, not prompt tricks

In the standard Thought → Action → Observation loop, single-step generation quality is near-saturated, but a long task's total pass rate is the product of per-step reliability — ~95% per step collapses below 8% over fifty independent steps. The lever on "how long it can run unattended" is therefore not a smarter single prompt but systematic governance of uncertainty across the sequence: keeping the goal aligned after each step, stopping deviations from compounding along the step chain, and keeping effective attention as the window bloats. That is exactly what convergent-loop.md models; this page grades how completely.

The degradation effects (the evaluation dimensions)

Maturity is measured by how completely the system defends against long-run failures. The first three (Context Rot / Error Compounding / Goal Drift) are physical consequences of probabilistic generation plus bounded attention over a long sequence — none is cured by "a better prompt." The fourth (Specification Gaming) is a relationship failure (agent optimizes a proxy spec, not the real goal), not a physical one — it is orthogonal to flow control, its defense lives on a separate axis (see §Orthogonal axis below), and it does not advance L1–L4. L1–L4 grade only the first three.

Degradation	Definition	Mature defense	Mechanism in this skill
Context Rot	Redundant thinking and verbose tool output (thousand-line logs) fill the window; the agent loses memory and degrades late. Attention is U-shaped — middle content is used least.	Explicit pruning + state-summary folding at tier transitions.	Context folding in convergent-loop.md: the inner→outer transition passes only loop_state.py summary (one folded sentence) + Diff + blueprint, never the inner stderr trail; the Coder keeps only the blueprint + distilled feedback + last 1–2 inner iterations.
Error Compounding	A tiny per-step deviation (wrong file, failed regex, hallucinated path) is amplified exponentially down the step chain.	Structured self-correction + a separate reflection pass (diagnose ≠ fix).	Dual-ring self-correction: fast_gate.py PostToolUse decision:block → next-turn self-fix + short-circuit; the arch-contracts.md gate emits structured 越权日志 (file:line + suggestion); the outer-ring independent code-reviewer is a second reflection pass; gate附加条件 forbids "delete the failing test to get Exit 0".
Goal Drift	On large cross-module tasks the agent is pulled into a sub-problem and forgets the global goal.	An external, read-only intent anchor + periodic diff-against-original.	intent-blueprint.md: frozen blueprint with three-layer read-only enforcement (PreToolUse blueprint_guard.py DENY + revision channel + reviewer check); diff-to-blueprint check → intent-drift verdict → hard rollback (snapshot.py restore) + reverse-prompt injection.
Specification Gaming (orthogonal)	The agent optimizes a proxy spec ("passes tests", type-checks, self-review) instead of the real goal (semantically correct, meets intent) — silent semantic failure. Root cause: any executable validation is a proxy for the real goal, and the gap is irreducible (specification problem). Key tension: L4's "proactive test self-verification" is same-family verification → a carrier of this degradation, not a defense.	An different-family oracle (external verification source whose blind-spot set differs from the agent's): production-runtime regression, formal-spec, human semantic gate, or a different model/prompt-family independent review. Self-certification cannot cross this — fresh context or same-model reviewers still share the blind spot.	Partial (ADR #40) — SolidForge has a PARTIAL first-class different-family-oracle gate: infra/scripts/hetero_review.py runs an adversarial review on a DIFFERENT model family (cross-family blind-spot check) at the orchestration layer, additive over the same-family primary. PARTIAL, not complete — different commercial model families share training-data/RLHF overlap, so it raises the same-family ceiling without fully removing the spec-gaming gap. Production-runtime regression / formal-spec / human-semantic gate remain future. The outer-ring code-reviewer is same-model (shared blind spot) → defends Goal Drift, NOT Specification Gaming. This axis is partially defended, not a silent green.

Orthogonal axis: verification-source decoupling (Specification-Gaming defense)

The fourth degradation (Specification Gaming) is defended NOT by advancing along L1–L4 but along an orthogonal second axis: verification-source decoupling — from same-family (agent self-verification) to an different-family oracle (external source whose blind-spot set differs from the agent's). This is NOT L5 (numbering it so would fake same-axis progression). L4 is the intrinsic-flow-control ceiling: the self-certification paradox (agent self-test = same-family, reproduces the blind spot) means "a stronger agent" cannot cross it — the oracle must be external (production runtime, formal spec, human semantic gate, a different model). SolidForge now has a PARTIAL first-class different-family-oracle gate (infra/scripts/hetero_review.py, ADR #40 — a different model-family adversarial review at the orchestration layer); it is PARTIAL because commercial model families share training-data/RLHF overlap. Production-runtime / formal-spec / human-semantic oracles remain future. This axis is partially defended (rule 3 — partial, not silent).

The maturity ladder (L1–L4)

An evaluation heuristic organized by escalating flow-control capability. The grades advance by which degradation effects are defended:

Grade	Flow-control trait (degradation defended)	Deliverable task	Run horizon (estimate)
L1 Single-Turn	No autonomous loop, no state. Driven by explicit user turn-taking.	Single-function generation, local explanation, syntax fix.	1 step / one interaction.
L2 Fixed Loop	ReAct-style loop but flow control is hardcoded; the agent perceives tool formats but has no exception handling or degradation. An unexpected error dead-loops or aborts. (Error compounding undefended.)	Single-file bug fix, simple feature in a clear context.	single-to-tens of steps.
L3 State Routing	The prompt becomes state-machine control language; structured exception handling and degradation are built in (test fails → trigger a reflection template). Error compounding partly defended.	Multi-file refactor, small independent module, initial test-driven.	tens of steps.
L4 Autonomous Closed-Loop	Deep runtime binding: proactively generates tests to self-verify, prunes memory to sustain token efficiency, anchors intent against drift, and an external state machine forces convergence via a circuit breaker. The first three degradation effects systematically defended (Specification Gaming is orthogonal — see §Orthogonal axis).	Multi-file defect fix or closed-loop feature, converged under the 3-degradation defense. Capacity vs demand: capacity = the 3-degradation defense (what this grade measures); demand = task fuzziness / codebase novelty / difficulty / run-horizon — demand is NOT capacity (a demanding run stress-tests capacity but does not define it; ADR #38).	hundreds of steps (demand — run-lifetime, not capacity).

Transitions: L2→L3 is hardcoded flow control → state-machine flow control; L3→L4 is passive exception handling → proactive runtime governance and self-verification. The run horizon is measured in steps (provider-normalized work units), not wall-clock: wall-clock = work / provider throughput, so it confounds the token provider and is NOT a capability signal (a slow provider must not make a run look "more L4"). Time appears only as a cost/hang guard. The step counts above are observed-interval estimates that drift with model capability — not authoritative thresholds. The run record reports steps + terminal_cause for exactly this reason (see convergent-loop.md § Event log + run record).

Self-assessment: where this skill sits

Verdict: L4 architecture / L3.5 operational — a snapshot to re-grade after a real long-horizon run. Every L4 mechanism the model names is present, and the circuit breaker is implemented harder than the model's wording ("external state machine forces convergence") — it is a hook-level physical intercept, not a prompted suggestion.

Evidence — each named L4 mechanism, present, where:

L4 mechanism (named by the model)	Present	Where
Deep runtime binding	yes	Hooks wired to real Edit/Write: fast_gate.py (PostToolUse), counters.py + blueprint_guard.py (PreToolUse). Opt-in install makes them hook-enforced, not advisory.
Proactive test self-verification	yes	TDD is the default (RED→GREEN per task; REFACTOR is the post-convergence tail); flaky stabilization + coverage threshold; gate附加条件 blocks "delete the failing test".
Memory pruning for token efficiency	yes	Context folding + loop_state.py summary; error fingerprints are normalized (shifting line numbers stripped) so the log stays compact.
Intent anchor against drift	yes	intent-blueprint.md frozen blueprint, three-layer read-only guard, diff-to-blueprint, hard rollback + reverse prompt.
External state machine forces convergence	yes — and physically	loop_state.py: priority breakers hard-terminate > escalate > degrade > suspend > ok (fingerprint ≥ N=3 → escalate; inner iteration ≥ M=8 → degrade, suspend if budget ≥80%; token/time/cost cap → hard-terminate). counters.py PreToolUse DENIES edits once the status is terminal — the loop cannot thrash past the breaker at all.

What holds the grade at L3.5 rather than "proven L4" (each is honest, not a knock):

1. Enabling the plugin is the L3→L4 lever; arming is the substance lever. The skill is usable two ways: methodology-only (no plugin — the agent follows SKILL.md and runs gates manually; advisory, loses determinism, ~L3) or plugin-enabled (Layer 1: the hooks fast_gate.py PostToolUse decision:block, counters.py PreToolUse DENY, the loop_state breaker fire on edits in the enabled project — that external enforcement is the L4 trait, "an external state machine forces convergence"). The hooks fire on ALL edits in the enabled project, not just pd invocations. /solidforge:arm-tools (Layer 2) is a separate lever: it provisions per-language arch-configs + gate dev-deps so the enforced tool-gates have substance. An enabled-but-unarmed project runs state-machine enforcement (counters / breaker / blueprint-guard) but the tool-based gates degrade to coverage notes until armed — never a silent green. See design-decisions.md §2 + §19 (project-scoped, not global hooks).
2. Unproven at probe-grade demand (ADR #38). No recorded run at probe-grade demand (novel codebase + long-horizon + unattended) yet. Capacity (3-degradation defense) is demand-independent and CAN be evidenced by self-edits (a self-edit that converges with the 3 defenses → l4-evidenced); but self-edit demand is light (familiar codebase) → weak evidence. This caveat is about demand-weighted evidence: does capacity still hold under the hardest demand? That stress-test requires an external novel codebase (self-edit is maximally familiar, so it can never be a probe-grade stress test). See extending.md § Self-application.
3. Default caps sit on the L3/L4 seam. cap_M=8 inner iterations and time_cap_W=1800s (30 min) land exactly at the L3 (≈10 min) ↔ L4 (20 min– hours) boundary — even the designed horizon straddles the line.
4. Budget is approximated and time is not a capability axis. Hooks cannot read Claude's real token usage, so tokens stay an estimate. Wall-clock is measurable, but it equals work divided by provider throughput, so it confounds the provider — it is a cost/hang guard only, not a capability signal. The provider-independent hard limit is therefore the step cap (step_cap_S, default 200 work units); a time/token/cost budget hit is judged inconclusive on capability, not a failure (design-decisions.md §6, §13). For a different-family (non-Anthropic) leg the cost figure's USD is itself a fiction — the Anthropic-compatible API returns tokens only (no price field), so CC's reported USD cannot reflect real provider spend; it is a runaway breaker, not an accounting instrument, and the real provider-independent bounds are step_cap_S + round count (ADR #42).
5. Gate coverage is uneven across platforms. Rust and Java have no first-class layer/dependency-direction enforcer (Java's Checkstyle ImportControl covers only declared package-layer rules); their gates degrade honestly via the coverage array and defer the rest to the outer ring (design-decisions.md §8). Go is the exception — a STRONG gate (the internal/ compiler-enforced boundary + golangci-lint depguard + compiler cycle rejection; ADR #36), so L4 coverage is platform-dependent for the thin gates (Rust/Java) but not for Go.
6. Design-fidelity coverage is advisory + external-gated. When a project arms an external design skill (Impeccable), its 44-rule deterministic detector is the design gate (per-edit advisory + convergence detect sweep); beyond those rules, visual fidelity + runtime a11y stay outer-ring (the reviewer visual line + /impeccable critique/audit). The detect findings' severity is inherited (advisory in practice). See external-skills.md.
7. API-contract ruleset coverage is advisory + external-gated. When a project arms Spectral, its spectral:oas + .spectral.yaml ruleset is the API-spec gate (convergence spectral_adapter.py sweep, advisory). It lints the SPEC's ruleset compliance — complementary to arch_contract_api.py (presence/freshness/path), NOT a replacement — and does NOT verify code matches spec (outer-ring / contract tests). Spectral's error/info severity levels collapse to warning (越权日志 schema enum is blocker|warning; advisory never blocker). See external-skills.md.
8. Source-SAST coverage is advisory + external-gated. When a project arms Semgrep, its ruleset (.semgrep/ / semgrep.yml, or --config auto) is the source-code SAST gate (convergence semgrep_adapter.py sweep, advisory) — the CVE-pattern-code axis (OWASP top-ten, injection, traversal, weak-crypto). Complementary to /security-review (LLM, semantic) and arch_contract_deps.py (secrets + dependency CVEs), NOT a replacement. SAST is false-positive-prone, so findings are advisory review input (never auto-Blocker); ERROR/WARNING/INFO collapse to warning (schema enum). Secrets-in-history + dependency CVEs stay arch-contract-deps. See external-skills.md.
9. Docs-prose-quality coverage is advisory + external-gated. When a project arms Vale (with a committed .vale.ini + styles/, or scaffolded via arm.py --scaffold-configs vale — which also runs vale sync), its ruleset is the prose gate (convergence vale_adapter.py sweep, advisory) — terminology/voice/spelling/inclusiveness. This fills the docs-QUALITY axis that blueprint-crafting (which checks upstream-doc STRUCTURE) and the language arch gates (which lint CODE) do not cover. Vale styles are opinion (no objective default — no config means a no-op, not a silent green); suggestion/warning/error collapse to warning (schema enum); technical accuracy / example correctness / link validity stay outer-ring. See external-skills.md.
10. API breaking-change coverage is advisory + external-gated. When a project arms oasdiff, the convergence oasdiff_adapter.py sweep diffs each tracked OpenAPI/Swagger spec against its git HEAD version and reports breaking changes (advisory) — the backward-compat axis, complementary to Spectral (spec style) and arch_contract_api.py (presence/path), neither of which diffs versions. Only specs tracked in git can be diffed (a brand-new spec has no base → skipped with an explicit note, not a silent green); findings are advisory (a breaking change may be intentional, e.g. a major-version bump) — never an auto-Blocker. See external-skills.md.
11. Dependency-license coverage is advisory + external-gated. When a project arms Trivy (license mode), the convergence license_adapter.py sweep inventories dependency licenses from lockfiles (advisory) — the legal/compliance axis, complementary to arch_contract_deps (secrets + CVEs, NOT licenses). Trivy is security-scanner-shaped (license is one mode); without a project policy (allow/deny list) the output is a raw inventory, not a verdict; per-license severity is advisory (license policy is opinion); copyleft/compatibility analysis is legal judgment → outer ring. See external-skills.md.
12. IaC-misconfig coverage is advisory + external-gated + opt-in. When a project arms Checkov, the convergence iac_adapter.py sweep scans Terraform/Kubernetes/Dockerfile for misconfig (advisory) — the infra-config axis. It no-ops cleanly when no IaC files are present (out of the app-language platform model platforms.json — an external-skill gate, NOT a platform). Findings are advisory (IaC misconfig is context-dependent, false-positives common); checkov output is file-level (line often absent → 0); runtime/cloud misconfig stays outer-ring. See external-skills.md.
13. Assertion-quality is a REMAINING GAP (test PRESENCE + execution COVERAGE are now defended; assertion SUFFICIENCY is not; different-family adversarial review is PARTIAL). arch_contract_tests.py (a) Blocks naked "delete the failing test" via the AC→test-name set gate — the frozen Intent Blueprint's AC→test mapping (declared at RED) is diffed against per-language collected test names, so a deleted mapped test is a test-name-missing Blocker (P1); (b) measures per-language execution coverage (pytest-cov / go test -coverprofile / cargo tarpaulin / vitest --coverage) and emits a coverage-below-threshold WARNING when the blueprint declares an NFR coverage floor — measure-only by default, WARNING not Blocker (coverage is execution, not assertion quality; P3, rule 4). What STILL stays a gap is assertion SUFFICIENCY (does the test actually catch bugs?) — mutation testing is the eventual engine-level different-family oracle and is heavy + per-language (mutmut / stryker / cargo-mutants), so it is NOT forced as an external-skill adapter; no engine fits the cross-cutting advisory-gate model (rule 3 — stated as a gap, not faked). NOTE (ADR #40): a PARTIAL different-family defense is now available at the ORCHESTRATION layer via infra/scripts/hetero_review.py (a cross-family adversarial review, additive over the same-family primary) — this is complementary to, NOT a replacement for, mutation testing: it catches a different category (enforcement-gaps / silent-failures / diagnostic-UX) per Phase 0, and is itself PARTIAL (commercial model families share training-data overlap).
14. Fast-gate format remediation is commit-stratified; the stratify discipline is heuristic. The fast-gate's FORMAT Blocker (format-emitting checks: Python ruff format, Java google-java-format, Go gofmt, Rust rustfmt; Swift/Web/Python fast-gate checks are LINTS) DETECTS deterministically (PostToolUse, formatter-diff non-empty), but the stratification itself — run the formatter, commit the pure-format change as a standalone style: commit, redo the logic edit — is model-executed/gate-nudged: heuristic, honestly labeled, never claimed a deterministic gate (commit-stratification.md). Under the documented-fallback Option A (advisory demotion at the convergence adapter, NOT implemented), Java/Go/Rust would lose their ONLY fast-gate check (their fast-gate checks are format-only; lint-grade checks live at the convergence gate) — that gap must be emitted as a coverage note, never silently greened.

Closing the gap to proven L4 (tracked follow-ups)

Caveat 2 above ("Unproven at probe-grade demand") is the demand-weighted-evidence gate on the grade (ADR #38: capacity can be self-evidenced; this caveat is about whether capacity holds under probe-grade demand — the hardest stress test). It is the only one of the five that can be retired by work rather than by re-derivation. Retire it by completing all four, then re-grade this page. The run-record's l4_assessment block (see convergent-loop.md § Event log + run record) is the instrumented mirror of this rubric — it computes the provisional verdict from a real run's evidence, so no item below needs manual recording.

• Enable the Solid Forge plugin in a real target project (Layer 1) and run /solidforge:arm-tools (Layer 2) and confirm hook-enforcement live (PreToolUse DENY fires; PostToolUse decision:block fires) — converts the envelope from advisory (L3) to enforced (L4). Verification path: see hooks-reference.md § Live verification.
• Run one real long-horizon task unattended as an L4 probe: at init declare the task descriptor (--codebase-novelty novel --req-clarity fuzzy --difficulty high --attended false --target-horizon-steps 60) on a low-contamination unfamiliar codebase, run unattended (>60 steps), then at the terminal status emit loop_state.py run-record. The descriptor declares demand (fuzzy/novel/high/horizon — NOT capacity criteria, ADR #38); the probe is a demanding run that stress-tests capacity. The record's l4_assessment block normalizes outcome, step count, budget, and every breaker firing / outer verdict / rollback with its trigger. provisional_verdict: l4-evidenced is the capacity signal (3-degradation defense under convergence); a probe-grade + long-horizon + capacity-met run is the strongest L4 evidence, but capacity itself is demand-independent.
• Add an autonomy-style eval to evals/evals.json — assert the loop converges, not merely that the skill activates (the current six cases test triggering only).
• Re-grade this self-assessment after the probe-grade run. If capacity holds under probe-grade demand, strike caveat 2 (demand-weighted evidence met) and move the verdict from "L3.5 operational" to "proven L4". (Note: a self-edit run CAN produce l4-evidenced on capacity alone, but cannot retire this caveat — it lacks probe-grade demand.)

Using this page as a self-review gate (sibling of extending.md)

When you add or change any convergence-loop mechanism, answer two questions before merging:

1. Which degradation effect does it serve (Context Rot / Error Compounding / Goal Drift / Specification Gaming)? A mechanism that serves none is dead weight. Note: Specification Gaming's defense is orthogonal (different-family oracle) — a same-family mechanism (self-test, same-model reviewer) does NOT defend it even if it claims to.
2. Does it preserve or raise the grade — or silently lower it?

Regression red lines (removing any one drops a level — do not):

• The state-machine circuit breaker (loop_state.py priority breakers).
• The terminal-state deny hook (counters.py PreToolUse DENY).
• Context folding at the inner→outer transition.
• The frozen intent anchor + diff-to-blueprint + hard rollback (intent-blueprint.md).
• TDD as the default (RED→GREEN per task; REFACTOR as the post-convergence tail, ADR #56).
• The dual ring (deterministic inner + semantic outer).

Definition of done for a maturity-preserving change: the red lines above are all intact, this self-assessment table still matches the code on disk, and infra/test/disconnect_check.py is green.

Industrial calibration anchors (2026-06, re-verify before citing)

Background only — these calibrate where the frontier is; they are not a target for this skill. Treat all numbers as time-stamped and re-verify before reuse (the field iterates monthly):

• SWE-bench Verified is saturated and contaminated. Self-reports reach ~93.9% (Claude Mythos, Anthropic), but under the standardized mini-SWE-agent harness frontiers top out around ~76.8% (Claude 4.5 Opus); an audit found ~59.4% of test cases defective, and OpenAI deprecated it as an evaluation. Do not read "Verified 90%+" as a robust capability signal.
• SWE-bench Pro (the recommended successor) represents the real difficulty — unfamiliar, multi-module, under-documented. Frontiers sit around 50–55% (GPT-5.3-Codex reproduced up to 54.0%). This is the more credible reference for "real resolution rate on a low-contamination unfamiliar codebase."
• Per-instance cost under mini-SWE-agent is ~$0.36–0.96 (official leaderboard, 2026-02), a rough proxy for "API cost of one defect fix." Real complex autonomous tasks cost far more (ACU-billed) — economy, not loss of control, is now the binding constraint on unbounded runs.
• Run horizon: long-horizon software-engineering tasks commonly need tens to hundreds of environment interactions (Cognition); public records exist of ~5-hour autonomous runs. No industry standard step count; "60–100 steps without drifting on a multi-file refactor" is observable but non-standardized.

Provenance & recency

• Provenance (private upstream). The framework on this page originates from the ai-coding-agent-maturity page in a private internal wiki (the fedaot-wiki). This page is the self-contained public reference — a publication port that carries the framework inline so it reads correctly without the upstream. Maintainers: re-sync this port from the fedaot-wiki when the upstream changes.
• The L1–L4 ladder is an evaluation heuristic synthesized from the CSA "Levels of Autonomy for Agentic AI" (2026-01) gradient idea, not an industry-mandated standard. Run horizons are observed-interval estimates.
• Benchmark data is pinned to 2026-06 and drifts monthly — re-verify before citing.
• The self-assessment is a snapshot: re-grade it after any change to the red-line mechanisms above, and after the first real long-horizon run that would let claim 2 (unproven at L4 scale) be retired.