System Improvement Pattern

Use this pattern when the target of change is the agent or harness itself: prompting, Skill guidance, tool routing, graders, retry/stop policy, evaluation, cost, or observability. Product work can be an inner goal-completion loop, but this outer loop asks whether a versioned system change improves later outcomes.

Composition

Slot	Contract
Trigger	Failed or stalled agent run, repeated human rework, tool-routing error, grader disagreement, cost/latency anomaly, regression, or bounded periodic trace review.
Procedure Owner	Trace or failure analysis, eval-case construction, procedure refinement, routing calibration, stall detection, grader calibration, or controlled comparison. Use a Skill only when selected as the durable procedure owner.
Tools / Access	Sanitized traces and run logs, review evidence, eval runner, regression suite, holdout data, version control, token/latency records, and provider observability or connector.
Artifact	Failure taxonomy, reproducible eval case, versioned prompt/Skill/tool/grader change, comparison report, improvement review, or rollback decision.
Verifier	Baseline and candidate comparison on predeclared primary metric plus guardrails, holdout or later comparable tasks, regression checks, cost per success, and human calibration where judgment matters.
State	Baseline version, candidate version, dataset and selection version, experiment window, metrics, guardrails, review decision, release or rollback state, and unresolved causes.
Stop Rule	Adopt only when the declared improvement clears its threshold without guardrail regression; otherwise retain, revise, or roll back. Stop with insufficient evidence rather than claiming effectiveness.

Flow

1. Bound the trace, review, or evaluation window and sanitize sensitive data.
2. Separate observed failures from hypotheses and cluster only comparable runs.
3. Convert a production or review failure into a stable eval or regression case before optimizing against it.
4. Version the baseline, candidate, data selection, primary metric, guardrails, and stop/revert condition.
5. Change one owned surface or a clearly declared bundle.
6. Compare against holdout or a fair later window; inspect regressions and cost, not only the target metric.
7. Route the change through review, then adopt, retain for more evidence, or roll back. Continue monitoring after adoption.

Improvement Targets

Target	Candidate procedure shape	Useful check
Failure understanding	trace-failure-analyzer, review-feedback-miner	Classification explains inspected failures and preserves unknowns.
Evaluation coverage	eval-case-builder, regression-suite-curator	The case reproduces the original failure and remains stable.
Procedure quality	skill-refiner, prompt or reference refinement	Comparable task success improves without new omissions.
Tool selection	tool-routing-optimizer	Invalid or wasteful tool calls fall without reducing success.
Loop control	loop-stall-detector, retry/stop calibration	Attempts, latency, or cost fall without hiding failures.
Grading	grader-calibration, skill-ab-evaluator	Automated judgment aligns better with held-out human review.

Candidate names describe capability shapes; they are not an installed-Skill inventory.

Scenario Sketches

• Self-improving procedure: repeated instructions or mistakes become a bounded trigger, reusable procedure, validation, stop rule, and owner; a current-task success is readiness evidence, not later effectiveness.
• Production failure to eval: preserve the smallest sanitized inputs that reproduce the failure and add the case before changing the system.
• Review rework mining: group explicit human feedback by root cause and compare later rework on a declared cohort.
• Grader calibration: compare grader and human decisions on a held-out set; document disagreements instead of tuning only to the reviewed sample.
• Cost or stall control: optimize cost per successful outcome and keep success, safety, and quality guardrails visible.

Compact Example

# Issue-Triage Procedure Improvement

When: A bounded sample shows repeated false positives or missed actionable
issues from the current triage procedure.

See: Versioned procedure, sanitized labeled examples, actual decisions,
reviewer corrections, baseline metrics, and existing eval coverage.

Do: Add stable regression cases, change one owned procedure or rubric, and run
a predeclared comparison. Do not tune against private raw traces or the test set
alone.

Check: Precision and recall on holdout or a fair later cohort improve while
noise, safety, and cost guardrails do not regress.

Stop: Adopt after threshold and review evidence pass; otherwise keep the
baseline, revise the candidate, or stop with insufficient evidence.

Leave: Baseline and candidate versions, dataset/selection version, metrics,
guardrails, review decision, and rollback reference.

Effectiveness Boundary

Learning Loop Detection Patterns may produce Opportunity or Readiness candidates from bounded episodes. Only the existing Learning Capture intervention-ledger contract can support an effectiveness claim from a comparable later or held-out result without guardrail regression. This pattern must not create a second effectiveness ledger or call the current task's tests proof of future improvement.

Use Goal Completion for the inner versioned change and Loop State Ledger only for the smallest replayable experiment state. Return to the pattern index or Loop Discovery when the intervention owner is unclear.