Multi-Shot Optimization

Renamed. runMultiShotOptimization was retired. The live API is runImprovementLoop (proposer-agnostic, gated promotion) driven by gepaProposer, with compareProposers for head-to-head proposer lift. This doc was rewritten to the live API; see also feature-guide.md and concepts.md.

runImprovementLoop is the public entry for GEPA-style optimization over a whole task trajectory — the thing you improve is not a single model call but an agent system prompt, tool descriptions, a routing policy, or any scaffolding that affects the entire run. It is the OUTER loop: it improves the SURFACE the inner workers run.

The shape

You own a few seams; the loop owns the release-critical glue (paired seeds, the held-out re-score, the promotion gate, provenance):

• baselineSurface — the current surface (a prompt string, or a CodeSurface).
• dispatchWithSurface(surface, scenario, ctx) — run one task to completion under a candidate surface; return the artifact the judges score.
• judges — score the artifact ({ composite, dimensions }).
• proposer — proposes candidate surfaces each generation: gepaProposer (reflective + Pareto frontier) or evolutionaryProposer (mutator).
• gate — defaultProductionGate (held-out significance + red-team + reward-hacking + canary). Ships ONLY on a CI-lower-bound held-out lift.

Minimal example

import {
  runImprovementLoop,
  gepaProposer,
  defaultProductionGate,
} from '@tangle-network/agent-eval/contract'

const result = await runImprovementLoop({
  baselineSurface: currentSystemPrompt,
  scenarios: trainScenarios, // optimizer-visible
  holdoutScenarios, // DISJOINT — only the gate sees these
  dispatchWithSurface: async (surface, scenario) =>
    runYourAgentToCompletion({ scenario, prompt: String(surface) }),
  judges: [myJudge],
  proposer: gepaProposer({
    llm: { apiKey, baseUrl },
    model: 'gpt-5',
    target: 'enforce a strict output schema',
  }),
  populationSize: 4,
  maxGenerations: 4,
  gate: defaultProductionGate({ holdoutScenarios, deltaThreshold: 0 }),
  autoOnPromote: 'none', // or 'pr' (+ ghOwner/ghRepo) to open a PR on ship
  runDir,
})

if (result.gateResult.decision === 'ship') {
  deploy(result.winnerSurface) // the proposer's candidate, gated on a real held-out lift
}

Discipline (what makes it trustworthy)

• Holdout is disjoint + gated. holdoutScenarios must not overlap the training pool. The gate re-scores baseline vs winner on the holdout and ships only when the paired-bootstrap CI lower bound clears deltaThreshold; a few-instance swing at thin n is held (few_runs), not promoted.
• No-op never ships. If no candidate beats the baseline, the winner IS the baseline (empty diff) and the loop forces hold — it does not score baseline-vs-itself and read model noise as lift.
• Provenance falls out. result.promotedDiff + emitLoopProvenance give the auditable candidate→gate→promote chain (rationale, content hashes, a held-out lift recomputable from the emitted record).

Reach for compareProposers when the question is "which proposer wins" rather than "improve this surface", and see tests/campaign/presets.test.ts for the executable contract (no-op guard, fail-loud holdout, gate promotion).