Evaluation

The @open-multi-agent/core/eval subpath measures agent and multi-agent quality offline or through best-effort online sampling. Evaluation observes completed results; it never changes the business result.

Use runtime verification and evaluation for different jobs:

	runConsensus() / per-task verify	Evaluation
When	During one business run	Offline in batches or asynchronously after a live run
Changes the business result	Yes: may accept, revise, or reject	No
Measures	One result	Cases, versions, regressions, and trends
Failure meaning	Affects the runtime verdict	Produces scorer_error; never becomes score zero
Output	ConsensusResult	EvalRecord, EvalRunReport, and GateVerdict

The two mechanisms compose: verification protects a single run, while an EvalSet can detect changes in verification pass rate over time.

Scorers

import { defineScorer, type ScorerContext } from '@open-multi-agent/core/eval'

const exact = defineScorer({
  name: 'exact-match',
  version: '1',
  score({ output, evalCase }) {
    const hit = output === evalCase.expected
    return { score: hit ? 1 : 0, pass: hit }
  },
})

const context: ScorerContext = {
  evalCase: { id: 'capital-france', input: 'Capital of France?', expected: 'Paris' },
  output: 'Paris',
  metadata: { promptVersion: 'v2' },
  signal: new AbortController().signal,
}

const result = await exact.score(context)
console.log(result.score) // 1

Scores must be finite numbers from 0 through 1. pass is optional so a later gate can apply its own threshold. defineScorer() freezes the scorer definition and validates both synchronous and asynchronous results. A scorer may omit version, but OMA warns once per scorer name because a gate then cannot distinguish scoring-logic drift from target drift. Bump the version whenever a rule, prompt, judge model, or judge configuration changes.

Scorer failures are not zero scores

A scorer that throws, rejects, or exceeds its timeout has not measured quality. runEvalSet() records that outcome as an EvalRecord with status: 'scorer_error', normalizes the error, continues to later scorers, and excludes the failure from score averages, percentiles, and pass rates. Do not replace scorer failures with { score: 0 }.

If the target itself throws, the sample produces one target_error record under the reserved scorer name _target; its scorers do not run. The eval subpath also defines the EvalRecord shape and schema major version.

Reference scorers

Reference scorers are deliberately small examples, not universal quality standards. Import them from @open-multi-agent/core/eval:

Factory	Score meaning	Required data and missing-data behavior
toolCallSuccessScorer()	Successful tool spans / all tool spans	Uses trace status when available. A result-only ToolCallRecord has no error flag, so completed result calls are treated as successful. No calls returns 1 with details.tool_calls = 0 and an explicit not-applicable reason.
structuredOutputComplianceScorer(schema?)	1 when AgentRunResult.structured exists and, when supplied, passes the Zod schema	Intended only for targets whose agent config has outputSchema. Missing structured output is a measured failure (0), not missing infrastructure.
costBudgetScorer({ maxTokens?, maxCostAmount? })	Hard step: 1 within every observable ceiling, otherwise 0	Tokens come from the OMA result; cost comes from StoredRun.costs. Unavailable dimensions are named in reason and details.data_complete; no observable dimension returns 1 with applicable: false. Multiple currencies throw and become scorer_error instead of being added incorrectly.
dependencyUtilizationScorer()	Completed dependency-bearing task spans / all dependency-bearing task spans	Requires a trace. This is a conservative dependency-chain completion proxy; it proves linked prerequisites and the dependent task completed, not that the model semantically used prerequisite text.
duplicateWorkScorer({ threshold? })	1 - duplicatePairs / comparedPairs	Requires both a trace and TeamRunResult. Trace identifies task IDs; actual outputs come from agentResults. Similarity is Jaccard over normalized character trigrams. Fewer than two outputs returns explicit not-applicable 1.
noProgressScorer({ maxStallTurns? })	1 within the allowed consecutive stalls; above it, maxStallTurns / observedMaximum	Requires a trace. A stall is a failed task-agent attempt with LLM work, no tool call, and no completed task. This measures agent attempts, not semantic reasoning turns.
createAnswerRelevancyScorer({ judges, ... })	Mean judge score for direct relevance to input and expected output	Thin createJudgeScorer() wrapper with a fixed { score, reason } schema. Treat it as a prompt template to version and validate against your own data.

The three structure-aware scorers expose behavior specific to a multi-agent DAG, but remain privacy-aware: traces do not persist task output bodies. Dependency utilization and no-progress therefore use honest structural proxies, while duplicate-work reads outputs from the in-memory TeamRunResult and uses the trace only to select task executions. Use these three with offline runEvalSet(..., { traceStore }); online sampling does not load a trace for a scorer.

import {
  createAnswerRelevancyScorer,
  toolCallSuccessScorer,
} from '@open-multi-agent/core/eval'

const scorers = [
  toolCallSuccessScorer(),
  createAnswerRelevancyScorer({
    version: 'relevancy-prompt-v1',
    judges: [{ name: 'judge', model: 'claude-sonnet-4-6', provider: 'anthropic' }],
  }),
]

Quick start: offline evaluation in five minutes

import {
  defineEvalSet,
  defineScorer,
  runEvalSet,
  type EvalTarget,
} from '@open-multi-agent/core/eval'

const set = defineEvalSet({
  name: 'greetings',
  version: '1.0.0',
  cases: [
    { id: 'a', input: 'hi', expected: 'HI', tags: ['upper'] },
    { id: 'b', input: 'yo', expected: 'YO', tags: ['upper'] },
  ],
  defaults: { concurrency: 2 },
})

const target: EvalTarget = async (input) => ({
  output: String(input).toUpperCase(),
})

const exact = defineScorer({
  name: 'exact',
  version: '1',
  score({ output, evalCase }) {
    const pass = output === evalCase.expected
    return { score: pass ? 1 : 0, pass }
  },
})

const report = await runEvalSet(set, target, {
  scorers: [exact],
  repeats: 2,
  metadata: { prompt_version: 'v2' },
})

console.log(report.records.length)          // 4
console.log(report.aggregates[0]?.avg)      // 1
console.log(report.aggregates[0]?.passRate) // 1

defineEvalSet() validates non-empty names, versions, and cases; requires case IDs to be unique; and returns a deeply frozen copy. Treat version as the content version and bump it whenever cases change. filterTags selects cases matching any requested tag. repeats and concurrency override the set defaults.

Each case/repeat target runs once, then that sample's scorers run serially. Different samples run in parallel up to concurrency (default 2). Aborting stops new samples from being scheduled, waits for already-started samples, and returns a partial report with aborted: true.

Report percentiles use the nearest-rank method. For two sorted scores, p50 is the lower score and p95 is the higher score. passRate only includes scored records that explicitly contain pass; scorer errors are excluded from every score denominator. byTag repeats the same aggregation for each case tag. Target token usage is counted once per sample, even when multiple scorers run, and costs are summed only within the same currency.

Sample production runs online

Online evaluation is opt-in on OpenMultiAgent. A settled top-level run only performs a synchronous sampling decision and bounded queue admission; scorers and store writes run later and never change the business result.

import { OpenMultiAgent } from '@open-multi-agent/core'
import {
  InMemoryEvalStore,
  defineScorer,
} from '@open-multi-agent/core/eval'

const onlineStore = new InMemoryEvalStore()
const lengthScorer = defineScorer({
  name: 'length',
  version: '1',
  score({ output }) {
    const length = String(output).length
    return { score: Math.min(1, length / 200), pass: length >= 40 }
  },
})

const orchestrator = new OpenMultiAgent({
  evaluation: {
    scorers: [lengthScorer],
    sample: 0.05,
    maxConcurrent: 1,
    maxQueueLength: 100,
    budget: { maxEvaluationsPerMinute: 30 },
    store: onlineStore,
  },
})

const run = await orchestrator.runAgent(agent, prompt)
// The business result does not wait for lengthScorer or onlineStore.
console.log(run.success)

await orchestrator.evaluation.forceFlush({ timeoutMs: 1_000 })
const page = await onlineStore.query({ runId: [run.identity!.runId] })
console.log(page.items[0]?.source) // online

runAgent, runTeam, runTasks, runFromPlan, runConsensus, and restore all use the same evaluator owned by the OpenMultiAgent instance. Its evalRunId therefore remains stable for that instance lifetime. Each sampled run produces one EvalRecord per scorer with source: 'online', no EvalSet or case ID, and a runRef containing the exact logical run and attempt.

Numeric sampling uses Math.random() < sample. A rule can select by normalized status and validated run metadata without implementing tail sampling:

const failuresOnly = new OpenMultiAgent({
  evaluation: {
    scorers: [lengthScorer],
    sample: (context) =>
      context.status.code !== 'ok'
      && context.metadata['deployment'] === 'canary',
    store: onlineStore,
  },
})

A throwing sampling rule is treated as false and diagnosed. A scorer throw, rejection, or timeout produces a scorer_error record. A rejected store append drops that sample's record batch. Queue overflow, exhausted budgets, callbacks, and all evaluation failures are isolated from the original run result.

Online defaults are deliberately conservative: evaluation is off when the configuration is omitted or sample is 0; maxConcurrent is 1, maxQueueLength is 100, payload persistence is none, diagnostics warn at most once per code per 60 seconds, and there is no implicit rate or cost cap. diagnostics: 'silent' must be explicit. getStats() returns cumulative sampled, enqueued, completed, dropped, failed, and storeFailed counts.

maxEvaluationsPerMinute counts scorer evaluations, so one sampled run with three scorers consumes three units. maxCostPerHour uses the caller's existing OrchestratorConfig.estimateCost function and the model usage surfaced by framework-backed scorers such as createJudgeScorer. The cap uses the same caller-defined unit returned by estimateCost, can overshoot by currently running scorer work, and resumes as the rolling hour advances. Rule scorers without model usage cost zero. Custom model-backed scorers cannot be costed unless they use a framework scorer that reports its internal usage. Configuring maxCostPerHour without estimateCost leaves the cap inactive and emits one payload-free warning; it never silently blocks runs.

storePayloads: 'none' gives scorers a content-free run-input description and does not persist input or output. 'redacted' gives scorers and the record a bounded, redacted input string and persists a bounded, redacted output; 'full' does the same without redaction and must be an explicit privacy decision. Scorers always receive the candidate output in memory so they can score it. In particular, a judge scorer sends that output to its configured model provider; do not enable an external judge for data that may not leave its trust boundary.

Lifecycle ownership

The application owns evaluator lifecycle. OMA installs no signal handlers and does not call process.exit(). All evaluator timers are unreferenced, so they do not keep a CLI or serverless process alive. This also means a process crash or natural exit can lose queued work: the first implementation is in-process, best-effort, and intentionally not durable. A durable or cross-process scoring queue is a separate future integration, not an EvalStore guarantee.

// Serverless/FaaS: flush this invocation; keep a shared singleton usable.
const result = await orchestrator.runAgent(agent, prompt)
const evaluation = await orchestrator.evaluation.forceFlush({ timeoutMs: 1_500 })
return { result, evaluation: evaluation.status }

// Short-lived CLI: settle accepted samples before natural process exit.
try {
  await main()
} finally {
  await orchestrator.evaluation.forceFlush({ timeoutMs: 5_000 })
  await orchestrator.evaluation.shutdown({ timeoutMs: 5_000 })
}

// Long-lived server: stop traffic, then drain and close on graceful shutdown.
async function stopServer() {
  await stopAcceptingAndWaitForInflight(server)
  await orchestrator.evaluation.forceFlush({ timeoutMs: 10_000 })
  await orchestrator.evaluation.shutdown({ timeoutMs: 10_000 })
  await provider?.shutdown()
}
// Register stopServer with your server/process framework if desired.

forceFlush() waits for the samples accepted before its watermark and returns ok, partial, timeout, or error plus cumulative counts. shutdown() atomically rejects new samples, flushes its cutoff, and is idempotent: repeated or concurrent calls share the first result. OpenMultiAgent.shutdown() remains the existing team-registry reset; evaluator shutdown is explicit through orchestrator.evaluation.

On the online-evaluation maintainer benchmark (Node 22, 50,000 same-process direct admissions), sampling plus bounded enqueue measured approximately 0.42 µs p95 (0.30 µs mean) on the implementation host. Absolute microseconds vary by host; CI additionally retains the existing observability same-host regression gate for the unconfigured path.

EvalStore

Use InMemoryEvalStore for short-lived local runs, tests, or an adapter prototype. Pass it to runEvalSet() to persist one atomic batch per completed case/repeat sample:

import {
  InMemoryEvalStore,
  runEvalSet,
} from '@open-multi-agent/core/eval'

const store = new InMemoryEvalStore()
const storedReport = await runEvalSet(set, target, {
  scorers: [exact],
  store,
})

const first = await store.query({
  evalRunId: storedReport.evalRunId,
  scorer: ['exact'],
  order: 'time_asc',
  limit: 100,
})

EvalStore.append() is atomic per batch and idempotent by recordId. Queries can filter by evaluation run, referenced OMA run, EvalSet name, scorer, source, status, and inclusive after / exclusive before timestamps. Results use the stable (timestampUnixMs, recordId) order. The default page limit is 100 and the maximum is 1,000.

Cursors are opaque snapshots. Appends after the first page do not create gaps or duplicates in that pagination sequence. A cursor is valid only for the same store instance and normalized query; changing filters, deleting records, or reopening a file store invalidates it. Do not parse or persist cursors as data.

The optional InMemoryEvalStore({ maxRecords }) capacity is a hard limit. A batch that would exceed it is rejected atomically. Use retention explicitly when eviction is intended:

await store.applyRetention({
  maxAgeMs: 30 * 24 * 60 * 60 * 1_000,
  maxRecords: 10_000,
  sources: ['offline'],
})

await store.delete({
  evalSetName: 'greetings',
  before: new Date('2026-01-01T00:00:00.000Z').toISOString(),
})

Deletion and retention are idempotent. In their shared DeleteResult, runIds contains affected evalRunId values, runsDeleted counts distinct affected evaluation runs, and recordsDeleted counts records. A sources retention scope applies both age and count limits only to those sources; when it is the only field, all records in the selected sources are deleted.

For durable local storage, import the Node-only implementation separately:

import { FileEvalStore } from '@open-multi-agent/core/eval/file'

const fileStore = await FileEvalStore.open('./eval-results/history.ndjson', {
  onDiagnostic(diagnostic) {
    console.warn(diagnostic.code, diagnostic.message)
  },
})

await fileStore.append(storedReport.records)
await fileStore.flush()
await fileStore.compact()
await fileStore.close()

FileEvalStore is a single-process reference implementation, not a production database or a cross-process coordination layer. It keeps an append-only, schema-versioned NDJSON mutation log and rebuilds its in-memory index on open. A committed batch is visible in full or not at all; a process or machine crash can lose at most the last, not-yet-durable batch. flush() is the explicit fsync boundary. Recovery truncates only an incomplete final line or batch and emits a diagnostic; complete corruption fails loudly.

Compaction writes <file>.compact.tmp, fsyncs it, atomically renames it over the target, and then fsyncs the parent directory where supported. A stale temp file never overrides an existing target. Use a database-backed EvalStore adapter when multiple processes, large data volumes, or server-side aggregation are required.

Stores preserve unknown fields within the supported schema major so future minor additions survive a round trip. A higher schemaVersion major is rejected rather than downgraded. There is intentionally no aggregation method on EvalStore: calculate trends from queried records in memory. Needing aggregation pushdown is a signal to introduce a database adapter, not to add file-specific concepts to the interface.

Persistence is fail-open for the evaluation run. If a sample batch cannot be stored, runEvalSet() still returns its complete records and aggregates and adds one payload-free entry to report.warnings for that sample.

Evaluate OMA runs

Use the convenience targets when the system under evaluation is an OMA agent, team, or fixed plan:

import { Team, type AgentConfig, type PlanArtifact } from '@open-multi-agent/core'
import {
  targetFromAgent,
  targetFromPlan,
  targetFromTeam,
} from '@open-multi-agent/core/eval'

declare const agent: AgentConfig
declare const team: Team
declare const plan: PlanArtifact

const agentTarget = targetFromAgent(agent, {
  metadata: { prompt_version: 'v2' },
})
const teamTarget = targetFromTeam(team)
const planTarget = targetFromPlan(team, plan)

void agentTarget
void teamTarget
void planTarget

Agent and team targets convert non-string input with String(input) and use it as the prompt or goal. Plan targets replay the supplied PlanArtifact; the plan fixes the tasks and goal. These wrappers return the OMA result alongside the primary output, inject eval_case and one-based eval_repeat run metadata, and add available model/provider fingerprints. The runner uses the result identity for runRef and the result usage for report totals. When traceStore is provided, it also loads the matching StoredRun into ScorerContext.trace.

Record metadata merges in this order, with later values winning: case metadata, runEvalSet() metadata, then metadata echoed by a convenience target (including its configuration fingerprint).

Load EvalSets and write reports

File I/O is isolated in the Node-only @open-multi-agent/core/eval/file subpath. The root package and @open-multi-agent/core/eval do not import this entry point.

import {
  loadEvalReport,
  loadEvalSet,
  loadGatePolicy,
  writeEvalReport,
} from '@open-multi-agent/core/eval/file'

const setFromJson = await loadEvalSet('./evals/greetings.json')
const fileReport = await runEvalSet(setFromJson, target, { scorers: [exact] })

await writeEvalReport(fileReport, { format: 'json', path: './report.json' })
await writeEvalReport(fileReport, { format: 'markdown', path: './report.md' })
await writeEvalReport(fileReport, { format: 'junit', path: './report.junit.xml' })

const policy = await loadGatePolicy('./evals/gate.json')
const baseline = await loadEvalReport('./evals/baseline.json')

loadEvalSet() parses JSON, applies the same validation and deep freezing as defineEvalSet(), and includes the resolved file path plus the first schema issue in validation errors. writeEvalReport() creates parent directories as needed and supports:

• json: the authoritative, pretty-printed EvalRunReport representation.
• markdown: metadata, scorer and tag aggregates, failed samples, and totals for human review. Long failure reasons are truncated.
• junit: one testcase per record. pass: false becomes <failure>; scorer_error and target_error become <error>; records without pass and without an error are successful testcases. XML names and messages are fully escaped.

loadGatePolicy() and loadEvalReport() validate their schema-versioned JSON contracts and report the resolved file path plus the first invalid field. Loaded objects are defensively copied and deeply frozen.

Run evaluations from the CLI

After building or installing the package, a no-network target can be evaluated from a shell or CI job:

oma eval run --set ./evals/greetings.json --target ./evals/target.mjs \
  --report json --report junit --out ./eval-results \
  --gate ./evals/gate.json --baseline ./evals/baseline.json \
  --meta prompt_version=v2

The target module must default-export an EvalTarget function or { target, scorers? }. An optional --scorers module default-exports a Scorer[]; scorer names must be unique across both sources. The CLI dynamically imports and executes these user modules with the current process permissions, so they must be trusted.

Reports are written below <out>/<evalRunId>/; --out defaults to ./eval-results. --report is repeatable and defaults to JSON. --meta key=value is also repeatable and all values are strings. See the CLI reference for the complete argument and exit-code contract.

Without --gate, low scores and pass: false records do not change the exit code. With --gate, the stdout summary includes the verdict and its path, and the exact { pass, failures, warnings } object is written to <out>/<evalRunId>/verdict.json. A failed gate or every selected target failing exits 1. Usage/file/module errors exit 2. --baseline requires --gate.

Gate quality in CI

evaluateGate() is pure logic exported from @open-multi-agent/core/eval:

import { evaluateGate } from '@open-multi-agent/core/eval'

const verdict = evaluateGate(report, {
  schemaVersion: 1,
  thresholds: [
    // A rule scorer plus passRate=1 is a deterministic quality gate.
    { scorer: 'exact', metric: 'passRate', min: 1 },
    { scorer: 'relevancy', metric: 'avg', min: 0.8 },
    { scorer: 'relevancy', metric: 'p50', min: 0.85, tag: 'critical' },
  ],
  maxScorerErrorRate: 0.1,
  maxTargetErrorRate: 0,
  baseline: {
    maxRegression: 0.05,
    perScorer: { exact: 0 },
  },
}, baseline)

if (!verdict.pass) process.exitCode = 1

The equivalent JSON policy is:

{
  "schemaVersion": 1,
  "thresholds": [
    { "scorer": "exact", "metric": "passRate", "min": 1 },
    { "scorer": "relevancy", "metric": "avg", "min": 0.8 },
    { "scorer": "relevancy", "metric": "p50", "min": 0.85, "tag": "critical" }
  ],
  "maxScorerErrorRate": 0.1,
  "maxTargetErrorRate": 0,
  "baseline": {
    "maxRegression": 0.05,
    "perScorer": { "exact": 0 }
  }
}

Thresholds support avg, p50, p95, min, and passRate, with optional tag scoping and inclusive min/max boundaries. A missing scorer, tag, or passRate source is a configuration failure rather than a silent pass. The health defaults fail when scorer errors exceed 10% of scored plus scorer-error records, or when any selected target fails.

Every verdict contains only pass, failures, and warnings. A failure has a stable kind, optional scorer/metric/tag coordinates, the observed actual, the configured limit, and a human-readable message. For failures about availability rather than a measured score, missing_scorer uses actual: 0 and limit: 1, while baseline_mismatch uses actual: 1 and limit: 0.

A baseline is an ordinary JSON EvalRunReport, not a second file format. The recommended workflow is:

1. Run the accepted target with --report json and copy its report.json to a reviewed location such as evals/baseline.json.
2. Commit that report together with its versioned EvalSet and gate policy.
3. In CI, compare the candidate report with --baseline.
4. Update the baseline only after intentionally reviewing and accepting the behavior change; the CLI never updates it automatically.

Set name or version mismatches fail by default. Set baseline.allowSetMismatch to true only when a warning plus skipped regression checks is intended. When a scorer version differs, OMA warns and skips that scorer's regression checks because a changed judge prompt or model does not produce a comparable score. Threshold and health checks still run. If baseline rules are configured but no baseline report is supplied, OMA warns and skips regression checks.

Use oma eval gate when report generation and quality enforcement are separate CI stages. It prints the exact verdict JSON to stdout:

oma eval gate --report ./candidate/report.json --gate ./evals/gate.json \
  --baseline ./evals/baseline.json

A GitHub Actions job can preserve both machine-readable reports while letting the gate control the step status:

- name: Run deterministic evaluation gate
  run: |
    oma eval run --set ./evals/set.json --target ./evals/target.mjs \
      --gate ./evals/gate.json --baseline ./evals/baseline.json \
      --report json --report junit --out ./eval-results || exit 1

- name: Upload evaluation JUnit report
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: evaluation-junit
    path: eval-results/**/report.junit.xml

Routing stability regression EvalSet

The frozen run-team-routing-stability@1.0.0 EvalSet in packages/core/tests/fixtures/eval/routing-stability-set.json measures whether equivalent runTeam() goals keep the same executed topology when prompt length or language changes. Each family contains short and detailed English variants plus Chinese, Japanese, and Korean translations. Governance families carry the same governanceIntent: 'required', requiredRoles, and requiredOrder declaration on every variant; benign families carry no declaration.

The test injects one deterministic LLMAdapter into every worker and the coordinator. Every model call returns the same fixed text, including a valid two-role coordinator plan. It also injects a valid low-risk TaskProfiler fixture for benign Single candidates and fails the test unless Hybrid profiling finishes with outcome: 'applied'; the gate therefore cannot pass by silently falling back from invalid profile output. The suite makes no network request and needs no API key. Within one family, the goal text is therefore the only routing input that changes. The measured topology comes from buildExecutionReceipt(result) and the result.tasks short-circuit marker, and contains only:

• single-short-circuit versus task graph;
• the worker roles that actually executed; and
• cross-role dependency edges.

Model output, generated task IDs, timing, token usage, and scheduler start order are excluded. For n variants, flip rate is the number of unordered variant pairs with different canonical topologies divided by n * (n - 1) / 2. Length invariance compares the fixture's short/detailed English pair; language invariance compares its explicitly paired English/Chinese variants.

packages/core/tests/fixtures/eval/routing-stability-gate.json applies three absolute thresholds to the governance tag: routing-stability minimum 1 (zero flips), length-invariance minimum 1, and language-invariance minimum 1. It also gates benign routing-stability at 0.95 (at most 5% pair flips); benign length/language submetrics remain monitored. Scorer and target error limits are both zero. The existing CI npm test matrix blocks a change that makes a declared route depend on goal language or length or pushes benign topology flips over the limit. A negative-control test injects a fake declared router that collapses Chinese variants to one role and asserts that both the routing-stability and language-invariance thresholds fail.

The current snapshot below is emitted in the test's [routing-stability] EvalSet report:

Family	Pair flips	Flip rate	Length invariant	Language invariant
Declared wire transfer	0 / 10	0%	100%	100%
Declared key rotation	0 / 10	0%	100%	100%
Declared governance total	0 / 20	0%	100%	100%
Undeclared DNS	0 / 10	0%	100%	100%
Undeclared database comparison	0 / 10	0%	100%	100%
Undeclared benign total	0 / 20	0%	100%	100%

The target for undeclared benign routing is at most 5% pair flips and at most 5% length mismatches (at least 95% length invariance). Update the frozen corpus version and the documented snapshot only after reviewing an intentional measurement change.

Semantic routing policy EvalSet

packages/core/tests/fixtures/eval/semantic-routing-set.json freezes the V1 Hybrid policy contract independently from provider behavior. It covers short independent-evidence and independent-review goals, permission isolation, consequential side effects, conflicting objectives, ordinary short and long single-task negatives, equivalent Chinese/Japanese/Korean goals, and a prompt-injection sample.

Each fixture contains a reviewed TaskProfile, framework-computed facts, and the expected deterministic recommendation. CI validates the strict profile schema and requires every fixture to match exactly. This proves Policy behavior without treating one provider's current classification as the semantic contract. Real-provider E2E is limited to checking that the one-call Profiler integration can produce a valid profile; it is not the sole routing oracle.

Shadow evaluation belongs in CI and canary release work, not the public runtime default. Before promoting a major release, measure the reviewed end-to-end corpus against these gates:

• zero false-Single results on critical cases;
• at least 95% reviewed routing accuracy;
• at most 1% invalid/failed Profiler outputs;
• median Profiler token overhead no greater than 5% of representative total usage; and
• P95 end-to-end latency regression no greater than 10%.

Running the real-provider Shadow gate

packages/core/tests/e2e/semantic-routing-shadow.test.ts runs the reviewed synthetic corpus through the actual LLMTaskProfiler, then applies the same deterministic policy that Hybrid routing uses. It is skipped unless SEMANTIC_ROUTING_SHADOW=1 is set, executes no workers or tools, and does not send user data. The profiler uses the shipping LLMTaskProfiler default of 800 output tokens per fixture, so the canary validates the production contract. The optional SEMANTIC_ROUTING_SHADOW_BASE_URL and SEMANTIC_ROUTING_SHADOW_REGION values map to the selected adapter; Bedrock uses its normal AWS credential chain instead of the API-key variable.

For DeepSeek V4, the built-in profiler explicitly uses non-thinking mode: this is a bounded JSON classification call, and DeepSeek otherwise enables thinking by default. The production route and this gate use the same setting.

Set credentials only in your local shell or CI secret store; never paste a key into a command, repository file, or test log. For example, map an existing local provider secret without printing it:

export SEMANTIC_ROUTING_SHADOW=1
export SEMANTIC_ROUTING_SHADOW_PROVIDER=openai
export SEMANTIC_ROUTING_SHADOW_MODEL=gpt-4o-mini
export SEMANTIC_ROUTING_SHADOW_API_KEY="$OPENAI_API_KEY"
npm run test:semantic-routing-shadow -w @open-multi-agent/core

The report contains only provider/model identifiers, aggregate accuracy, failure and mismatch case IDs, P95 profiler latency, and aggregate token use. It deliberately omits goals, inferred reasons, raw model output, and all configuration secrets. The executable gate enforces zero invalid profiles, zero critical false-Single outcomes, and at least 95% reviewed routing accuracy. Record the output as release evidence. Measure the remaining end-to-end token and latency regression gates in the production-shadow canary before declaring a provider/model supported for Hybrid routing.

Historical success rates, online adaptive learning, and Team-to-Single optimization require separately versioned evaluation, monitoring, and rollback and are not part of V1.

Memory evaluation metrics

MemoryExtractionSample and MemoryRetrievalSample are experimental input shapes for future memory scorers; this release adds no memory runtime and no automatic memory writer. The following metrics can be implemented with existing rule or judge scorers:

Stage	Metric	Definition
Extraction	Yield	Valid extracted records relative to conversation, token, latency, or monetary cost. Report raw counts beside any ratio.
Extraction	Duplicate and conflict rate	Share of records that repeat, contradict, or add no durable information. Rule checks can catch exact duplicates; semantic conflict needs a versioned judge.
Extraction	Staleness annotation rate	Share of time-sensitive records carrying enough provenance or expiry information to identify staleness risk.
Extraction	Scope leakage	Private content written into team scope. This is a safety gate: any non-zero leakage fails.
Extraction	Cost and reasons	Extraction latency and tokens, plus the distribution of skipped and merged reasons.
Retrieval	Relevance	Judge score between the query and retrieved records. Version the rubric and judge configuration.
Retrieval	Omission	Available positive records that should have been returned but were not.
Retrieval	Pollution	Run the same case with and without retrieved memory; a lower primary score after injection is harmful pollution.
Retrieval	Added cost	Extra tokens and latency caused by retrieval and prompt injection.

Automatic extraction or consolidation should not be enabled by default until it passes both a versioned offline EvalSet gate and online sampling. Scope leakage is always a hard safety gate, independent of average quality.

Privacy

EvalSet cases may contain private user data. storePayloads therefore defaults to 'none', so records contain scores, reasons, metadata, and run references but no input/output snapshots. 'redacted' serializes each payload field, caps it at 8 KiB, and applies OMA's existing secret redaction. 'full' keeps the serialized text without redaction but still applies the 8 KiB cap; opt into it only for data you are prepared to retain. A model-based judge necessarily sends the evaluated output to the configured judge model regardless of record payload storage.

Reproducibility and the absence of a seed

OMA's current provider contract has no cross-provider seed parameter or LLM response recording. Adding a seed to EvalSet would therefore promise determinism the framework cannot provide. Use repeats to sample nondeterministic behavior and compare the aggregate statistics. targetFromPlan() fixes the orchestration plan, but model responses can still vary.

Use OMA agents as judges

import { z } from 'zod'
import { createJudgeScorer } from '@open-multi-agent/core/eval'

const relevancy = createJudgeScorer({
  name: 'relevancy',
  version: 'prompt-v1',
  judges: [
    { name: 'judge-a', model: 'claude-sonnet-4-6', provider: 'anthropic' },
    { name: 'judge-b', model: 'gpt-5', provider: 'openai' },
  ],
  quorum: 2,
  timeoutMs: 30_000,
  verdictSchema: z.object({
    score: z.number().min(0).max(1),
    pass: z.boolean(),
    reason: z.string(),
  }),
})

const result = await relevancy.score(context)
console.log(result.score, result.pass)

Judge scores are averaged. When the verdict schema returns a boolean pass, the scorer returns pass: true after the configured quorum is reached. The default verdict schema contains only score and reason, so the default result leaves pass undefined.

result.details.judges, result.details.models, and result.details.scores are parallel arrays: values at the same index describe one judge. This flat representation remains compatible with trace attribute values while preserving model-drift evidence. Bump the scorer version whenever judge models, configuration, or prompts change.

FAQ

Should a scorer error count as zero?

No. It means quality was not measured. OMA records scorer_error, excludes it from score denominators, and lets gate health limits decide whether the evaluation infrastructure is reliable enough.

Why did a baseline scorer comparison produce a warning?

Candidate and baseline scorer versions differ, or one side omitted the scorer. OMA still applies absolute thresholds and health checks, but skips an invalid apples-to-oranges regression comparison. Review the scorer change and create a new accepted baseline intentionally.

Can evaluation delay or fail the business response?

Offline evaluation is a separate call. Online evaluation performs only a synchronous sampling and bounded-queue decision after the run settles; scoring and persistence are best-effort and isolated. Call forceFlush() when the host must wait for accepted samples before exit.

Does targetFromPlan() make an LLM run deterministic?

It fixes the task graph and avoids another coordinator decomposition. Model responses can still vary because OMA has no cross-provider seed contract. Use repeats and compare distributions.

Where can I see complete examples?

Run examples/patterns/eval-offline-regression.ts for a no-key two-target gate or examples/patterns/eval-online-sampling.ts for FileEvalStore lifecycle.