Eval sample format

September 4, 2026 · View on GitHub

An eval-samples file is the versioned test-set document omk eval / omk doctor run against. Its samples array contains cases, each with a prompt plus optional rubric, assertions, and metadata. JSON and YAML are first-class formats: use eval-samples.json for generated output or eval-samples.yaml for hand authoring.

For designing a rigorous sample set (what to test, how many, the metadata fields), see sample design — this page is the field-by-field format reference.

Storage locations

Recommended layouts:

Project-shared samples: put eval-samples.json or eval-samples.yaml at the project root. Use this for A/B comparisons where variants must run on the same test set.
Skill-local samples: put eval-samples.json or eval-samples.yaml under <skill>/.omk/. A private sample set therefore requires a directory skill (<skill>/SKILL.md).

omk eval auto-discovers skill-local samples only when exactly one treatment identifies a skill. Multi-variant comparisons should use project-shared samples or an explicit --samples path.

Auto-discovery recognizes only those two canonical names. If both JSON and YAML exist in the same scope, omk fails with an ambiguity error rather than silently choosing one. .yml, samples.*, flat-skill sidecars such as <name>.eval-samples.*, and split directories are not auto-discovered. You can still load a custom JSON / YAML file or a split directory explicitly with --samples.

Every file must declare schemaVersion: omk.eval-sample-set/v2. Legacy top-level arrays are rejected. The root document, every sample, assertion, mock, and nested contract are strict: unknown fields fail before execution instead of being ignored. The published JSON Schema is schemas/eval-samples/v2/eval-sample-set.schema.json.

{
  "schemaVersion": "omk.eval-sample-set/v2",
  "samples": [
    {
      "sample_id": "s001",
      "prompt": "Review this code for security issues",
      "context": "function auth(u, p) { db.query('SELECT * FROM users WHERE name=' + u); }",
      "rubric": {
        "security": {
          "criterion": "Identifies the injection vulnerability and explains its impact",
          "weight": 0.6
        },
        "actionability": {
          "criterion": "Provides a directly usable parameterized-query fix",
          "weight": 0.4
        }
      },
      "assertions": [
        { "type": "contains", "value": "SQL", "weight": 1 },
        { "type": "contains", "value": "parameterized", "weight": 1 },
        { "type": "not_contains", "value": "safe", "weight": 0.5 }
      ]
    }
  ]
}

The root may also contain requires with tools, files, env, and preflight string arrays. No other root fields are accepted.

Fields

Field	Type	Required	Description
`sample_id`	`string`	yes	Unique sample ID
`prompt`	`string`	yes	User prompt sent to the model
`context`	`string`	no	Extra context (e.g. code). Wrapped in a code block and appended to the prompt. URLs are auto-fetched at runtime.
`cwd`	`string`	no	Per-sample working-directory override (runtime context for this one case)
`rubric`	`object`	no	Named, independently judged dimensions; every value contains `criterion` and `weight`
`rubric.<name>.criterion`	`string`	yes	One non-empty scoring criterion for this dimension
`rubric.<name>.weight`	`number`	yes	Positive weight in `(0, 1]`; all rubric weights for the sample must sum to 1
`assertions`	`array`	no	Assertion checks; see assertion types
`assertions[].type`	`string`	yes	Assertion type
`assertions[].value`	`string\|number`	depends	Check value (required for `contains`, `min_length`, `cost_max`, etc.)
`assertions[].values`	`array`	depends	String array (required for `contains_all`, `contains_any`)
`assertions[].pattern`	`string`	depends	Regex pattern (required for `regex`)
`assertions[].flags`	`string`	no	Regex flags (default `"i"`)
`assertions[].schema`	`object`	depends	JSON Schema object (required for `json_schema`, via ajv)
`assertions[].reference`	`string`	depends	Reference text (required for `semantic_similarity`)
`assertions[].threshold`	`number`	no	Pass threshold; default depends on type — `3` for LLM-scored types, `0.5` for `rouge_n_min` / `bleu_min`, `1` for `mock_hit`
`assertions[].fn`	`string`	depends	Path to a custom assertion JS file (required for `custom`)
`assertions[].weight`	`number`	no	Weight (default 1)
`assertions[].not`	`boolean`	no	Invert a valid pass/fail reading; works with any type
`assertions[].n`	`number`	no	n-gram order for `rouge_n_min` (default 1)

The loader validates this contract before any model call. A rubric must contain at least one dimension; dimension names and criteria must be non-blank, weights must be finite and positive, and the per-sample sum must equal 1 within 1e-9. The published JSON Schema expresses the local shape and bounds; the runtime validator additionally enforces the cross-property weight sum. Invalid input is a configuration error and is never counted as model failure.

Metadata & sandbox fields

A sample can also carry metadata (documentation / diagnostics only — these never enter grading / judge / verdict) and sandbox fields (for evals decoupled from the real environment). Full guidance lives in sample design; here is the field index:

Field	Type	Purpose
`capability`	`string[]`	capability dimensions this sample covers (drives coverage diagnostics)
`difficulty`	`'easy' \| 'medium' \| 'hard'`	difficulty bucket (strict enum)
`construct`	`string`	what it measures: `necessity` / `quality` / `capability` (custom allowed)
`provenance`	`'human' \| 'llm-generated' \| 'production-trace'`	data source
`covers`	`{ targetKind, ref }[]`	optional declared skill-structure anchors for high-value samples; used by Skill Map only
`mocks`	`object[]`	tool-call interception list — requires an executor with mock-interception support
`mocksStrict`	`boolean`	deny any tool call that matches no mock (default `true`; set `false` only to explicitly allow pass-through)
`tripwire`	`boolean`	trap sample: the LLM is expected to fail (default `false`)
`environment`	`object`	prompt-only preconditions: `cli_available` / `files_available` / `notes`; does not materialize files or env vars

The loader also validates cross-field references. Every mock_hit: "Tool:N" must identify the Nth declared mock for that exact tool; a missing mock or out-of-range ordinal is a configuration error. Executor compatibility is checked separately before evaluation. See executors for the support matrix.

mocks[].tool uses the same source-neutral identity namespace as trace assertions (Bash, Read, Edit, and so on). Executor adapters normalize runtime-native names such as exec_command, command_execution, and apply_patch before matching. Exact native names remain accepted for backward compatibility and custom tools.

covers is optional and intentionally explicit, not inferred from prompt text. Use it first on critical or high-signal samples, so Studio can draw declared structure edges without forcing every sample to become a maintenance task. Omitting covers means the structure edge is undeclared in Skill Map, not proven untested:

Studio also surfaces this declaration in the Skill Map node detail panel: selecting a node shows whether its structure relation is explicitly declared by sample.covers.

schemaVersion: omk.eval-sample-set/v2
samples:
  - sample_id: release-risk-summary
    prompt: "Summarize release risk and rollback plan."
    covers:
      - targetKind: reference
        ref: references/release-policy.md
      - targetKind: workflow
        ref: release
      - targetKind: workflow_node
        ref: release.check

Allowed targetKind values are skill, skill_file, frontmatter, reference, script, hard_rule, workflow, and workflow_node. For reference / script, ref is the path relative to the skill root. For hard_rule / workflow, use the rule or workflow id. For workflow_node, use workflowId.nodeId. This field never enters grading, the judge prompt, the verdict, or the sample fingerprint.

URL auto-fetching

URLs in prompt and context are auto-fetched before evaluation and inlined into the text. Useful when referencing online docs, API references, etc.:

{
  "schemaVersion": "omk.eval-sample-set/v2",
  "samples": [{
    "sample_id": "s001",
    "prompt": "Generate test cases from this PRD: https://wiki.example.com/prd/feature-x"
  }]
}

During the host Resolve stage, URLs are replaced with the resolved content before the Evaluation Core Dataset is compiled. Each canonical URL is resolved once, its normalized UTF-8 bytes are sealed as a digest-addressed content resource, and those same bytes enter the Dataset input and Definition digest. Transport details (HTTP vs MCP) remain non-canonical lineage, so changing transport alone does not change the measurement identity.

Resolution order is MCP first for matching URLs (for example SSO-protected private docs), then safe HTTP for the rest or as an MCP fallback. Resolution is fail-closed: if any non-placeholder URL cannot be resolved, the evaluation does not continue with the raw URL. This prevents transient network state from silently changing the construct being measured.

Private-doc URLs: drop a .mcp.json config file into the project dir, or pass --mcp-config <path>:

{
  "mcpServers": {
    "docs": {
      "command": "npx",
      "args": ["@example/docs-mcp-server"],
      "env": { "DOCS_API_TOKEN": "xxx" },
      "urlPatterns": ["docs.example.com"],
      "fetchTool": {
        "name": "fetch_doc",
        "urlTransform": {
          "regex": "docs\\.example\\.com/([^/]+/[^/]+)/([^/?#]+)",
          "params": { "namespace": "\$1", "slug": "\$2" }
        },
        "contentExtract": "data.body"
      }
    }
  }
}

Public URLs: fetched through a bounded HTTP resolver. Redirect targets are revalidated, loopback/private/link-local destinations are rejected, only textual UTF-8 responses are accepted, and response size is capped. Private-network or authenticated documents must use an explicitly configured MCP resolver; credentials in URL authorities are rejected. RFC placeholder domains such as example.com remain literal and are never fetched.

The project-root .mcp.json is auto-discovered. --mcp-config or eval.yaml.mcpConfig overrides it. Resolver-owned MCP clients live for one Resolve session only and are always closed before Core compilation continues. urlPatterns entries are hostname allowlists: use an exact host such as docs.example.com or an explicit subdomain wildcard such as *.example.com; path/query substring matching is not allowed.

Scoring strategy

1. Assertion score

Rule-based local checks; each assertion yields pass/fail.

Formula:

Pass rate = sum of passed assertion weights / total weight (0–1)
Score = 1 + pass_rate × 4 (mapped to 1–5)
Example: 3 assertions (weight 1 each), 2 pass → pass rate 2/3 → score = 1 + 0.67 × 4 = 3.67

For the composite, assertions are split into two independent layers — a factScore (factual checks) and a behaviorScore (behavioral checks) — each scored with the formula above over its own assertions.

2. Rubric score

Each rubric dimension is compiled into a separate judge call, so one criterion cannot gain priority from its position beside another criterion in the same prompt. The judge scores every applicable dimension from 1–5. OMK then computes the sealed weighted mean only when every planned dimension is observed; one missing dimension makes the rubric aggregate missing. Every applicable dimension also participates in release-time judge dissent and uncertainty gates.

3. Composite score

The composite is the mean of the layered scores that are present — there are three layers:

Layer	Source
`factScore`	factual assertions (`contains` / `regex` / `json_` / `equals` / `semantic_similarity` / `tool__contains` …)
`behaviorScore`	behavioral assertions (length / word-count / `cost_max` / `latency_max` / `turns_` / `tools_` / `custom` …)
`judgeScore`	Weighted aggregate of independently judged rubric dimensions

composite = mean(present layers). A layer with no assertions (or no judge configured) is dropped from the mean, not counted as zero. A sample with no observed layer has no numeric composite score.

See the scoring pipeline for the full derivation, the equal-weight caveat, and how the multi-layer verdict gate relates to the composite.

Assertion types

30+ types in two families. Deterministic ones are checked locally (no model call); LLM-scored ones invoke the judge and return a 1-5 score gated by threshold.

Deterministic (local, no LLM call):

Type	Description
`contains` / `not_contains`	substring must / must-not appear
`regex`	regex match
`min_length` / `max_length`	length bounds
`json_valid` / `json_schema`	JSON validation
`starts_with` / `ends_with`	prefix / suffix
`equals` / `not_equals`	exact match
`word_count_min` / `word_count_max`	word-count bounds
`contains_all` / `contains_any`	multi-value match
`cost_max` / `latency_max`	cost / latency caps
`tools_called` / `tools_not_called` / `tools_count_min` / `tools_count_max`	agent tool-call assertions
`tool_output_contains` / `tool_input_contains` / `tool_input_not_contains`	a tool's input/output must (or, for `_not_`, must not) contain the given content
`mock_hit`	a declared sandbox mock was actually hit by a tool call (see sample design)
`turns_min` / `turns_max`	conversation-turn bounds
`rouge_n_min`	ROUGE-N recall ≥ threshold (`reference` holds the gold text; `n` defaults to 1; `threshold` defaults to 0.5)
`levenshtein_max`	edit distance ≤ value (for "output should be near-identical to reference")
`bleu_min`	BLEU-4 ≥ threshold (unsmoothed; degenerates to 0 on short text)
`custom`	custom JS function (30 s timeout)

LLM-scored (invoke the judge, 1-5, threshold defaults to 3):

Type	Description
`faithfulness`	output stays grounded in `sample.context` (anti-hallucination)
`answer_relevancy`	output directly answers `sample.prompt`; catches dodging, topic drift, verbosity
`context_recall`	gold facts in `sample.context` are actually used in the output (`reference` may enumerate the gold facts)
`semantic_similarity`	holistic semantic similarity to `reference`

Universal modifier:

Any assertion takes not: true to invert (replaces paired not_contains / not_equals etc; legacy types remain as aliases):

- type: regex
  pattern: "TODO|FIXME"
  not: true              # output must NOT contain TODO/FIXME

For async judge-backed and custom assertions, inversion happens only after a valid raw pass/fail reading exists. Provider failure, timeout, cancellation, budget censoring, missing input, and invalid output remain failures or missing evidence; not: true never turns infrastructure or protocol failure into a pass.

Composition (assert-set):

assert-set combines child assertions with any (OR) or all (AND) and supports nesting:

- type: assert-set
  mode: any              # at least one child must pass (mode: 'all' = all must pass)
  children:
    - { type: contains, value: "parameterized" }
    - { type: contains, value: "prepared statement" }
    - { type: regex, pattern: "bind\\(.*\\?" }

Children can independently use not: true; nested assert-sets can express any boolean shape over deterministic assertions. Async judge-backed assertions (semantic_similarity, faithfulness, answer_relevancy, context_recall, and custom) must remain top-level because an assert-set is evaluated synchronously; nesting one is rejected before execution.

Layered scoring note. An assert-set is attributed to the fact/behavior layered score only when its leaf children are homogeneous — all fact-type or all behavior-type. A mixed-layer assert-set (e.g. one contains + one max_length) has no single honest layer, so it is left out of the layered composite (it still counts toward the flat assertion pass/fail). If you want a sample's signal to land in the layered composite, prefer leaf assertions, or keep each assert-set within one layer.

Custom assertion

// my-assertion.mjs
export default function(output, { sample, assertion }) {
  return { pass: output.includes('SQL'), message: 'checked for SQL keyword' };
}