jev-packs format spec v0 (canonical)

This document is the contract. jevassert.packs implements the loader from it, jev-table reuses it for column specs, and scripts/validate.py enforces it in CI. Changes to this file are spec changes — they bump spec: and need a migration note in CHANGELOG.md.

Layout

<pack>/
  pack.yaml         # required — questions and metadata
  cases.jsonl       # required — golden cases, one JSON object per line
  README.md         # required — purpose, provenance, rationale
  CHANGELOG.md      # required — pack history
  evidence.md       # required once `tested` is set — generated by jevassert
  predictions.jsonl # optional — jevassert recording backing evidence.md

index.json at the repo root is the registry listing.

pack.yaml

spec: 0                  # int, required, must be 0
id: rag-passage-relevance        # required; kebab-case; must equal directory name
version: 0.1.0           # required; semver
license: CC0-1.0         # required; SPDX identifier
tested: null             # required; null, or "jev-<semver>" matching evidence.md
description: >-          # required; one-line purpose plus explicit non-goals
  Score whether a retrieved passage is relevant to a question.
state:
  description: The tuple the questions are evaluated on.
  fields: [question, passage]    # required; non-empty list of keys
questions:               # required; non-empty mapping, key = question id
  relevant:
    type: noul           # required; noul | choice | score
    instructions: "Does `passage` contain information that helps answer `question`?"
  quality:
    type: score
    instructions: "How useful is `passage` for answering `question`?"
    levels: [junk, weak, ok, strong, unknown]
thresholds:              # optional; per-question auto-accept floors
  relevant: {true: 0.85, false: 0.85}

Question types

type	answer	required keys	notes
noul	boolean	instructions	Yes/no question.
choice	one option key	instructions, options	options maps key → one-line meaning; at least 2.
score	one level	instructions, levels	Ordinal, low → high; 2–10 levels. Optional level_descriptions.

Rules, all CI-enforced:

1. instructions is a single, atomic, criteria-first question in English. No examples, no compound "and also" clauses, no hints about expected answers.
2. Every closed set — choice options and score levels — must contain the label unknown. Jev cannot abstain; the question set must offer abstention as an answer.
3. Question ids are snake_case; option/level keys are snake_case.
4. Underlying evidence must fit in the state: if a question needs information not present in state.fields, the question is wrong, not the case.
5. score questions may add level_descriptions, a map from level label to a situational description. Loaders send the description to the API instead of the bare label; this measurably improves scoring. Keys must be declared levels and values non-empty strings.

Thresholds

thresholds maps question id → label → minimum probability required to auto-accept that label. If no label clears its floor, the decision layer should route the item to review (or answer unknown). A label absent from the map has no auto-accept floor. Values are floats in (0, 1].

Thresholds are a suggestion for consumers; jevassert gates use their own gates.yaml. They are the pack author's recommended operating point.

tested / evidence.md

tested: null means the pack has no recorded evidence yet (status: provisional in index.json). When jevassert records a run, it writes evidence.md and the author sets tested to the exact server-returned model version (e.g. jev-1.13.0). CI requires evidence.md to exist if and only if tested is non-null.

cases.jsonl

One JSON object per line:

{"id": "rpr-0001", "state": {"question": "What is the refund window?", "passage": "Refunds are accepted within 30 days of delivery."}, "expect": {"relevant": true, "quality": "strong"}}

• id — unique within the pack, non-empty. Recommended <pack-short>-NNNN.
• state — keys must be exactly state.fields from pack.yaml.
• expect — exactly one gold label per question, keys exactly the question ids. noul expects a JSON boolean; choice/score expect one of the declared option/level keys (including unknown).

Gold-label rules:

• Labels are decided from the state alone; no outside context.
• Prefer hard, realistic cases over synthetic clean ones. Include boundary and adversarial cases; note genuinely ambiguous ones with unknown.
• Minimum 50 cases per pack. Balance classes where the question allows it.
• No production or private data. Cases are public by design.

index.json

{
  "spec": 0,
  "updated": "2026-09-19",
  "packs": [
    {
      "id": "rag-passage-relevance",
      "version": "0.1.0",
      "path": "packs/rag-passage-relevance",
      "license": "CC0-1.0",
      "status": "provisional",
      "evidence": null
    }
  ]
}

• One entry per directory in packs/; ids, versions and paths must match pack.yaml.
• status: provisional (no evidence) or verified (evidence.md exists).
• evidence: null or repo-relative path to the report.