Guard interface

What a guard actually is: (commitments, action) -> block?. This document specs the whole interface, the part that's shipped, the part landing this round, and the part that's agreed on but not built yet.

Status, read this first

Piece	State
Case shape	shipped
Guard shape (name, kind, note, block)	shipped
Label strip (block() never sees Case.label)	landing this round, alongside these docs
GuardMeta / required meta field	landing this round, alongside these docs
kind: 'external'	landing this round (reserved, no loader consumes it yet)
meta rendered in run.mjs's printed table	landing this round
--guard <path> out-of-tree loader	specified below, not built
Load-time fail-closed probe	specified below, not built
Enforced outer timeout	specified below, not built
RESULTS-EXTERNAL.md	specified in REPORTING-STANDARD.md, not built

If you've checked out a commit where src/guards.ts's Guard type doesn't have meta yet, that's the label-strip/meta change not merged on your branch, not a doc error.

Case

From src/guards.ts:

export interface Case {
  id: string;
  difficulty: string;
  label: 'drift' | 'clean';
  commitments: string[];
  action: string;
}

label is ground truth, used only for scoring in run.mjs, after a guard has already answered. id and difficulty are corpus metadata, and on a public, stable corpus they are an answer oracle: a guard reading only those two fields scores ~0.85 balanced kappa while doing no judging (the regression test in test/run.test.mjs keeps that impossible). block() therefore receives exactly commitments and action — never label, id, or difficulty.

Guard

export type Guard = {
  name: string;
  kind: 'real' | 'archetype' | 'baseline' | 'external';
  note: string;
  meta: GuardMeta;
  block(c: GuardCase): Promise<boolean>;   // GuardCase = Pick<Case, 'commitments' | 'action'>
};

kind describes what the guard is standing in for, not how well it does:

• real: an actual published guard plugin, encoded to score it (barricadeGuard, safeguardGuard, guardianGuard in src/guards.ts).
• archetype: models a strategy, not any specific plugin (denylistGuard).
• baseline: allowAll / blockAll, there to show catch rate alone means nothing.
• external: reserved for a guard supplied through the loader, once it exists.

GuardMeta

The idea for a required, construction-time declaration of how a guard actually runs (not what it claims in a README) came from PR #1's author. We adopted it and generalized it: every guard now has to carry one, in-tree or out.

export interface GuardMeta {
  transport: 'http' | 'in-process' | 'local-model';
  model: string | null;
  temperature: number | null;
  live: boolean;   // true iff the call leaves the process
}

In-tree declarations, once this lands:

guard	transport	model	temperature	live
denylistGuard, barricadeGuard, safeguardGuard, guardianGuard, allowAll, blockAll	in-process	null	null	false
judgeGuard(endpoint, model)	local-model	the model argument	0	true

judgeGuard's temperature: 0 in meta matches the temperature: 0 already pinned in its own fetch body in src/guards.ts, it isn't a new claim, it's existing behavior finally declared somewhere the results table can render it.

The honest limit on meta

meta enforces presence, not truthfulness. Nothing stops a guard from declaring { transport: 'in-process', live: false } and calling the network anyway. This is the same class of problem PR #1 surfaced by disclosing its own numbers came from a different transport than ours: a self-reported field is only as good as the person filling it in. What closes that gap is module hashing, pinning the actual code instead of a claim about it, and that's next round's work, not this one's. Don't read meta as verified. Read it as "the author was asked to say this out loud, somewhere a diff review can catch it later if it's wrong."

Fail-closed is load-bearing, not incidental

From run.mjs:

try { blocked = await g.block(c); } catch (e) { blocked = true; } // a guard that errors fails closed

Every uncaught error in block() becomes a BLOCK. That's the harness's whole safety property, and it only holds if a guard lets its own transport and parse errors propagate. A guard that catches internally and returns false inverts this silently, its apparently low false-block rate might just be dropped calls, not judgment. This is exactly the pattern in PR #1's diff (if (!res.ok) return false; // fail open and catch { return false; // fail open }), and it's why it isn't being merged as-is. If you're wiring a guard against a real endpoint: let it throw. Don't catch to be polite.

An errored call still counts as a block for scoring, which means a guard whose transport is entirely dead (every call throws) scores identical to a guard that genuinely catches everything: catch, falseBlock, and balancedKappa alone cannot tell the two apart. A contributor's live run hit this directly, a 402-everywhere run scored row-for-row identical to block-all, and only checking the transport layer by hand surfaced it. Because of that, run.mjs now counts and prints exceptions per guard (the errors column, count/total) and marks a run where every case errored INVALID: errored on every case, so a dead transport is visible from the table itself, not only from re-reading the guard's own logs.

If your guard talks HTTP, use httpJudgeGuard

Do not write your own fetch. httpJudgeGuard in src/guards.ts is the shape to build on, and it makes the fail-closed property above impossible to get wrong rather than merely required.

httpJudgeGuard({
  name, kind, note, meta,
  endpoint,
  headers,
  mapCaseToBody: (c) => ({ /* your API's request shape */ }),
  parseBlock:    (body) => /* true means BLOCK */,
  timeoutMs,   // optional, defaults to 15000
})

mapCaseToBody and parseBlock are both required and there is no default request or response shape, so no judgment API is privileged by the structure. Your two functions are the only place your API's field names appear.

Why this is stronger than the load-time probe below. The factory's own fetch handling throws on a non-OK response or a transport error, and parseBlock runs only after a confirmed-OK response. A guard author never receives a code path on which an error could be caught and turned into an allow. The probe detects a guard that fails open; this makes one impossible to write through the factory.

A worked example of the two functions, for an imaginary API that returns { "risk": "high" }:

httpJudgeGuard({
  name: 'example-judge',
  kind: 'external',
  note: 'illustration only, not a real service',
  meta: { transport: 'http', model: null, temperature: null, live: true },
  endpoint: 'https://judge.example/v1/assess',
  headers: { Authorization: `Bearer ${apiKey}` },
  mapCaseToBody: (c) => ({ text: c.action, rules: c.commitments }),
  parseBlock: (body) => (body as { risk?: string }).risk === 'high',
});

That is the whole shape. Any in-tree configuration of the factory is one example among others and carries no endorsement of the service behind it.

If your guard reaches a live endpoint, gate it in run.mjs behind a cheap reachability probe the way judgeGuard is gated. Without one, an unreachable host means every case waits out the full per-call timeout in turn.

A note on what a guard's endpoint may return. The harness reads only what your parseBlock extracts, and a non-OK response throws carrying the status alone, so a response body never reaches the run output. Keep it that way. Some services return descriptive text in error and discovery responses that is written to be read by an agent rather than by a person, and a benchmark should never be the surface that carries it.

Specified, not built: the loader, the probe, the timeout

None of this exists yet. It's written down here so the shape is agreed before it's built, and so nobody mistakes the plan for the product.

--guard <path> loader. Points run.mjs at an out-of-tree module exporting something shaped like Guard, scored alongside the in-tree guards with kind: 'external'.

Load-time fail-closed probe. Before scoring any case, the loader calls block() once with fetch stubbed to reject, and refuses to load any guard that resolves to false instead of throwing. One call at load time, not one per case, it's checking the guard obeys the fail-closed contract above, not re-checking it every run.

Enforced outer timeout. A bound on how long block() is allowed to run before the harness treats it as a fail-closed error. This is a harness gap today, not a constraint being imposed on outsiders: no in-tree guard has any timeout at all. judgeGuard's own call into the judge library sets no timeout, and the only explicit AbortSignal anywhere in this repo's code guards run.mjs's pre-flight liveness probe (endpointUp(), 800ms), not any guard's verdict call. PR #1 already passes AbortSignal.timeout(15000), which means it's more careful about this than anything currently in-tree. When the outer timeout lands, it closes a hole that applies to judgeGuard too, not just to guards coming in through a loader.

Minimal worked example: an out-of-tree guard

This is what a guard module will look like once the loader exists. It doesn't import anything from holdline, it just has to match the shape.

// my-guard.mjs
export default function myGuard() {
  return {
    name: 'my-guard',
    kind: 'external',
    note: 'blocks anything that touches /etc',
    meta: { transport: 'in-process', model: null, temperature: null, live: false },
    async block(c) {
      // c has { id, difficulty, commitments, action } but never `label`
      return /\/etc\b/.test(c.action);
    },
  };
}

Under twenty lines, and every field in it is either load-bearing (block) or accountable (kind, meta). Once --guard ./my-guard.mjs exists, this is the whole contract.