cmd/evaluate

Jev question semantics (primitives, confidence, jaggedness): docs/jev/README.md. This file is the Go/MCP layer.

Tests sit beside their module — table-driven over httptest and in-memory MCP transports, plus two real-subprocess suites:

Test file	Covers
client_test.go	transport handling, retry policy, env construction (newClientFromEnv)
request_test.go	the request contract matrix at the tool boundary
response_test.go	the response contract matrix (exact upstream bytes)
reliability_test.go	bounded timeouts and bounded parallel retries
tools_test.go	guidelines/schema single source, validate, the served tool surface
setup_test.go	adapter transactions through the runner seam, the Desktop write, pi rendering/discovery, the pi adapter through the driver, atomic replacement
setupdriver_test.go	setup-driver orchestration with fake adapters
jevstub_test.go	the one stand-in Jev: tests needing a valid response derive it, never author it
update_test.go	archive extraction, checksum verification, staging, version policy
stdio_test.go	the real stdio framing proof (subprocess)
piadapter_test.go	the rendered pi adapter speaking MCP to the real binary (offline stub)
live_test.go, baseline_live_test.go	opt-in live smoke and latency baseline (task smoke)

File	Holds
main.go	cobra command tree; serve is the production composition root (client from env, mcp.StdioTransport at the edge)
client.go	Client.Evaluate owns the whole request contract — model default, validate (and its error wording), send, response validation against the request. POSTs the request; retries 429/529 with backoff up to 3 times, honoring Retry-After (capped at 60s); errors past the 16 MiB body cap; TYPESAFE_API_URL overrides the full endpoint (stubs, proxies) without changing which key selects the route; upstream error bodies embed verbatim up to 2 KiB, clipped with a marker past that
tools.go	canonical evaluate semantics: the guidelines slice, tool description, evaluateInputSchema (the canonical input schema, the same derivation the SDK serves), tool registration. Tool-facing prose only: request errors are the client's
setup.go	the setup driver and both compositions: runMCPSetup composes env/cfg and hands defaultHostAdapters to runSetupDriver, runPiSetup hands it the pi adapter; renders pi.ts (writePiExtension). OMP, PyThinker, and other MCP-capable CLIs/agents use the same binary via manual stdio registration (see root AGENTS.md Hosts)
setup_hosts.go	the host adapters — Claude Code, Codex, Claude Desktop, pi — each owning its complete registration transaction (detection, CLI or file protocol, rollback, legacy cleanup); replaceFile is the one atomic write every file-registering adapter uses
update.go	self-update from a checksum-verified GitHub release
pi.ts	pi extension template, embedded and rendered by setup.go

Constraints

• The Go request type (evaluateIn) and its jsonschema tags are the only authored structural source. The guidelines slice and tool description live in tools.go and flow everywhere: the guidelines are served as the MCP server instructions (evaluateInstructions) and rendered into pi.ts as a JSON array; evaluateInputSchema() returns the canonical input schema — the same object the SDK serves on the wire — and writePiExtension inserts it verbatim as __EVALUATE_SCHEMA__. pi.ts carries the contract (plain JSON Schema; pi passes parameters to providers uninterpreted), it does not restate it: structure and prose both change in tools.go only, and TestEvaluateInputSchemaIsServed pins the wire and the adapter to the same bytes.
• Stdout of evaluate mcp carries the MCP protocol; diagnostics go to stderr.
• Server composition is transport-independent: newServer builds tools, instructions, and version with no transport knowledge, and serve is the production composition root — client from env, mcp.StdioTransport at the edge, nothing in between. Contract tests drive newServer over in-memory transports; stdio framing itself is proven only by the real-subprocess paths (stdio_test.go and the pi adapter round-trip) — an injected stand-in does not test StdioTransport.
• route prefers TYPESAFE_API_KEY over OPENROUTER_API_KEY, so a stray OpenRouter key cannot re-bill an existing setup. An explicit model passes through unmapped, whichever route is live.
• validate rejects the criteria shapes the API definitely refuses, plus the documented answer-space ceilings (primitives.md: choice up to 255 options, score 2–10 levels — the tool description promises the same). Unknown question types pass through, because the API enumerates more of them than this tool documents; the one-level score floor stays accepted because live evidence showed the API accepts it.
• The client owns both sides of the upstream Jev contract: validate keeps requests valid before they reach Jev, and validateEvaluateResponse keeps successful responses valid before they cross the MCP boundary — documented envelope present (model, usage with both token counts; presence only, additive evolution rides along), exact answer-ID correspondence, and each answer judged against the criteria of the request that produced it. The request is the trusted specification; the response never defines the bounds used to validate itself (score range comes from len(request.criteria), never from the response's legend). Documented required fields are pinned (confidence on choice/score, probabilities in [0,1] summing to 1); undocumented types pass through both ways. Both halves run inside Client.Evaluate — validate and the route's default model before any byte is sent, the response check before any byte returns — so the tool handler is registration only and no direct caller can reach Jev with a request the contract refuses. Validation never normalizes: a valid body returns byte-for-byte. Response errors name the offending answer and never embed the upstream body. HTTP-status errors do embed the upstream body — the API's detail is the useful part — but clipped at 2 KiB (errBodyCap), so a hostile upstream cannot turn its own oversized response into an oversized tool error. Response-contract changes land in client.go only, pinned by response_test.go.
• Any change that causes validateEvaluateResponse to reject an additional successful-response shape must pass task smoke against the live TypeSafe API before landing. API docs define the intended contract; the offline matrix pins it mechanically; the live smoke proves the current service still satisfies it.
• Validation errors are deterministic: every surface that reports invalid questions or answers walks sorted ids (validate, validateEvaluateResponse), so the same input yields the same error on every run. Map iteration order is randomized in Go — never choose which error to report by ranging a map.
• update separates its effects: prepareUpdate(ctx, currentVersion, stageDir, apiBase) owns network, verification, and staging (httptest-reachable through apiBase); applyUpdate(target, artifact) owns only the atomic rename. Cleanup responsibility starts only when prepareUpdate succeeds — on every failure, including the sentinels, prepare removes its own staged file.
• Version ordering is policy, not string equality: classifyVersions returns a pure relation (unknown/behind/equal/newer); unknown and behind update, equal returns errUpToDate, newer returns errNewerThanLatest with the release tag riding along for the message. Only a valid installed version participates in ordering (unknown local state may rejoin the release train); a malformed remote tag is a hard error. semver.IsValid gates Compare because Compare sorts invalid versions below valid ones. The errNewerThanLatest path deliberately returns verifiedArtifact{tag} with no staged file so runUpdate can name the release; if a sentinel path ever needs to carry more than the tag, split release info from the staged artifact instead of widening the struct's meaning.
• pi.ts must stay valid TypeScript once the __EVALUATE_*__ placeholders are replaced with JSON-marshaled literals. The adapter round-trip test (piadapter_test.go) proves the seam end to end: a strip-types import covers the rendered syntax (node --check cannot: it does not run the type-stripping transform, so import type is a SyntaxError under it on newer Node), then a harness imports the exported callEvaluate and drives its initialize → tools/call → decode chain — including the tool-error branch — against a real binary subprocess, pointed at a local Jev stub through TYPESAFE_API_URL. JSON-RPC request ids, the protocol version, and the subprocess lifecycle stay adapter-owned: pinned by that test, not generated from Go — the ownership split is recorded in docs/ADR-002-adapter-owned-protocol-mechanics.md. pi -e cmd/evaluate/pi.ts remains the manual check; there is no tsc and no Node dev dependency (the runner needs Node ≥ 22.6 for type stripping).

Tool reference

evaluate takes state (raw evidence to judge), questions (id → {type, instructions, criteria?}) and an optional model.

Type	criteria
noul	optional {"true": ..., "false": ...} descriptions
choice	required: map of option to description
score	required: ordered array of level descriptions, low to high

Score answers are 0-indexed: N levels score 0 to N-1, so 3.87 over 5 levels sits between levels 3 and 4, not 3.87/5. The response carries a legend mapping index to level, plus a probabilities entry per level. Full API docs: https://docs.typesafe.ai/api

Manual client config

Point any MCP client — including OMP, PyThinker, and other MCP-capable CLIs/agents — at /absolute/path/to/evaluate mcp with TYPESAFE_API_KEY (or OPENROUTER_API_KEY) in its env; restart Claude Desktop after a config change. For stock pi by hand, copy pi.ts to ~/.pi/agent/extensions/evaluate.ts (or $PI_CODING_AGENT_DIR/extensions/) and replace __EVALUATE_BINARY__ with the quoted absolute path to the binary and __EVALUATE_GUIDELINES__ with a JSON array of guidance strings. For pi with an MCP adapter, add {"command":"/absolute/path/to/evaluate","args":["mcp"]} under mcpServers.evaluate in that adapter's mcp.json and reload; that path and the extension both register a tool named evaluate.