README.md

┌────────────────┐   evaluate tool    ┌──────────┐   POST /v1/systemone   ┌──────────────┐
│ Claude Code    │ ─────────────────▶ │ evaluate │ ─────────────────────▶ │ TypeSafe API │
│ Claude Desktop │   MCP stdio        │  binary  │     or /decisions      │  or          │
│ Codex          │                    │          │     retries 429 / 529  │  OpenRouter  │
│ pi · OMP       │ ◀───────────────── │          │ ◀───────────────────── │              │
│ PyThinker · …  │   typed JSON       │          │                        │              │
└────────────────┘                    └──────────┘                        └──────────────┘

The tool contract is canonical in this binary: description, field docs, and agent guidance live in cmd/evaluate/tools.go. Claude Code, Claude Desktop, and Codex register that MCP surface. Stock pi has no MCP client, so evaluate setup pi writes an extension that registers the same tool and talks MCP to this binary. Pi that already loads MCP servers (pi-mcp-adapter and similar) registers evaluate mcp in that adapter's config, same as any other stdio server. Adapters own transport and registration only; they do not author Jev semantics.

Install

macOS and Linux, amd64 and arm64:

curl -fsSL https://raw.githubusercontent.com/PyModel/typesafe-mcp/main/install.sh | sh

Installs to ~/.local/bin. Add it to PATH if needed: export PATH="$HOME/.local/bin:$PATH". With Go 1.27+:

go install github.com/PyModel/typesafe-mcp/cmd/evaluate@latest

evaluate update upgrades in place from a checksum-verified GitHub release.

Setup

Get a key at https://console.typesafe.ai/

TYPESAFE_API_KEY=your-key evaluate setup

That finds the agents installed on this machine, lists them, and asks which to register with — Enter takes all of them. --yes skips the question; --host omp --host opencode names hosts outright. Installing the binary does not register it: run this once after install.sh.

Host	How it is registered
Claude Code, Codex	their own mcp add CLI (user scope)
Claude Desktop	claude_desktop_config.json
pi (stock)	the evaluate.ts extension, below
OpenCode	mcp in ~/.config/opencode/opencode.json[c]
OMP (Oh My Pi)	~/.omp/agent/mcp.json
PyThinker	~/.pythinker-code/mcp.json (or $PYTHINKER_CODE_HOME)
Cursor	~/.cursor/mcp.json
Gemini CLI, Qwen Code	mcpServers in ~/.gemini/settings.json, ~/.qwen/settings.json
VS Code (Copilot)	servers in the user mcp.json
Zed	context_servers in ~/.config/zed/settings.json
Windsurf, Roo Code, Crush, Amp, Factory Droid	each one's own MCP config file

Config files are edited in place: the evaluate entry is inserted or replaced, and everything else — comments in the JSONC ones included — stays as written. A file that does not parse is refused, not rewritten. Aider has no MCP client; Warp keeps MCP servers inside the app (Settings → AI → MCP).

evaluate setup mcp is the older, narrower form: Claude Code (user scope) and Codex when their CLIs are on PATH, and Claude Desktop when the app is installed. Every TYPESAFE_* variable in the shell is copied into the client config, plus OPENROUTER_API_KEY. Re-run to refresh.

Already on OpenRouter? Use that key and evaluate calls OpenRouter's Decisions endpoint, billed there:

OPENROUTER_API_KEY=your-key evaluate setup mcp

TYPESAFE_API_KEY wins if both are set, so a leftover OpenRouter key cannot reroute an existing TypeSafe setup. OpenRouter's Decisions path is still /api/alpha/ and may move.

For stock pi:

evaluate setup pi

Writes ~/.pi/agent/extensions/evaluate.ts (or $PI_CODING_AGENT_DIR/extensions/). Run /reload in pi. The extension reads the API key from the shell pi runs in; nothing is baked into the file.

If pi already loads MCP servers (pi-mcp-adapter and similar), add this binary to the adapter config (~/.pi/agent/mcp.json, or .mcp.json in the project):

{
  "mcpServers": {
    "evaluate": {
      "command": "/absolute/path/to/evaluate",
      "args": ["mcp"]
    }
  }
}

Reload pi. The adapter inherits TYPESAFE_API_KEY or OPENROUTER_API_KEY from the shell pi runs in, unless you set env on the server. Use one registration: the extension and an mcp.json entry both expose a tool named evaluate.

Any other MCP-capable agent or CLI: register the same stdio server by hand — command /absolute/path/to/evaluate, args ["mcp"], with TYPESAFE_API_KEY or OPENROUTER_API_KEY in its env — using the mcpServers entry above in whatever file that host reads. The tool, its schema, and its guidance are identical on every host. Details: cmd/evaluate/AGENTS.md.

Host	Attach with
Every host in the table above	evaluate setup
Claude Code, Codex, Claude Desktop only	evaluate setup mcp
pi (stock) only	evaluate setup pi
pi with an MCP adapter, other MCP clients	manual stdio registration of evaluate mcp

With the TypeSafe skill

TypeSafe ships an official agent skill that teaches a coding agent the question types, patterns, and cookbooks. The two are complementary, not alternatives:

	Gives the agent	Use it when
TypeSafe skill	knowledge: how to design questions, thresholds, and System One patterns	the agent is writing code that calls TypeSafe
evaluate (this repo)	a tool: a live route to Jev from inside the agent loop	the agent itself needs a judgment now — classify, route, score, validate, check a claim against evidence

# Claude Code
claude plugin marketplace add typesafe-ai/skills
claude plugin install typesafe@typesafe-ai

# other agents
npx skills add typesafe-ai/skills --skill typesafe-ai

evaluate does not depend on the skill: its tool description and server instructions carry the guidance needed to call it correctly.

The tool

Interactive map with themes, path tracing, and a relationship lens: docs/architecture.html.

One tool, evaluate. Read-only. Request:

Field	Required	Meaning
state	yes	Evidence to judge: plain text, or a JSON object/array with named fields. What you observed, not your verdict about it.
questions	yes	Map of question id → {type, instructions, criteria?}. Ids are for your code; Jev never sees them, so instructions must carry the full meaning.
model	no	Defaults to jev-latest on TypeSafe, or ~typesafe/jev-latest on OpenRouter.

Type	Criteria	Answer
noul	optional {true, false} descriptions	Probability the condition holds, in [0, 1]. No separate confidence. Near 0.5 is uncertain, not medium intensity.
choice	required map of option → description (or null). Include a no-match option when nothing may fit. Max 255 options.	Chosen option, per-option probabilities, confidence (how peaked the distribution is).
score	required ordered array of level descriptions, low to high (2–10 levels)	Probability-weighted position, 0-indexed: N levels score 0 to N-1, so 3.87 over 5 levels sits between levels 3 and 4, not 3.87/5. Response includes legend and per-level probabilities.

Malformed noul / choice / score criteria are rejected locally, with questions["id"].criteria in the error, before anything is sent. The documented answer-space limits are enforced locally too: more than 255 choice options or 10 score levels fail before anything reaches the API. Unknown question types pass through; the API documents more types than this tool.

Successful responses are validated against your request before they reach you: every question id must be answered exactly once, and each answer is checked against the criteria you sent — distribution keys, score range, probability sums, confidence. A malformed or partial upstream response becomes a tool error naming the offending answer, never silent garbage.

Independent questions over the same state belong in one call. They run in parallel and cannot see each other's answers.

Do not put your conclusion in state. Jev treats that as evidence, and the confidence that comes back is agreement with you, not an independent check.

Example:

Use evaluate to decide whether this ticket is urgent and which team should own it: Help! My payouts have been failing for 3 days.

{
  "state": "Help! My payouts have been failing for 3 days.",
  "questions": {
    "is_urgent": {"type": "noul", "instructions": "Does this convey urgency?"},
    "department": {"type": "choice", "instructions": "Which team should handle this?",
      "criteria": {"billing": "Payments, refunds", "technical": "Bugs, outages", "sales": "Pricing"}}
  }
}

The tool returns the raw TypeSafe JSON, answers keyed by the same ids.

Runtime

• 429 and 529 are retried with exponential backoff (up to 3 retries). Other API errors come back as tool errors.
• HTTP timeout 60s. Response body capped at 16 MiB.
• Safe under parallel calls: each call owns its retry policy, and concurrent requests cannot amplify retries or serialize into each other's backoff windows.
• Single static binary: no Node, no Python runtime for the server.

How to write questions, batch, and read confidence: docs/jev/. Live TypeSafe docs: https://docs.typesafe.ai · https://docs.typesafe.ai/api

Quality gates

The tool's contract is pinned by tests at every layer:

• Request matrix — malformed inputs, criteria shapes, answer-space limits, and model defaults, exercised through the real MCP boundary, with golden error texts and proof that errors never echo your state back.
• Response matrix — every successful Jev response is validated against the questions that produced it (answer ids, criteria match, distributions) before it can cross the tool boundary.
• Real stdio proof — a subprocess suite speaks newline-delimited JSON-RPC to the shipped binary: clean flow, hostile-frame behavior, and the stdout-is-protocol-only invariant.
• Concurrency — parallel calls keep aggregate retries bounded and backoff windows overlapped.
• Live gate — task smoke proves the implementation still agrees with the production API (opt-in, needs a key); any change that tightens what the tool accepts or rejects must pass it before landing.

CI runs offline and deterministic; live and model-driven checks are explicit opt-in tasks. Eval plan: docs/evals/PLAN.md · Product roadmap: docs/ROADMAP.md.

Evals

The tool-selection gate (docs/evals/PLAN.md §4–§6) measures whether a coding agent reaches for evaluate exactly when semantic judgment is the right primitive — and not otherwise. A real agent loop (temperature 0, five turns max) works against the shipped binary plus harness-side decoy tools (read_file, run_command, answer_directly); every tool call is graded against a rubric fixed at authoring time over a 20/20/10 corpus, three repetitions per scenario. A naive always-evaluate policy must fail the corpus (vacuity), or the numbers mean nothing.

Three numbers are pre-registered as pass thresholds: tool_precision (appropriate evaluate calls ÷ all evaluate calls), request_validity (schema-accepted calls ÷ attempted calls), and ambiguous_acceptance (ambiguous scenarios whose majority passed). The remaining metrics are reported, not gated. The first gated run — deepseek-v4-flash served locally — is red: precision 0.778 < 0.90 and request_validity 0.815 < 1.00. A second independent run at a later head reproduced every aggregate and the same seven failing scenarios exactly (trend chart above). Per the pre-registered failure attribution, every failure is single-model: six positives answered in prose (missed_call_rate 0.300) and one case where the model looped evaluate five times on schema-rejected arguments. That is a model limitation, not product ergonomics, so no tool-description rewrites are triggered.

Run records live under evals/results/ — append-only, one JSON per run, committed so the history is auditable. Each record keeps every rep's observed tool calls, so a grading fix is re-scored offline instead of costing another model run (ADR-003). Regenerate the charts with task charts after a run; task eval runs the gate itself (opt-in: a local OpenAI-compatible model plus a TypeSafe or OpenRouter key).

Agent working practices

House rules: AGENTS.md. Pack (verification, orchestration, skills, prompts): agents/. Roles: roster/. Workflows: playbooks/. Failure principles: ANTIPATTERNS.md.

Contributing

Maintained by elkaix under PyModel. Issues and PRs: PyModel/typesafe-mcp.

Task: task check (scripts/check.sh) runs gofmt, go vet, and tests with -race in an environment with no keys, as CI does; task hooks installs it as git hooks (lint on commit, the full gate on push). task inspect opens the MCP Inspector against a local build. task smoke checks the live API. Contributor notes: AGENTS.md. Domain terms: docs/CONTEXT.md.

MIT. See LICENSE.