Typesafe MCP

Give your AI agent typed decisions instead of free text. evaluate is an MCP server that lets Claude Code, Claude Desktop, Codex, and pi call TypeSafe's Jev model and get back probabilities they can branch on. One command, evaluate setup mcp, registers it with the first three (if detected); evaluate setup pi covers pi.

┌──────────────┐  evaluate   ┌──────────┐  POST /v1/systemone  ┌──────────────┐
│ Claude Code  │ ──────────▶ │ evaluate │ ───────────────────▶ │ TypeSafe API │
│ Claude Desk. │   (stdio)   │  (MCP)   │  retries 429 / 529   │  ── or ──    │
│ Codex        │             │          │                      │  OpenRouter  │
│ pi           │ ◀────────── │          │ ◀─────────────────── │              │
└──────────────┘ typed JSON  └──────────┘   POST /decisions    └──────────────┘

Why this exists

Problem: When an agent needs a yes/no call, a routing decision, or a severity rating, it usually asks an LLM, then parses prose and hopes the format holds. The answer has no probability attached, so the agent cannot tell a confident "yes" from a coin flip.

Solution: evaluate exposes one tool, evaluate, that sends state plus typed questions to Jev and returns structured answers with probabilities. Nothing to parse and no prompt formatting to maintain. evaluate setup mcp wires it into Claude Desktop, Claude Code, and Codex in one step, and evaluate setup pi installs the equivalent extension for pi.

Quickstart

1. Install (macOS and Linux, amd64 and arm64):

curl -fsSL https://raw.githubusercontent.com/itsmostafa/typesafe-mcp/main/install.sh | sh

It installs to ~/.local/bin. If that is not on your PATH, add it with export PATH="$HOME/.local/bin:$PATH". With Go, you can instead run go install github.com/itsmostafa/typesafe-mcp/cmd/evaluate@latest. Run evaluate update to upgrade in place.

2. Register with your agents (get a key at https://console.typesafe.ai/)

TYPESAFE_API_KEY=your-key evaluate setup mcp

Already on OpenRouter? Use that key instead and evaluate routes through OpenRouter's Decisions endpoint, billed to your OpenRouter account:

OPENROUTER_API_KEY=your-key evaluate setup mcp

TYPESAFE_API_KEY wins if both are set. OpenRouter's Decisions endpoint is still on its /api/alpha/ path and may move.

Using pi? It has no MCP client, so evaluate ships a pi extension instead:

evaluate setup pi

That writes ~/.pi/agent/extensions/evaluate.ts, which registers evaluate as a native pi tool and talks to evaluate mcp for you. Run /reload in pi to pick it up. Unlike the MCP clients, nothing is baked into the file: the extension reads your key from the shell pi runs in.

3. Ask your agent a judgment question

"Use evaluate to decide whether this ticket is urgent and which team should own it: Help! My payouts have been failing for 3 days."

The agent calls evaluate with:

{
  "state": "Help! My payouts have been failing for 3 days.",
  "questions": {
    "is_urgent": {"type": "noul", "instructions": "Does this convey urgency?"},
    "department": {"type": "choice", "instructions": "Which team should handle this?",
      "criteria": {"billing": "Payments, refunds", "technical": "Bugs, outages", "sales": "Pricing"}}
  }
}

It gets back the raw response JSON, with each answer under the same id you gave it.

What you get

• One-command setup across clients. evaluate setup mcp registers with Claude Code (user scope) and Codex when their CLIs are on PATH, and with Claude Desktop when it is installed. Every TYPESAFE_* variable in your shell is carried over, plus OPENROUTER_API_KEY. Re-run it to update. evaluate setup pi installs the pi extension.
• Answers your code can branch on. Three question types: noul (probability a condition holds), choice (one option from a map), score (position on ordered levels).
• Rate limits handled for you. 429 and 529 responses are retried with exponential backoff. Other API errors come back to the agent as tool errors it can read and act on.
• Several questions, one call. Batch independent questions over the same state; they run in parallel.
• Agents that use it well out of the box. The server ships usage guidance (narrow questions, JSON state, no-match options, evidence not verdicts) to the client, so the agent writes better questions without extra prompting.
• A single static binary. No runtime, no Node, no Python. evaluate update upgrades it in place from a checksum-verified release. Read-only tool, 60s request timeout, responses over 16 MiB are rejected, never truncated.

About TypeSafe

TypeSafe builds System One models: small units of AI intelligence you use like programming primitives. Instead of generating text, they turn natural language and application state into typed judgments and probabilities that code can combine. Jev is one of them.

Website · Docs · API reference · Console

Reference

Field	Required	Description
state	yes	Content to judge: plain text, or a JSON object/array with named fields. Raw observed evidence, not your conclusion about it
questions	yes	Map of question id to {type, instructions, criteria?}
model	no	Defaults to jev-latest, or ~typesafe/jev-latest on OpenRouter

Criteria shape per question type, the 0-indexed score answers, and manual client config: cmd/evaluate/CLAUDE.md. Malformed criteria are rejected locally, before the request, with the field path you sent. Full API docs: https://docs.typesafe.ai/api

Contributing

Issues and pull requests are welcome. The repo uses Task: task check runs gofmt, go vet and the tests with -race, and task inspect opens the MCP Inspector against a local build. CLAUDE.md covers the conventions.

If evaluate saves you some prompt-parsing, a star helps others find it.