jev-eval-mcp

An eval-first MCP server for TypeSafe's Jev, a System One model that returns typed judgments (noul, choice, score) with probabilities instead of generated text.

Most Jev integrations expose "ask the model a question". The hard part in practice is not asking — it is knowing whether a question works and where to put the threshold. This server is built around that:

tool	use it for
jev_ask	prototype a question against one state and see the full probability distribution
jev_map	run a question set over many items, get one compact row each, filter/sort/save
jev_eval	measure question variants against labeled examples: accuracy, calibration, threshold sweep, worst misses

jev_map exists because the expensive part of triaging 500 files, findings, or tickets is reading them into the agent's context. jev_eval exists because a threshold picked by vibes is the usual reason a classifier gate misbehaves in production.

Install

Requires Node 20+ and a TypeSafe API key from https://console.typesafe.ai/settings/keys.

Register the server with your MCP client. Claude Code:

claude mcp add jev -e TYPESAFE_API_KEY=sk-... -- npx -y jev-eval-mcp

Or in a mcp.json-style config:

{
  "mcpServers": {
    "jev": {
      "command": "npx",
      "args": ["-y", "jev-eval-mcp"],
      "env": { "TYPESAFE_API_KEY": "sk-..." }
    }
  }
}

For opencode, in ~/.config/opencode/opencode.json (or a project-level opencode.json):

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "jev": {
      "type": "local",
      "command": ["npx", "-y", "jev-eval-mcp"],
      "enabled": true,
      "environment": { "TYPESAFE_API_KEY": "{env:TYPESAFE_API_KEY}" }
    }
  }
}

From a clone, build first (npm install && npm run build) and point the client at node /path/to/jev-mcp/dist/index.js instead.

Environment: TYPESAFE_API_KEY (required), JEV_MODEL (default jev-latest), TYPESAFE_BASE_URL (default https://api.typesafe.ai).

Questions

A question is one of three types, matching the Jev API:

{ "type": "noul",   "instructions": "Is this ticket urgent?" }
{ "type": "score",  "instructions": "How severe is this?", "criteria": ["trivial", "minor", "major", "critical"] }
{ "type": "choice", "instructions": "Which team owns this?", "criteria": { "billing": "payments", "platform": "availability" } }

Questions are keyed by an id you choose; ids are not sent to the model, so the meaning belongs in instructions. All questions in one call are answered independently against the same state in a single request, so asking several at once costs one round trip.

Typical loop

1. jev_ask — try two or three wordings on a state you understand.
2. jev_eval — run the promising ones over labeled examples (inline, or a .jsonl/.json file of { id?, state, label? }). Read the threshold sweep and the worst misses; pick a threshold from the table, not from intuition.
3. jev_map — apply the question you measured to the real workload, sorting or filtering by the answer and using save_path to keep the bulk out of context.

> jev_eval dataset_path=examples/support-tickets.jsonl variants={plain, detailed}

16 labeled items · 2 variant(s) · jev-1.13.0 · 268 ms · 4987 input tokens

| variant  | headline metrics                                                    |
| plain    | best F1 1.000 @ threshold 0.50 · AUC 1.000 · Brier 0.012 · ECE 0.097 |
| detailed | best F1 1.000 @ threshold 0.50 · AUC 1.000 · Brier 0.021 · ECE 0.095 |

Labels are coerced to the question type: booleans/yes/no/1/0 for noul, the option key for choice, and either a level name or its index for score.

Thresholds are only as good as the dataset behind them, and they are tied to a model version — jev_eval reports the resolved version so a rerun after an upgrade is comparable.

Development

npm run typecheck && npm run lint && npm test   # metrics unit tests, no network
npm run build && npm run smoke                  # live end-to-end over stdio, needs TYPESAFE_API_KEY
node scripts/call.ts jev_ask '{"state": "...", "questions": {}}'   # drive one tool by hand

The tool descriptions in this server are themselves measured with jev_eval: examples/tool-routing.jsonl holds labeled requests and scripts/routing-eval.json compares the shipped descriptions against a terse variant, so a reworded description can be checked for routing regressions instead of argued about.

node scripts/call.ts jev_eval - < scripts/routing-eval.json

MIT licensed.