Jev as a judge for agent evals
September 19, 2026 · View on GitHub
Agent evaluators usually fall into two categories: deterministic code and LLM-as-a-judge. Code is fast and reliable but limited to behavior that can be expressed as explicit logic. LLM judges can evaluate open-ended agent behavior, but they add cost, latency, and variance.
This project tests a third option: Jev, TypeSafe AI's System One decision model. We compare Jev with GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 on the same fixed agent runs, measuring binary accuracy, score reliability, cost, and latency.
What is Jev?
Jev is not an autoregressive LLM and does not generate text. It evaluates typed questions against structured state and returns typed answers with probabilities.
Jev supports three question types:
| Type | What it returns | Example evaluator question |
|---|---|---|
Noul | Probability that a yes-or-no judgment is true | Is the final answer grounded in the retrieved evidence? |
Score | An ordered rubric score, probabilities, and confidence | How useful is the answer? |
Choice | One option, probabilities, and confidence | Did the agent search appropriately? |
Multiple atomic questions can be evaluated in parallel against the same state.
Why use a decision model as a judge?
Agent evaluation is a decision task: given an agent's state and behavior, assign a score that provides feedback. Jev is designed for this pattern. It evaluates typed questions against structured state and returns typed answers with probabilities. Autoregressive models, on the other hand, reach a judgment through token-by-token generation. In our experiment, that decision-first design coincided with lower latency, lower cost, and lower variance.
That does not make any judge correct by default. A repeatable evaluator can still be consistently wrong, so we compare each judge with human labels and keep accuracy separate from reliability.
Experiment
We built a weather agent with Deep Agents and gave it access to Tavily web search. We defined five cases in a LangSmith dataset:
| Request type | Location | User need |
|---|---|---|
| Current conditions | Seattle | Report the weather right now |
| Weekend forecast | Austin | Describe the expected weekend weather |
| Decision support | Dublin | Decide whether to bring an umbrella |
| Longer-range forecast | Tokyo | Report the extended forecast |
| Ambiguous location | Springfield | Handle a request without a unique place |
We ran the weather agent once for each case and stored its complete output, including the final answer, evidence, tool calls, and expected behavior. Freezing the agent runs meant that only the judges could introduce variation between repetitions.
Each judge evaluated the five captured runs 100 times with two signals:
| Evaluator | What it measures | Output |
|---|---|---|
quality | Grounding, search behavior, and usefulness | Continuous score from 0 to 1 |
does_pass | Overall success | Binary decision: 0 or 1 |
A human reviewer labeled each fixed response against the same rubric. We use those labels as the oracle for binary accuracy. We use the continuous quality score only to measure reliability through variance, not accuracy.
Results
Accuracy
Accuracy measures whether a judge's binary does_pass decision agrees with the human oracle. Across 500 repeated decisions per judge, Jev matched every human pass-or-fail label.
| Judge | Pass-or-fail accuracy |
|---|---|
| Jev | 100.0% |
| GPT-5.6 Terra | 99.8% |
| GPT-5.6 Luna | 96.4% |
| Claude Sonnet 4.6 | 80.0% |
This is a small corpus with five agent runs and one human reviewer. The result describes this experiment; it is not a general ranking of judge accuracy.
Reliability
Reliability asks whether a judge produces the same score when the agent behavior is unchanged. We measured it using the observed variance of each judge's continuous quality scores. Lower variance is better.
Jev had the lowest observed mean per-case variance: 0.0000149. Luna was higher, Terra was higher, and Claude was higher.
| Judge | Mean quality variance | Relative to Jev |
|---|---|---|
| Jev | 0.0000149 | 1× |
| GPT-5.6 Luna | 0.00647 | 433× |
| GPT-5.6 Terra | 0.01364 | 913× |
| Claude Sonnet 4.6 | 0.00137 | 92× |
This experiment cannot establish why Jev varied less. One hypothesis is that a model designed to return bounded decisions is a better fit for this task than a model designed for autoregressive generation. The result is observational, not evidence that the model architecture caused the lower variance.
Cost
| Judge | Average cost per call | Average latency | Total evaluator cost |
|---|---|---|---|
| Jev | $0.00035 | 0.44 s | $0.34 |
| GPT-5.6 Luna | $0.00039 | 2.50 s | $0.39 |
| GPT-5.6 Terra | $0.00289 | 2.83 s | $2.90 |
| Claude Sonnet 4.6 | $0.02811 | 2.16 s | $28.17 |
At $0.00035 per call, Jev makes repeated judgments and frequent regression checks inexpensive. These costs depend on the prompts, inputs, and provider pricing at the time of the run.
What abundant evaluation changes
The important result is not simply a lower evaluation bill. When high-quality judgment becomes cheap enough to use broadly, builders can evaluate more agent runs, test more dimensions, and measure more changes without narrowing the feedback loop around cost.
That can speed up the entire agent development lifecycle. Agent engineers can turn more traces into feedback, catch regressions sooner, and move faster as they build, test, monitor, and deploy agents. Low cost can also amplify mistakes, so human review, representative datasets, and judge alignment still matter.
Run the project
Requires Python 3.13+, a Tavily API key, a TypeSafe API key, and a workspace-scoped LangSmith API key with Gateway access.
cp .env.example .env
# Add your API keys to .env
uv sync
Run the local evaluation without uploading an experiment:
uv run python main.py
Upload the dataset and record an evaluation experiment in LangSmith:
uv run python src/evals/offline_evals.py
Reproduce the repeated-judge benchmark:
uv run python src/evals/judge_reliability.py
Use --local with the benchmark command to run without uploading an experiment.
Configuration
| Variable | Purpose |
|---|---|
TAVILY_API_KEY | Web search used by the weather agent |
TYPESAFE_API_KEY | Jev evaluator access |
LANGSMITH_API_KEY | LangSmith tracing, datasets, evaluation, and Gateway access |
LS_LLM_GATEWAY_KEY | LangSmith Gateway model invocation |
LANGSMITH_GATEWAY | Routes the weather agent through LangSmith Gateway when true |
LANGSMITH_TRACING | Enables LangSmith traces when true |
LANGSMITH_PROJECT | LangSmith project for traces |
WEATHER_AGENT_MODEL | Weather-agent model identifier |
Reproducibility
The published LangSmith experiment is benchmark-jev-luna-terra-sonnet (6d08df72-c878-458c-b7c5-a7824ee6e721), started at 2026-09-18T17:53:25Z.
The LLM judges ran through LangSmith Gateway with openai/gpt-5.6-luna, openai/gpt-5.6-terra, and anthropic/claude-sonnet-4-6. Jev ran through langchain-typesafe==0.0.1a2. The experiment used deepagents==0.7.15, langchain-openai==1.6.2, langsmith==0.12.6, and tavily-python==0.8.3.
We did not set temperature, top-p, seed, or max tokens for the LLM judges, so provider and gateway defaults applied. The experiment metadata did not expose the hosted Jev service version.
Project layout
src/weather_agent/— Deep Agents weather agent and Tavily toolsrc/evals/dataset.py— LangSmith dataset definitionsrc/evals/judges/— Jev and LLM evaluatorssrc/evals/offline_evals.py— Dataset evaluation runnersrc/evals/judge_reliability.py— Repeated-judge benchmarkanalysis/— Accuracy, variance, cost, and latency analysisassets/— Archived benchmark data and generated chartstests/— Focused analysis tests