Claude Code Evaluation Guide

August 1, 2026 · View on GitHub

This guide covers how to use EvalBench for evaluating Claude Code agent workflows using MCP Servers (HTTP and stdio). It includes configuration reference, evaluation dataset format, scoring metrics, and step-by-step instructions for running evaluations locally.

The Claude Code generator mirrors the Gemini CLI generator in this repo — same evalset format, same orchestrator, same scorers — so most of what you know about Gemini CLI evaluation carries over.

Overview
Architecture
Prerequisites
Quick Start
Configuration Reference
Authentication
- Vertex AI
- Direct Anthropic API
MCP Servers
Scorers
End-to-End Examples
Troubleshooting

Overview

EvalBench's Claude Code integration enables automated, multi-turn evaluation of agentic AI workflows powered by Anthropic's Claude Code CLI. The CLI acts as the orchestrator that connects to MCP server backends and executes scenarios defined in an evaluation dataset. A simulated user powered by an LLM drives multi-turn conversations following a conversation plan.

Key Capabilities

Multi-turn evaluation with LLM-powered simulated users
Two auth modes: Vertex AI (GCP ADC) or direct Anthropic API key
Two MCP transport modes: HTTP/SSE (with Google Cloud OAuth auto-injection) and stdio
Pinned CLI versions via npm exec (matches Gemini CLI's pattern)
8 built-in scorers covering correctness, efficiency, and behavior quality
CSV and BigQuery reporting

What carries over from the Gemini CLI integration

Aspect	Same / Different
Evalset JSON format	Same — `scenarios[]` with `id`, `starting_prompt`, `conversation_plan`, `expected_trajectory`, `max_turns`, `env`
`dataset_format`	New: `agent-format` (or keep `gemini-cli-format` — both work)
`orchestrator`	New: `agent` (or keep `geminicli` — both work)
Scorers	Same (`trajectory_matcher`, `goal_completion`, `behavioral_metrics`, etc.)
Simulated user	Same (`simulated_user_model_config`)
Reporting	Same (CSV / BigQuery)
MCP server config	Same schema for HTTP servers (`httpUrl`, `authProviderType: google_credentials`, `headers`) — auto-translated to Claude Code's native format

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                          EvalBench Pipeline                        │
│                                                                     │
│  ┌──────────────┐    ┌──────────────────┐    ┌───────────────────┐  │
│  │  Run Config  │───▶│  AgentOrchestrator│───▶│  AgentEvaluator   │  │
│  │  (YAML)      │    │                  │    │                   │  │
│  └──────────────┘    └──────────────────┘    └────────┬──────────┘  │
│                                                       │             │
│  ┌──────────────┐              ┌──────────────────────┼──────────┐  │
│  │  Eval Dataset│              │     Per Scenario      │          │  │
│  │  (JSON)      │─────────────▶│                      ▼          │  │
│  └──────────────┘              │  ┌──────────────────────────┐   │  │
│                                │  │   ClaudeCodeGenerator     │   │  │
│  ┌──────────────┐              │  │  ┌──────────┐ ┌────────┐ │   │  │
│  │ Model Config │──────────────│─▶│  │MCP / API │ │Sim.    │ │   │  │
│  │ (YAML)       │              │  │  │ (Vertex) │ │User    │ │   │  │
│  └──────────────┘              │  │  └──────────┘ └────────┘ │   │  │
│                                │  └───────────┬──────────────┘   │  │
│                                │              │                  │  │
│                                │              ▼                  │  │
│                                │  ┌──────────────────────────┐   │  │
│                                │  │   Scorers (8 metrics)    │   │  │
│                                │  └──────────────────────────┘   │  │
│                                └─────────────────────────────────┘  │
│                                                                     │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │                  Reporting (CSV / BigQuery)                   │   │
│  └──────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

Flow:

The Run Config ties together the dataset, model config, scorers, and reporting.
The AgentOrchestrator (orchestrator: agent) loads the AgentEvaluator.
The evaluator instantiates ClaudeCodeGenerator based on generator: claude_code in the model config.
For each scenario in the evalset, the evaluator runs a multi-turn loop:
- Sends the starting prompt to Claude Code via npm exec --yes <version> -- -p <prompt>
- A SimulatedUser (LLM) generates realistic follow-up responses
- Tools and stats are accumulated across turns from the stream-json output
- Conversation continues until max_turns is reached or the simulated user sends TERMINATE
Results are scored and written to CSV and/or BigQuery.

Prerequisites

Python 3.10+ and project dependencies installed
Node.js and npm (for running Claude Code via npm exec)
Claude Code CLI — either:
- Globally installed: npm install -g @anthropic-ai/claude-code (then use claude_code_version: "claude"), or
- Pinned version (recommended for reproducibility): claude_code_version: "@anthropic-ai/claude-code@2.1.85" — npm exec will install it on first use
Authentication — either:
- Vertex AI: gcloud auth application-default login and Claude models enabled in your GCP project's Model Garden
- Direct API: export ANTHROPIC_API_KEY=sk-ant-...

Environment variables for the simulated user / scorer model:

export EVAL_GCP_PROJECT_ID=your_project_id
export EVAL_GCP_PROJECT_REGION=us-central1

Quick Start

1. Choose a run config

# Real MCP server (Cloud SQL Admin API):
export EVAL_CONFIG=datasets/claude-code-tools/example_run_config.yaml

# Fake MCP (offline testing):
export EVAL_CONFIG=datasets/claude-code-tools/example_run_fake_config.yaml

2. Run the evaluation

./evalbench/run.sh

Results land in results/<job_id>/ as CSV files.

Configuration Reference

1. Run Configuration

The top-level config that ties everything together.

Key	Required	Description
`dataset_config`	Yes	Path to the evalset JSON file
`dataset_format`	Yes	`agent-format` (or `gemini-cli-format` — alias)
`orchestrator`	Yes	`agent` (or `geminicli` — alias)
`model_config`	Yes	Path to the Claude Code model config YAML
`simulated_user_model_config`	Yes	Path to the model config for the simulated user LLM
`scorers`	Yes	Dictionary of scorer configurations
`runners.agent_runners`	Optional	Concurrency (default `10`). Set to `1` for sequential runs.
`reporting`	Optional	CSV and/or BigQuery output options

Example (example_run_config.yaml):

dataset_config: datasets/claude-code-tools/claude-code.evalset.json
dataset_format: agent-format

orchestrator: agent
model_config: datasets/model_configs/claude_code_model.yaml
simulated_user_model_config: datasets/model_configs/gemini_2.5_pro_model.yaml

# Run scenarios sequentially (default is 10 in parallel)
runners:
  agent_runners: 1

scorers:
  trajectory_matcher: {}
  goal_completion:
    model_config: datasets/model_configs/gemini_2.5_pro_model.yaml
  behavioral_metrics:
    model_config: datasets/model_configs/gemini_2.5_pro_model.yaml
  parameter_analysis:
    model_config: datasets/model_configs/gemini_2.5_pro_model.yaml
  turn_count: {}
  end_to_end_latency: {}
  tool_call_latency: {}
  token_consumption: {}

reporting:
  csv:
    output_directory: 'results'

2. Model Configuration

The model config defines the Claude Code CLI version, model, auth, environment, and MCP server setup.

Common Fields

Key	Required	Description
`claude_code_version`	Yes	Either `"claude"` (uses the globally installed binary) or an npm spec like `"@anthropic-ai/claude-code@2.1.85"` (uses `npm exec --yes`)
`generator`	Yes	Must be `claude_code`
`model`	Yes	Model ID (see Authentication for valid IDs)
`use_vertex`	Optional	`true` to route through Vertex AI; `false`/omit for direct Anthropic API
`vertex_project_id`	If `use_vertex`	GCP project for Vertex AI
`vertex_region`	If `use_vertex`	Vertex region (e.g., `us-east5`)
`env`	Optional	Environment variables passed to the CLI process
`setup.mcp_servers`	Optional	MCP server configurations (see MCP Servers)
`allowed_tools`	Optional	List of tool names to allow (e.g., `["Bash", "mcp__cloud-sql"]`)

3. Evaluation Dataset (Evalset)

Uses the shared scenario schema. See the agentic dataset format for the full field reference, including the canonical tool name format used in expected_trajectory.

Minimal example:

{
  "scenarios": [
    {
      "id": "cloud-sql-list-instances-01",
      "starting_prompt": "list all Cloud SQL instances in project astana-evaluation",
      "conversation_plan": "Ask the agent to list instances. If nl2code exists, get its state and verify it is RUNNABLE.",
      "expected_trajectory": ["cloud-sql__list_instances", "cloud-sql__get_instance"],
      "env": { "GOOGLE_CLOUD_PROJECT": "astana-evaluation" },
      "kind": "tools",
      "max_turns": 3
    }
  ]
}

Authentication

Claude Code supports two auth modes, controlled by use_vertex in the model config.

Vertex AI

Recommended on GCP — uses Application Default Credentials, no API key needed.

use_vertex: true
vertex_project_id: "astana-evaluation"
vertex_region: "us-east5"
model: "claude-opus-4-6"   # Vertex model ID format

Vertex model IDs (no date suffix, or @YYYYMMDD):

claude-opus-4-6
claude-sonnet-4@20250514

Requirements:

Run gcloud auth application-default login (locally) or use a service account with Vertex AI User role (Cloud Build / GKE)
Claude models must be enabled in your GCP project via Model Garden
Sets these env vars under the hood: CLAUDE_CODE_USE_VERTEX=1, ANTHROPIC_VERTEX_PROJECT_ID, CLOUD_ML_REGION

Direct Anthropic API

use_vertex: false
model: "claude-opus-4-20250514"   # Direct API model ID format
env:
  ANTHROPIC_API_KEY: "sk-ant-..."   # OR export it in your shell

Direct API model IDs (with date suffix):

claude-opus-4-20250514
claude-sonnet-4-20250514

Tip: Don't commit API keys into model_config.yaml. Prefer export ANTHROPIC_API_KEY=... in your shell, or use Secret Manager when running on Cloud Build / GKE.

MCP Servers

EvalBench accepts the same MCP server config schema as Gemini CLI for HTTP servers. The Claude Code generator auto-translates Gemini-style fields into Claude Code's native format at runtime:

Gemini-style field	Claude Code translation
`httpUrl`	→ `url` + auto-adds `type: "http"`
`authProviderType: google_credentials`	→ injects `Authorization: Bearer <ADC token>` (from `gcloud auth application-default print-access-token`, falling back to `gcloud auth print-access-token`) and sets a `headersHelper` so Claude Code re-mints a fresh ADC token on every connection (avoids ~1h expiry). Google API MCP endpoints reject the plain user token on tool calls — see Troubleshooting.
`oauth.scopes`	(dropped — Claude Code doesn't use Gemini's OAuth delegation)
`headers`	→ passed through as-is
`command` / `args` (stdio)	→ passed through as-is

HTTP MCP server (Cloud SQL Managed)

setup:
  mcp_servers:
    "cloud-sql":
      httpUrl: "https://sqladmin.googleapis.com/mcp"
      authProviderType: google_credentials
      oauth:
        scopes:
        - https://www.googleapis.com/auth/cloud-platform
      headers:
        X-Goog-User-Project: astana-evaluation

This generates the following mcp_servers.json for Claude Code:

{
  "mcpServers": {
    "cloud-sql": {
      "type": "http",
      "url": "https://sqladmin.googleapis.com/mcp",
      "headers": {
        "X-Goog-User-Project": "astana-evaluation",
      }
    }
  }
}

Stdio MCP server

setup:
  mcp_servers:
    "my-server":
      command: "python"
      args:
        - "path/to/server.py"
        - "--some-flag"

How it works under the hood

ClaudeCodeGenerator._setup_mcp_servers writes the translated config to <fake_home>/.claude/mcp_servers.json
The CLI is invoked with --mcp-config <path> so it loads only the configured servers (no host-machine pollution)
Each scenario runs in a sandboxed HOME (.venv/fake_home_claude/ locally, /tmp_sessions/<session_id>/fake_home in gRPC mode)

Scorers

See the scorer reference for the full catalog and configuration options.

Quick reference:

Scorer	Type	Description
`trajectory_matcher`	Deterministic	Jaccard or Levenshtein match between expected and actual tool trajectory. Native Claude Code tools (`Read`, `Bash`, `Edit`, `ToolSearch`, ...) are dropped from both sides by default — set `filter_native_tools: false` to score them too.
`goal_completion`	LLM	Did the agent accomplish the conversation plan?
`behavioral_metrics`	LLM	Hallucination rate + clarification rate
`parameter_analysis`	LLM	Qualitative feedback on tool parameters
`turn_count`	Deterministic	Number of conversation turns
`end_to_end_latency`	Deterministic	Total latency (model + tool execution)
`tool_call_latency`	Deterministic	Sum of tool execution durations
`token_consumption`	Deterministic	Total input + output tokens

End-to-End Examples

Example 1: Vertex AI + Cloud SQL Managed MCP

Goal: Use Claude Opus on Vertex AI to manage Cloud SQL instances.

# datasets/model_configs/claude_code_model.yaml
claude_code_version: "@anthropic-ai/claude-code@2.1.85"
generator: claude_code
model: "claude-opus-4-6"

use_vertex: true
vertex_project_id: "astana-evaluation"
vertex_region: "us-east5"

env:
  GOOGLE_CLOUD_PROJECT: "astana-evaluation"

setup:
  mcp_servers:
    "cloud-sql":
      httpUrl: "https://sqladmin.googleapis.com/mcp"
      authProviderType: google_credentials
      oauth:
        scopes:
        - https://www.googleapis.com/auth/cloud-platform
      headers:
        X-Goog-User-Project: astana-evaluation

Run:

gcloud auth application-default login
export EVAL_GCP_PROJECT_ID=astana-evaluation
export EVAL_CONFIG=datasets/claude-code-tools/example_run_config.yaml
./evalbench/run.sh

Example 2: Direct Anthropic API

claude_code_version: "@anthropic-ai/claude-code@2.1.85"
generator: claude_code
model: "claude-opus-4-20250514"
use_vertex: false

setup:
  mcp_servers:
    "cloud-sql":
      httpUrl: "https://sqladmin.googleapis.com/mcp"
      authProviderType: google_credentials
      headers:
        X-Goog-User-Project: astana-evaluation

Run:

export ANTHROPIC_API_KEY="sk-ant-..."
export EVAL_CONFIG=datasets/claude-code-tools/example_run_config.yaml
./evalbench/run.sh

Troubleshooting

`Error: Invalid MCP configuration: ... Does not adhere to MCP server configuration schema`

The MCP server config didn't translate correctly. Common causes:

Missing both httpUrl/url and command — the server needs at least one
Stale generated mcp_servers.json — delete .venv/fake_home_claude/.claude/mcp_servers.json and re-run

`There's an issue with the selected model (...). It may not exist or you may not have access to it.`

Vertex AI: The model isn't enabled in your GCP project. Visit Model Garden in the Cloud Console and enable it for your vertex_region. Or pick a different model that's already enabled (claude-opus-4-6, claude-sonnet-4@20250514).
Direct API: Your ANTHROPIC_API_KEY doesn't have access to the model. Check your console at https://console.anthropic.com.

Claude Code can't find auth credentials in the sandboxed HOME. Either:

Use use_vertex: true (uses GCP ADC, no login needed), or
Set ANTHROPIC_API_KEY in env or model config, or
Make sure your real ~/.claude/ has valid credentials (the generator copies them to the fake home)

`Error: --session-id can only be used with --continue or --resume if --fork-session is also specified.`

Already fixed — the generator passes --fork-session automatically when resuming. If you still see this, make sure you're on a current version of claude_code.py.

`When using --print, --output-format=stream-json requires --verbose`

Already fixed — --verbose is added automatically. Check your version of claude_code.py.

Empty/zero results in the summary

The simulated user failed to initialize. Check EVAL_GCP_PROJECT_ID is set if your simulated user model uses Vertex AI:

export EVAL_GCP_PROJECT_ID=astana-evaluation

Scenarios appear to run in interleaved order

This is expected — the default agent_runners: 10 runs scenarios concurrently. To force sequential execution, add this to your run config:

runners:
  agent_runners: 1

MCP tool call fails with `Incompatible auth server: does not support dynamic client registration`

Symptom: the agent connects to the MCP server and can list its tools, but the first real tool call fails with Incompatible auth server: does not support dynamic client registration.

Root cause: this is a misleading error. Google API MCP endpoints (e.g. sqladmin.googleapis.com/mcp) leave initialize and tools/list unauthenticated but require a valid bearer token on the actual tool call. When the injected token is missing, expired, or the wrong kind, the tool call returns 401 with a WWW-Authenticate: Bearer resource_metadata=... challenge. Claude Code always attaches an OAuth authProvider to HTTP MCP servers, so on that 401 it tries to complete an OAuth 2.0 flow — which begins with dynamic client registration (RFC 7591). These Google endpoints don't support dynamic client registration, so the OAuth fallback dies with that error instead of surfacing the underlying 401.

Fixes:

Use ADC, not the user token. The generator now injects gcloud auth application-default print-access-token (ADC) rather than gcloud auth print-access-token. The plain gcloud user token is rejected by these endpoints ("Request had invalid authentication credentials"), which is what triggered the fallback. Make sure you ran gcloud auth application-default login (not just gcloud auth login).
Token expiry on long runs (handled automatically). Google access tokens expire after ~1 hour. To avoid later scenarios hitting the same 401 → DCR error, the generator also emits a headersHelper on the MCP server config — a command Claude Code runs on every connection to mint a fresh ADC token (its JSON stdout overrides the baked static header). The baked token remains as a fallback for when gcloud/ADC isn't available in Claude Code's environment. No action needed; if you see stale-token 401s, confirm gcloud is on PATH and ADC is valid in the run environment.

Limitation: OneMCP / Google API MCP endpoints do not support OAuth dynamic client registration. Static bearer-token auth (via authProviderType: google_credentials) is the only supported path — Claude Code's interactive OAuth login flow will not work against them.

MCP server fails with `401 Unauthorized` (real Cloud SQL endpoint)

Usually a token problem (see the DCR entry above). Checklist:

Run gcloud auth application-default login with --scopes=https://www.googleapis.com/auth/cloud-platform.
Confirm your account has the required IAM roles (e.g., roles/cloudsql.admin).
Set the quota project header: headers: { X-Goog-User-Project: <project> }.
Verify directly: curl -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" -H "X-Goog-User-Project: <project>" -H "Content-Type: application/json" -H "Accept: application/json, text/event-stream" -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"list_instances","arguments":{"project":"<project>"}}}' https://sqladmin.googleapis.com/mcp

`npm exec` is slow on first run

npm exec --yes <package>@<version> downloads the package on first use (~30 sec). Subsequent runs use the cache.