MCP Host Simulation

June 15, 2026 · View on GitHub

MCP host simulation tests your MCP server through a real LLM (OpenAI, Anthropic, etc.), exactly as a user would interact with Claude Desktop or ChatGPT. The LLM decides which tools to call based only on their descriptions and schemas — making this the highest-fidelity test of tool discoverability, parameter clarity, and description quality.

When to Use

Use MCP host simulation when you need to verify:

Tool discoverability: Does the LLM know which tool to call for a given task?
Parameter clarity: Does the LLM fill in parameters correctly without hints?
Description quality: Does the tool description accurately represent what the tool does?
End-to-end behavior: Does the full chain of LLM → tools → response work?

For most regression testing, use direct mode (callTool). Reserve MCP host simulation for:

New tool description development and tuning
Evaluating tool calling accuracy across scenarios
Pre-release validation of tool schemas

Supported Providers

All providers use the Vercel AI SDK. Install ai plus the provider-specific package:

Provider	Env Variable	Install
`anthropic`	`ANTHROPIC_API_KEY`	`npm install ai @ai-sdk/anthropic`
`openai`	`OPENAI_API_KEY`	`npm install ai @ai-sdk/openai`
`google`	`GOOGLE_GENERATIVE_AI_API_KEY`	`npm install ai @ai-sdk/google`
`vertex-anthropic`	`GOOGLE_VERTEX_PROJECT`	`npm install ai @ai-sdk/google-vertex`
`mistral`	`MISTRAL_API_KEY`	`npm install ai @ai-sdk/mistral`
`azure`	`AZURE_API_KEY`	`npm install ai @ai-sdk/azure`
`deepseek`	`DEEPSEEK_API_KEY`	`npm install ai @ai-sdk/deepseek`
`openrouter`	`OPENROUTER_API_KEY`	`npm install ai @openrouter/ai-sdk-provider`
`xai`	`XAI_API_KEY`	`npm install ai @ai-sdk/xai`

Basic Usage

import { test, expect } from '@gleanwork/mcp-server-tester/fixtures/mcp';
import { runEvalDataset, loadEvalDataset } from '@gleanwork/mcp-server-tester';

test('LLM triggers the right tool', async ({ mcp }, testInfo) => {
  const dataset = await loadEvalDataset('./data/evals.json');
  const result = await runEvalDataset({ dataset }, { mcp, testInfo });
  expect(result.passed).toBe(result.total);
});

Eval dataset with MCP host simulation:

{
  "name": "tool-discovery-evals",
  "cases": [
    {
      "id": "search-trigger",
      "mode": "mcp_host",
      "scenario": "Find recent documents about quarterly planning",
      "mcpHostConfig": {
        "provider": "anthropic",
        "model": "claude-3-5-sonnet-20241022"
      },
      "expect": {
        "toolsTriggered": {
          "calls": [{ "name": "search", "required": true }]
        }
      }
    }
  ]
}

Multi-Iteration Accuracy

LLM responses are non-deterministic. Run each case multiple times and measure accuracy:

{
  "id": "search-accuracy",
  "mode": "mcp_host",
  "scenario": "Find documents about MCP testing",
  "mcpHostConfig": { "provider": "anthropic" },
  "iterations": 5,
  "accuracyThreshold": 0.8,
  "expect": {
    "toolsTriggered": {
      "calls": [{ "name": "search", "required": true }]
    }
  }
}

The case passes if search was triggered in at least 4 of 5 runs (80% accuracy).

Tool Call Assertions

`toolsTriggered` — Assert which tools the LLM called

"toolsTriggered": {
  "calls": [
    { "name": "search", "required": true },
    { "name": "get_document", "required": false }
  ],
  "order": "any",
  "exclusive": false
}

required: true — this tool MUST have been called
order: "strict" — calls must appear in the listed order
exclusive: true — no other tools may be called

`toolCallCount` — Assert number of tool calls

"toolCallCount": { "min": 1, "max": 3 }

MCPHostConfig Options

interface MCPHostConfig {
  hostType?: 'sdk' | 'cli' | 'browser' | 'desktop'; // Host type (default: 'sdk')
  provider?: LLMProvider; // Required for 'sdk', ignored for 'cli'
  model?: string; // Model name (provider-specific default if omitted)
  maxToolCalls?: number; // Max tool call steps (default: 10)
  temperature?: number; // LLM temperature (default: 0)
  maxTokens?: number; // Max response tokens
  apiKeyEnvVar?: string; // Override default env var name
  cli?: CLIConfig; // Required for 'cli' host type
}

type LLMProvider =
  | 'openai'
  | 'anthropic'
  | 'google'
  | 'vertex-anthropic'
  | 'mistral'
  | 'azure'
  | 'deepseek'
  | 'openrouter'
  | 'xai';

interface CLIConfig {
  command: string; // CLI command (e.g., 'claude', 'codex')
  args: string[]; // Arguments — use '{{scenario}}' as prompt placeholder
  outputFormat?: 'text' | 'json' | 'stream-json'; // How to parse stdout (default: 'stream-json')
  timeout?: number; // Command timeout in ms (default: 120000)
}

Host types:

sdk (default) — Programmatic via Vercel AI SDK. Reuses the framework's MCP connection. Requires provider.
cli — CLI-based hosts (e.g., Claude Code, Codex). Spawns a process with its own MCP connection. Requires cli.

MCPHostSimulationResult

The response for a mcp_host case is an MCPHostSimulationResult:

interface MCPHostSimulationResult {
  success: boolean;
  toolCalls: Array<{ name: string; arguments: Record<string, unknown> }>;
  response?: string; // Final LLM response text
  error?: string; // Error message if success=false
  llmDurationMs?: number; // Time in LLM calls (excludes tool execution)
  mcpDurationMs?: number; // Time in MCP tool execution
  conversationHistory?: Array<{ role: string; content: string }>;
}

Cost Considerations

LLM host simulation calls a real LLM API. Approximate costs:

Anthropic Claude 3.5 Sonnet: ~$0.003–0.01 per test (varies by tool count)
OpenAI GPT-4o: ~$0.005–0.02 per test

Recommendation: Use mode: "direct" for regression testing. Use mode: "mcp_host" selectively for tool description quality validation.

Runtime Tool Override Experiments

Use toolOverrides to compare tool metadata variants without changing your eval dataset or MCP server source. The dataset remains the behavioral contract; the override is runtime-only data passed to runEvalDataset.

import { compareEvalRuns } from '@gleanwork/mcp-server-tester';

const variant = {
  id: 'search-description-v2',
  description: 'Clarify that search is for internal docs and policies.',
  tools: {
    search: {
      description:
        'Search internal company documents, policies, wiki pages, and announcements. Use this when the user asks to find company information by topic.',
      inputSchema: {
        type: 'object',
        properties: {
          query: {
            type: 'string',
            description: 'Natural language document or policy query.',
          },
        },
        required: ['query'],
      },
    },
  },
};

const baseline = await runEvalDataset(
  { dataset, defaultLlmIterations: 10 },
  { mcp, testInfo }
);

const candidate = await runEvalDataset(
  {
    dataset,
    defaultLlmIterations: 10,
    toolOverrides: variant,
  },
  { mcp, testInfo }
);

const comparison = compareEvalRuns({
  baseline,
  candidate,
  labels: {
    baseline: 'baseline',
    candidate: variant.id,
  },
});

console.log(`Pass-rate delta: ${comparison.deltaPassRate}`);
console.log(`Tool F1 delta: ${comparison.deltaToolF1 ?? 'n/a'}`);
console.log(`Improved cases: ${comparison.improvedCases.length}`);

toolOverrides.tools is keyed by canonical MCP tool name. v1 supports description and inputSchema replacements only; tool renames, mocked responses, and dataset rewriting are intentionally out of scope.

For a complete runnable harness — including building a structured next-variant proposal from the comparison — see snippets/runtime-tool-override-experiment.ts.

Driving it from an agent: `runVariantExperiment`

The manual loop above — run baseline, inject a variant, compareEvalRuns, build a proposal — is the low-level path. runVariantExperiment wraps that whole loop into a single call so an AI or skill can optimize tool metadata autonomously:

Pass a static variants list for an A/B comparison, or a proposeVariants callback that returns the next candidate(s) from the previous round's evidence (history, bestSoFar).
Candidates are ranked by metric (passRate by default, or toolF1 / toolPrecision / toolRecall) and always compared against the original baseline, so the resulting proposal is directly applicable.
A variant that regresses any case is disqualified (unless allowRegressions: true), so the loop never crowns a description that fixes one case while breaking another.
The result carries a structured proposal with an apply / reject / inconclusive recommendation, the per-tool toolChanges, and the improved/regressed case ids.

The library owns the experiment mechanics; your proposeVariants callback owns the judgment of which variant to try next. runVariantExperiment never edits your MCP server source or dataset — it returns a proposal for you (or an agent) to act on.

import { test, expect } from '@gleanwork/mcp-server-tester/fixtures/mcp';
import {
  loadEvalDataset,
  runVariantExperiment,
  type ToolOverrideVariant,
} from '@gleanwork/mcp-server-tester';

// Static A/B: try a fixed set of tool-description variants and keep the winner.
test('optimize search description (static variants)', async ({
  mcp,
}, testInfo) => {
  const dataset = await loadEvalDataset('./data/host-evals.json');

  const variants: ToolOverrideVariant[] = [
    {
      id: 'search-v2-internal-docs',
      description: 'Clarify that search is for internal knowledge.',
      tools: {
        search: {
          description:
            'Search internal company documents, policies, wiki pages, and announcements. Use this when the user asks to find company information by topic.',
        },
      },
    },
    {
      id: 'search-v3-with-examples',
      description: 'Add example triggers to the search description.',
      tools: {
        search: {
          description:
            'Find internal company knowledge — docs, policies, wikis, announcements. Examples: "find the Q3 planning doc", "what is our PTO policy".',
        },
      },
    },
  ];

  const result = await runVariantExperiment(
    { dataset, variants, metric: 'passRate', defaultLlmIterations: 10 },
    { mcp, testInfo }
  );

  if (result.proposal?.recommendation === 'apply') {
    const pct = (result.proposal.delta * 100).toFixed(1);
    console.log(
      `Apply ${result.winner?.variant.id}: +${pct}% ${result.metric}`
    );
    console.log(
      `Improved cases: ${result.proposal.improvedCaseIds.join(', ')}`
    );
  }

  // The default guard never crowns a variant that regresses a case.
  expect(result.winner?.comparison.regressedCases ?? []).toHaveLength(0);
});

// Agent loop: propose the next variant from the previous round's evidence.
test('optimize search description (agent loop)', async ({ mcp }, testInfo) => {
  const dataset = await loadEvalDataset('./data/host-evals.json');

  const result = await runVariantExperiment(
    {
      dataset,
      metric: 'passRate',
      maxRounds: 4,
      minImprovement: 0.05,
      defaultLlmIterations: 10,
      async proposeVariants({ round, history, bestSoFar }) {
        // An agent inspects bestSoFar / history to decide the next rewrite.
        // Stop early once the best candidate has no remaining failures.
        const stillFailing =
          history.at(-1)?.best?.comparison.unchangedFailures.map((c) => c.id) ??
          [];
        if (round > 0 && stillFailing.length === 0) {
          return [];
        }

        return [
          {
            id: `search-round-${round}`,
            description: `Round ${round} refinement of ${
              bestSoFar?.variant.id ?? 'baseline'
            }.`,
            tools: {
              search: {
                description:
                  'Use search ONLY to find internal company knowledge (docs, policies, wikis, announcements). Convert the request into a concise topic query.',
              },
            },
          },
        ];
      },
    },
    { mcp, testInfo }
  );

  console.log(
    `Stopped after ${result.rounds.length} round(s): ${result.reason}`
  );
  console.log(JSON.stringify(result.proposal, null, 2));
});

Option	Default	Purpose
`variants`	—	Static candidates tried in round 0.
`proposeVariants`	—	Async callback returning the next candidates — the AI hook.
`metric`	`'passRate'`	Ranking metric: `passRate` / `toolF1` / `toolPrecision` / `toolRecall`.
`maxRounds`	`1`	Maximum optimization rounds.
`minImprovement`	`0`	Stop when a round's best gain falls below this.
`allowRegressions`	`false`	Allow a winner that regresses cases.

Project-Based A/B Testing

Run two Playwright projects with different MCP server configurations when the variant is not limited to runtime metadata. This is useful for comparing different server builds, tool behavior, auth scopes, response shapes, transports, or any change that should be exercised through a real MCP server process.

// playwright.config.ts
projects: [
  {
    name: 'baseline',
    use: {
      mcpConfig: {
        transport: 'stdio',
        command: 'node',
        args: ['./dist/server-v1.js'],
      },
    },
  },
  {
    name: 'server-v2',
    use: {
      mcpConfig: {
        transport: 'stdio',
        command: 'node',
        args: ['./dist/server-v2.js'],
      },
    },
  },
];

The MCP reporter groups results by project, letting you compare pass rates side-by-side. Prefer toolOverrides for description and input schema experiments; use project-based A/B testing when the real server surface or implementation changes.