README.md

Test your MCP server against the real protocol. No mocks. No imports. No language lock-in.

mcp-assert connects to your server exactly like Claude, Cursor, or any MCP client would: real stdio/SSE/HTTP transport, full initialize handshake, actual tool calls. It checks responses against expectations you define in YAML. If it passes mcp-assert, it works with every MCP client.

Your YAML        ──→  mcp-assert  ──→  MCP Server
(inputs + assertions)    (client)        (any language)
                            │
                        Pass / Fail

Your server can't tell the difference

mcp-assert speaks the full MCP protocol: initialize handshake, tools/list discovery, tools/call with real arguments. It finds bugs that unit tests miss because it tests over the wire, not in-process.

Adopted in production

• Wyre Technology: 25 MCP servers tested via shared baseline workflow using mcp-assert-action
• Ant Group (AntV): integrated into CI within 3 days of launch
• Vera: recommended test harness on project roadmap (#529)
• Fix PRs merged: Google, Grafana, LangChain, official MCP SDKs

The testing standard for MCP, like pytest for Python or Jest for JavaScript.

Add it to any MCP server project in one line:

- uses: blackwell-systems/mcp-assert-action@v1
  with:
    suite: evals/

Install

# npm (no Go required)
npx @blackwell-systems/mcp-assert

# pip (no Go required)
pip install mcp-assert

# Go
go install github.com/blackwell-systems/mcp-assert/cmd/mcp-assert@latest

# Homebrew
brew install blackwell-systems/tap/mcp-assert

# Docker
docker run blackwellsystems/mcp-assert audit --server "npx my-server"

# Snap (Linux)
sudo snap install mcp-assert --classic

# Scoop (Windows)
scoop bucket add blackwell-systems https://github.com/blackwell-systems/scoop-bucket
scoop install mcp-assert

# Winget (Windows)
winget install BlackwellSystems.mcp-assert

# curl | sh (macOS / Linux)
curl -fsSL https://raw.githubusercontent.com/blackwell-systems/mcp-assert/main/install.sh | sh

Quick Start

Audit any MCP server in seconds. No setup.

Point it at any server:

mcp-assert audit --server "npx my-mcp-server"

  Server: my-server
  Transport: stdio
  Score: 83%

  ✓ read_query      1ms  [E000] responds, returns content
  ✗ create_table    0ms  [E201] internal error: panic: nil pointer...
  ✓ list_tables     1ms  [E000] responds, returns content

  3 tools tested, 2 healthy, 1 crashed

Structured error codes classify issues instantly. See Error Reference for all 24 codes.

# Audit + generate starter YAML for CI
mcp-assert audit --server "npx my-mcp-server" --output evals/

# Edit the generated YAMLs: add expected content, setup steps, multi-step flows

# Run in CI with regression detection
mcp-assert ci --suite evals/ --threshold 95

Write assertions from scratch

# Scaffold your first assertion
mcp-assert init evals                   # Or: init evals --server "my-server" for auto-generation

# Run it
mcp-assert run --suite evals/ --fixture evals/fixtures

See the Getting Started guide for a full walkthrough.

Already using Vitest, Jest, Bun, PHPUnit, or pytest?

# Vitest
npm install -D @blackwell-systems/vitest-mcp-assert

// mcp.test.ts
import { describeMcpSuite } from '@blackwell-systems/vitest-mcp-assert'
describeMcpSuite('mcp server', 'evals/')

# pytest
pip install pytest-mcp-assert
pytest --mcp-suite evals/

Everything you can do

Command	What it does	Setup required
audit --server "..."	Scan any server, classify every tool as healthy/crashed/timed out	None
fuzz --server "..."	Throw adversarial inputs at every tool, find crashes and hangs	None
init --server "..."	Generate a complete test suite from tools/list + capture snapshots	None
run --suite evals/	Run YAML assertions, report pass/fail	YAML files
ci --suite evals/	Run with thresholds, baselines, JUnit XML, GitHub Step Summary	YAML files
coverage --suite evals/ --server "..."	Report which tools have assertions and which don't	YAML files
snapshot --suite evals/ --update	Capture responses as golden files for regression detection	YAML files
watch --suite evals/	Re-run on YAML changes, show diffs when status flips	YAML files
matrix --languages go:gopls,ts:tsserver	Same suite across multiple language servers	YAML files
intercept --server "..." --trajectory t.yaml	Proxy between agent and server, capture live tool call trace	Trajectory YAML
lint --server "..."	24 static analysis rules for agent usability; --fix auto-generates schema improvements	None

Start with audit (zero setup), then fuzz (adversarial testing), then init (generates everything), then customize the YAML for your specific assertions.

Zero-Effort Coverage

# Generate stub assertions for every tool the server exposes
mcp-assert generate --server "my-mcp-server" --output evals/ --fixture ./fixtures

# Capture actual outputs as snapshots
mcp-assert snapshot --suite evals/ --server "my-mcp-server" --update

# Assert nothing changed
mcp-assert run --suite evals/ --server "my-mcp-server"

Lint + Auto-Fix

Static analysis catches schema issues without executing tools. 24 rules detect problems that cause agents to fail:

mcp-assert lint --server "npx my-mcp-server"

  E  E103   create_entities       Required parameter "entities" has no description
  W  W114   generate_chart        Input schema is 5 levels deep. LLMs struggle with nesting
  W  W112   (server)              Server exposes 27 tools. LLM accuracy degrades beyond 20

5 error(s), 11 warning(s)

Auto-generate fixes:

mcp-assert lint --server "npx my-mcp-server" --fix

memory-server: 9 tools, 25 findings, 23 auto-fixable

  E103   create_entities   Add description: "The entities value (array)"
  W109   search_nodes      Add examples to "query": [search term]
  W116   read_graph        Append: "Returns the graph data as JSON."

23 fixes generated.

Use --strict in CI to fail on warnings:

mcp-assert lint --server "..." --strict --threshold 0

How It Differs From LLM-as-Judge Frameworks

For deterministic tools, mcp-assert is the better fit. For subjective outputs, LLM-as-judge frameworks remain the right choice. Use both if your server mixes tool types.

Dimension	LLM-as-judge eval frameworks	mcp-assert
Best for	Subjective outputs (prose, creative content)	Deterministic outputs (data, state, validation)
Grading	Language model scoring (flexible, costly)	Assertion-based (exact, free)
Speed	Seconds per test (LLM round-trip)	Milliseconds per test (no LLM)
CI cost	API calls on every run	Zero external dependencies
Reliability	Not measured	pass@k / pass^k per assertion
Regression	Not supported	Baseline comparison, fail on backslide
Multi-language	Not supported	Same assertion across N language servers

Why not just write tests?

You'd need MCP protocol bootstrapping, a server-agnostic runner (your Go tests can't test your TypeScript server), and eval features (regression detection, Docker isolation, JUnit output). mcp-assert handles all of that. One YAML file, any server, any language.

CI Integration

Use the mcp-assert GitHub Action for zero-setup CI:

- uses: blackwell-systems/mcp-assert-action@v1
  with:
    suite: evals/
    threshold: 95

Downloads the binary, runs assertions, uploads JUnit XML results, writes GitHub Step Summary. No Go toolchain required on your runners.

Or run directly:

mcp-assert ci --suite evals/ --threshold 95 --junit results.xml

See the CI Integration guide for JUnit XML, markdown summaries, badges, and regression detection.

pytest Integration

Run mcp-assert assertions as pytest test items:

pip install pytest-mcp-assert
pytest --mcp-suite evals/

Each YAML file becomes a pytest Item with pass/fail/skip semantics. Configure via pyproject.toml:

[tool.pytest.ini_options]
mcp_suite = "evals/"
mcp_fixture = "fixtures/"

Then just run pytest. See pytest-plugin/README.md for all options.

Vitest Integration

Run mcp-assert assertions as Vitest tests:

npm install -D @blackwell-systems/vitest-mcp-assert

Auto-discover all YAML files in a directory:

// mcp.test.ts
import { describeMcpSuite } from '@blackwell-systems/vitest-mcp-assert'
describeMcpSuite('mcp server', 'evals/')

Or run individual assertions:

import { test } from 'vitest'
import { runMcpAssert } from '@blackwell-systems/vitest-mcp-assert'
test('echo tool', () => runMcpAssert('evals/echo.yaml'))

Same YAML files work across Vitest, pytest, and the CLI. See vitest-plugin/README.md for all options.

Documentation

Full documentation is available at blackwell-systems.github.io/mcp-assert:

• Getting Started: install, scaffold, first run
• Writing Assertions: YAML format, all 18 assertion types + 4 trajectory types, 8 block types, 6 test framework plugins (pytest, Vitest, Jest, Bun, PHPUnit, Go test), setup steps, capture, fixtures
• CLI Reference: full command reference with flags and examples
• Examples: 65 example suites across 8 languages (606 assertions)
• CI Integration: GitHub Action, JUnit XML, regression detection
• Badge: add the "Works with mcp-assert" badge to your server README
• Architecture: internals and design decisions
• Roadmap: what's shipped and what's next
• Scorecard: 32 bugs found across 13 servers, 9 fix PRs submitted, 58 servers scanned

License

MIT