Benchmarks

This folder provides a small benchmark layer for the Agent Engineering Roadmap.

The goal is not to compare models. The goal is to check whether the example agents preserve important engineering behaviors.

Run

python benchmarks/benchmark_runner.py

Current Checks

Category	Check	What it protects
Tool use	Calculator result	Tool call correctness
RAG	Grounded approval answer	Retrieval and answer grounding
Workflow	Destructive action gate	Approval before high-risk execution
Security	Prompt injection blocked	Untrusted retrieval isolation
Observability	Guardrail trace exists	Replayable production debugging
Cost-latency	Model route respects budget	Cost and latency-aware task routing
Runtime	Checkpoint resume completes	Durable long-running workflows
MCP	Read-only token cannot call tools	Capability-scoped authorization
Memory governance	PII is redacted before storage	Safe memory retention
Identity permission	Missing scope is denied	Least-privilege agent access
Advanced eval	Release gate passes	Regression, safety, adversarial, and trace checks

Why This Helps

Many agent demos look impressive but do not stay reliable after changes.

This benchmark gives contributors a quick signal:

• Did tool behavior break?
• Did retrieval grounding regress?
• Did approval gates still trigger?
• Did prompt injection defenses still work?
• Did traces still capture guardrail decisions?
• Did cost-aware routing still choose the expected model?
• Did durable workflows still resume?
• Did MCP gateway permissions still deny unsafe access?
• Did memory governance still redact PII?
• Did agent permissions still deny missing scopes?
• Did the advanced release gate still pass?

Add A Benchmark

Add a new check when you add a new production behavior.

A good benchmark should:

• run without API keys
• test one behavior
• fail loudly
• print a useful detail line
• be stable in GitHub Actions