Awesome AI Agent Evaluation

Chinese

Benchmarks, eval harnesses, papers, datasets, and production checks for AI agents.

Focused on resources that make agent behavior easier to test, compare, debug, or monitor.

Included	Left out
Benchmarks with clear tasks	Generic agent demos
Frameworks for running evals	Prompt collections without evaluation
Papers with reusable methods	Marketing pages with no technical detail

Where to start

If you care about...	Read...
Software engineering tasks	Coding Agent Evaluation
Website and UI interaction	Browser / Web Agent Evaluation
API and function calling	Tool-Use and Function Calling
Regression and recovery	Reliability & Failure Recovery
Prompt injection and harmful actions	Safety & Robustness
Logs, traces, cost, and drift	Production Monitoring

Contents

Browse the list

• Evaluation Basics
• Benchmarks
• Evaluation Frameworks
• Coding Agent Evaluation
• Browser / Web Agent Evaluation
• Tool-Use and Function Calling
• Multi-Agent Evaluation
• Safety & Robustness
• Reliability & Failure Recovery
• Cost, Latency, and Efficiency
• Human Evaluation Rubrics
• Production Monitoring
• Papers
• Datasets
• Reports & Case Studies
• Related Awesome Lists
• Contributing

Evaluation Basics

• OpenAI Evals — Open-source eval framework and registry; useful as a reference for repeatable LLM test design. framework evals regression
• OpenAI Cookbook: Evaluating model performance — Practical examples for turning a task into a small, inspectable eval. guide evaluation-basics llm-as-judge
• Hamel Husain: Your AI Product Needs Evals — A good product-minded primer on why generic benchmarks are not enough. guide product-evals methodology
• Anthropic: Building effective agents — Useful framing for deciding whether a workflow needs an agent at all. guide agent-design methodology
• LangChain Agent Evals — Trajectory-level checks for tool calls, intermediate steps, and final answers. guide agent-trajectory framework
• Phoenix LLM Evals — Evaluation workflows built around traces, datasets, and experiment runs. guide observability llm-as-judge

Benchmarks

• SWE-bench — Real GitHub issues turned into patch tasks for coding agents. benchmark coding-agent real-world-tasks
• SWE-bench Verified — A smaller SWE-bench split with human-checked task quality. benchmark coding-agent verified
• SWE-bench Live — Newer issue-resolution tasks, useful when contamination is a concern. benchmark coding-agent live
• SWE-bench Multimodal — SWE-style issue tasks where visual context also matters. benchmark coding-agent multimodal
• SWE-bench Multilingual — Issue-resolution tasks spread across multiple programming languages. benchmark coding-agent multilingual
• GAIA — General assistant tasks that often require tools, web search, files, and multi-step reasoning. benchmark general-agent tool-use
• AgentBench — A broad interactive benchmark across OS, database, web, game, and reasoning environments. benchmark general-agent interactive
• OSWorld — Desktop computer-use tasks with execution-based grading. benchmark computer-use desktop-agent
• macOSWorld — macOS GUI tasks, with multilingual coverage and a safety-oriented subset. benchmark computer-use gui-agent
• AndroidWorld — Android app tasks with programmatic checks for mobile agents. benchmark mobile-agent computer-use
• ClawBench — Live-website browser-agent benchmark; 283 everyday tasks (V1 153 + V2 130) across 163 platforms. Two-stage scoring: HTTP-request interception at per-task URL/method schema + LLM judge on the intercepted payload. Paper · Live leaderboard
• WebArena — Self-hosted websites for testing whether browser agents actually change state correctly. benchmark browser-agent web
• VisualWebArena — WebArena-style tasks where screenshots and visual grounding matter. benchmark browser-agent multimodal
• WebArena Verified — Verified WebArena tasks with more deterministic evaluators. benchmark browser-agent verified
• WorkArena — Enterprise-style ServiceNow tasks for browser agents. benchmark browser-agent enterprise
• BrowserGym — Shared browser-agent environment wrapping WebArena, WorkArena, MiniWoB, and related tasks. framework browser-agent benchmark-suite
• Terminal-Bench — Terminal tasks for shell use, debugging, and system operations. benchmark terminal-agent tool-use
• tau-bench — Customer-service tasks where the agent must talk, follow policy, and call APIs. benchmark tool-use multi-turn
• tau2-bench / tau3-bench — Newer tau-style tasks for tool-agent-user interaction. benchmark tool-use enterprise
• Berkeley Function Calling Leaderboard — Function-calling tests covering API selection, arguments, multi-turn calls, and executability. benchmark function-calling tool-use
• ToolBench — API-heavy tasks for studying tool-use behavior at scale. benchmark tool-use api
• API-Bank — API selection and multi-step tool-use tasks. benchmark tool-use api
• WebShop — Shopping tasks that involve search, comparison, and goal-directed web actions. benchmark browser-agent commerce
• Mind2Web — Real website interaction traces for web navigation tasks. benchmark browser-agent web-navigation
• MiniWoB++ — Small synthetic browser tasks for fast, controlled experiments. benchmark browser-agent synthetic

Evaluation Frameworks

• Inspect AI — Evaluation framework with tools, sandboxes, scorers, logs, and a result viewer. framework evals safety
• Inspect Evals — Ready-made Inspect evals, including agent, coding, and safety tasks. framework benchmark-registry safety
• OpenAI Evals — Custom evals, model-graded checks, and reusable eval registries. framework llm-evaluation regression
• DeepEval — Pytest-style evals for LLM apps, with metrics for agents, tools, safety, and conversations. framework testing agent-eval
• promptfoo — Regression tests for prompts, models, and agents; works well in CI. framework regression ci
• Ragas — Metrics for RAG, tool use, and agent-style workflows. framework metrics rag
• LangSmith — Tracing, datasets, experiments, human review, online evaluators, and agent trajectory checks. platform observability agent-eval
• LangChain AgentEvals — Prebuilt evaluators for agent trajectories and tool-call behavior. framework agent-trajectory tool-use
• Arize Phoenix — Open-source tracing, dataset experiments, and LLM evaluators. platform observability evaluation
• Langfuse — Traces, prompts, evals, cost, latency, and production health in one place. platform observability production
• Braintrust — Datasets, experiments, scorers, tracing, and prompt iteration for AI products. platform experiments regression
• Giskard — Automated checks for quality, bias, hallucination, and security issues. framework quality safety
• TruLens — Instrumentation and feedback functions for LLM application behavior. framework observability feedback
• OpenTelemetry GenAI Semantic Conventions — Common trace and metric fields for GenAI calls. standard observability telemetry
• OpenInference — Open instrumentation conventions for LLM, RAG, and agent traces. standard observability tracing

Coding Agent Evaluation

• SWE-bench — The default issue-to-patch benchmark family for coding agents. benchmark coding-agent github-issues
• SWE-agent — Open coding agent and harness built around SWE-bench-style tasks. agent coding-agent swe-bench
• OpenHands — Open-source software development agent for end-to-end coding workflows. agent coding-agent open-source
• Aider — Terminal pair-programming agent; useful for repo-editing regression tasks. agent coding-agent terminal
• BigCodeBench — Practical code-generation tasks with execution tests. benchmark code-generation execution
• RepoBench — Repository-level code completion and retrieval benchmark. benchmark coding-agent repository-context
• HumanEval — Classic execution-based code benchmark; still useful as a baseline. benchmark code-generation baseline
• EvalPlus — Harder test suites for HumanEval and MBPP. benchmark code-generation execution
• LiveCodeBench — Newer contest-style coding problems, designed with contamination in mind. benchmark code-generation live

Browser / Web Agent Evaluation

• WebArena — Realistic self-hosted web tasks with state-based success checks. benchmark browser-agent self-hosted
• WorkArena — ServiceNow-based benchmark for enterprise knowledge-work tasks. benchmark browser-agent enterprise
• BrowserGym — Gym-style interface and shared action space for browser-agent benchmarks. framework browser-agent environment
• AgentLab — Tooling for running and analyzing BrowserGym-compatible web agents. framework browser-agent experiments
• Mind2Web — Real-website navigation tasks from user instructions. benchmark browser-agent navigation
• WebLINX — Browser interaction demonstrations from real-world web tasks. dataset browser-agent demonstrations
• VisualWebArena — Web tasks that need visual grounding, not just DOM text. benchmark browser-agent visual
• MiniWoB++ — Small controlled UI tasks for quick browser-agent iteration. benchmark browser-agent synthetic
• browser-use benchmark — Browser automation tasks spanning established and custom task sets. benchmark browser-agent automation

Tool-Use and Function Calling

• Berkeley Function Calling Leaderboard — Function-calling tests for relevance, arguments, executability, and multi-turn use. benchmark function-calling tool-use
• tau2-bench / tau3-bench — Multi-turn tool-use tasks with simulated users, policies, and database-state scoring. benchmark tool-use customer-service
• ToolBench — Large API-oriented benchmark for tool learning and tool use. benchmark tool-use api
• API-Bank — API choice, API calling, and multi-step tool chaining. benchmark tool-use api
• Gorilla — Datasets and methods focused on accurate API calls. dataset function-calling api
• ToolSandbox — Stateful tool-use tasks in simulated environments with execution traces. benchmark tool-use stateful
• Ragas Tool Call Metrics — Tool-call and agent-goal metrics for application-level evals. metrics tool-use agent-eval

Multi-Agent Evaluation

• AutoGenBench — Repeatable tasks and metrics for AutoGen multi-agent workflows. benchmark multi-agent framework
• AgentVerse — Framework for simulating groups of collaborating agents. framework multi-agent simulation
• MetaGPT — Software-company-style multi-agent workflows; useful for coordination studies. framework multi-agent software-engineering
• ChatDev — Multi-agent software development via role-based conversation. framework multi-agent collaboration
• CAMEL — Role-playing and society simulation infrastructure for agent research. framework multi-agent simulation

Safety & Robustness

• AgentDojo — Indirect prompt-injection benchmark for tool-using agents. benchmark safety prompt-injection
• AgentHarm — Harmful-request tasks for agent settings, including cybercrime and fraud. benchmark safety agent
• b3: Backbone Breaker Benchmark — Agentic security tasks around exfiltration, compromise, and misuse. benchmark security agent
• Purple Llama CyberSecEval — Cybersecurity safety evals for LLMs and agentic systems. benchmark security cyber
• HarmBench — Standardized adversarial tasks for harmful-behavior robustness. benchmark safety robustness
• PromptBench — Prompt robustness and adversarial prompt sensitivity tests. benchmark robustness prompts
• Garak — Scanner for jailbreaks, prompt injection, data leakage, and related LLM app risks. tool red-teaming security
• PyRIT — Automation framework for AI red-team workflows. tool red-teaming safety

Reliability & Failure Recovery

• Terminal-Bench — Good fit for studying recovery in shell-based tasks. benchmark reliability terminal
• tau2-bench / tau3-bench — Captures policy, conversation, and tool-use failures in support-style workflows. benchmark reliability tool-use
• LangSmith Online Evaluators — Online scoring for production traces and regressions. platform monitoring regression
• promptfoo Assertions — Deterministic, model-graded, and custom assertions for regression suites. framework regression ci
• DeepEval Metrics — Agent, tool-use, conversation, safety, and custom metrics. framework metrics regression
• Invariant — Guardrails and tests over agent traces, tool calls, and app behavior. framework guardrails agent-traces

Cost, Latency, and Efficiency

• Langfuse — Trace-level cost, latency, model usage, prompts, and eval scores. platform cost observability
• Helicone — Open-source request logging, cost tracking, and usage analytics. platform cost monitoring
• LiteLLM Proxy — Model routing, budgets, logging, and spend controls. tool cost routing
• OpenLLMetry — OpenTelemetry-compatible traces and metrics for LLM apps. tool observability telemetry
• OpenTelemetry GenAI Semantic Conventions — Shared telemetry fields for GenAI cost, latency, and usage analysis. standard telemetry metrics
• Phoenix Tracing — LLM, retrieval, and tool-use traces for debugging and latency analysis. platform latency tracing

Human Evaluation Rubrics

• LangSmith Annotation Queues — Human review queues for traces and structured feedback. human-eval annotation platform
• Braintrust Human Review — Human review over experiments and production traces. human-eval annotation platform
• Phoenix Human Feedback — Human labels and annotations alongside automated evals. human-eval annotation observability
• OpenAI Evals Model-Graded Templates — Reusable patterns for rubric-based and model-graded evals. rubric llm-as-judge templates
• Langfuse Scores — Human, heuristic, and model-based scores on traces. human-eval scoring monitoring

Production Monitoring

• LangSmith — Tracing, datasets, online evals, human review, and deployment monitoring. platform production monitoring
• Langfuse — Open-source observability, prompt management, evals, and production health tracking. platform production observability
• Arize Phoenix — Tracing, experiments, evals, prompt iteration, and self-hosted debugging. platform production observability
• Braintrust — Offline experiments, production logging, scorers, and datasets. platform production evalops
• Confident AI — Hosted reports, monitoring, and quality tracking around DeepEval. platform production monitoring
• Helicone — Request logs, user analytics, cost, latency, and provider behavior. platform monitoring cost
• WhyLabs LangKit — Text quality, prompt-injection, toxicity, and drift signals. tool monitoring quality
• TruLens — Feedback-based monitoring and evals for LLM applications. platform monitoring feedback

Papers

• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — The paper behind repository-level issue-to-patch evaluation. paper coding-agent benchmark
• AgentBench: Evaluating LLMs as Agents — Multi-environment benchmark for interactive LLM agents. paper general-agent benchmark
• GAIA: A Benchmark for General AI Assistants — Hard assistant tasks involving reasoning, tools, and external resources. paper general-agent benchmark
• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments — Desktop computer-use benchmark with execution-based grading. paper computer-use benchmark
• WebArena: A Realistic Web Environment for Building Autonomous Agents — Self-hosted websites for functional browser-agent evaluation. paper browser-agent benchmark
• VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks — WebArena-style tasks with visual grounding requirements. paper browser-agent multimodal
• WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? — Enterprise knowledge-work tasks for browser agents. paper browser-agent enterprise
• The BrowserGym Ecosystem for Web Agent Research — Shared environment and tooling for browser-agent research. paper browser-agent framework
• tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — Multi-turn tool-agent-user interaction in realistic domains. paper tool-use multi-turn
• Berkeley Function-Calling Leaderboard — Background on executable function-calling evaluation and BFCL. paper function-calling tool-use
• ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs — ToolBench and methods for API-using LLM agents. paper tool-use api
• API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs — API-augmented LLM evaluation and training. paper tool-use api
• AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents — Utility-security tradeoffs under prompt-injection attacks. paper safety prompt-injection
• AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents — Harmful-behavior benchmark for agentic settings. paper safety agent

Datasets

• SWE-bench on Hugging Face — Dataset of real GitHub issue-resolution tasks for coding-agent evaluation. dataset coding-agent github-issues
• SWE-bench Verified on Hugging Face — Engineer-validated subset for more reliable coding-agent measurement. dataset coding-agent verified
• GAIA Dataset — General assistant benchmark data with tasks requiring tools and external resources. dataset general-agent tool-use
• Mind2Web Dataset — Real website navigation task data for training and evaluating web agents. dataset browser-agent navigation
• WebLINX Dataset — Browser interaction demonstrations for real-world web tasks. dataset browser-agent demonstrations
• AgentHarm Dataset — Harmful-behavior tasks for agent safety evaluation. dataset safety agent
• BFCL Dataset — Function-calling benchmark data for tool invocation evaluation. dataset function-calling tool-use
• ToolBench Dataset — Tool-use data for API-oriented agent evaluation and training. dataset tool-use api

Reports & Case Studies

• SWE-bench Leaderboard — Submitted systems across SWE-bench variants. leaderboard coding-agent benchmark
• GAIA Leaderboard — Public submissions for the GAIA assistant benchmark. leaderboard general-agent benchmark
• OSWorld Leaderboard — Computer-use agent results on desktop task suites. leaderboard computer-use benchmark
• Terminal-Bench Leaderboard — Terminal-agent results and benchmark releases. leaderboard terminal-agent benchmark
• BFCL Leaderboard — Function-calling results across model providers. leaderboard function-calling tool-use
• Inspect Evals Documentation — Runnable Inspect evals for safety, coding, agent, and reasoning tasks. docs benchmark-registry safety

Related Awesome Lists

• Awesome-LLM-Eval — Broader LLM evaluation methods, datasets, tools, and leaderboards. awesome-list llm-evaluation related
• awesome-llm-agents — Broader LLM agent papers, projects, frameworks, and resources. awesome-list llm-agents related
• awesome-ai-agents — Agent frameworks, tools, and examples across the ecosystem. awesome-list ai-agents related
• awesome-production-machine-learning — Useful for connecting agent evaluation to production ML monitoring and operations. awesome-list production monitoring

Contributing

Contributions are welcome. Please read CONTRIBUTING.md before opening a PR.

License

This project is released under CC0-1.0.