🌟 Introduction

This repository tracks notable platforms and open-source projects for AI benchmarking and LLM evaluation. These tools and leaderboards provide standardized, reproducible ways to measure model performance across reasoning, coding, knowledge, safety, instruction following, and real-world capabilities.

Keywords: LLM Evaluation, AI Benchmarking, Model Leaderboard, GPT-4 Benchmarks, LLM Metrics, Open Source AI, Evaluation Frameworks, RAG Evaluation, Code Generation Benchmarks.

Examples include LMSYS Chatbot Arena, Artificial Analysis, Open LLM Leaderboard (Hugging Face), Vellum LLM Leaderboard, and LiveBench (the category leaders). Tools listed here emphasize rigorous evaluation, human voting, automated metrics, contamination-free testing, and multi-dimensional scoring.

Open-source emphasis: This section is heavily expanded with every major active project for self-hosting, custom benchmarking, local evaluation, and full transparency — ideal for researchers, companies, and developers who want reproducible and auditable results.

Contributions welcome! Open a PR to add/update entries. Keep descriptions factual and link to official sites.

📚 Table of Contents

• ☁️ SaaS / Hosted Platforms
• 🛠️ Open-Source GitHub Projects
• 🤝 How to Contribute
• ⚠️ Disclaimer

☁️ SaaS / Hosted Platforms

🏗️ Core Benchmarking Platforms & Leaderboards

• LMSYS Chatbot Arena (formerly Chatbot Arena)
  The most popular human-voted blind arena for comparing LLMs through crowdsourced Elo ratings. Highly trusted for real-world conversational ability.

• Artificial Analysis
  Independent, high-quality benchmarking platform with detailed metrics on quality, speed, price, latency, and context window.

• AI Model Index
  Reconciles source-level evidence from major AI model leaderboards into transparent, auditable indexes with coverage, confidence, benchmark-version, and source-policy context.

• Open LLM Leaderboard (Hugging Face)
  The gold standard automated leaderboard for open models across multiple academic and reasoning benchmarks.

• LiveBench
  Contamination-resistant benchmark with frequently updated questions to prevent data leakage.

• Vellum LLM Leaderboard
  Enterprise-focused leaderboard with practical business use case evaluations.

🧪 Advanced & Specialized Platforms

Other notable mentions: Arena (various forks), HELM, BigBench, and industry-specific benchmarks (e.g., Legal, Medical, Finance).

🛠️ Open-Source GitHub Projects

📐 Dedicated Benchmarking & Evaluation Frameworks

• DeepEval
  Popular open-source framework for LLM evaluation with RAGAs, G-Eval, and custom metrics.
• lm-evaluation-harness
  The most widely used open-source framework for evaluating LLMs. Supports dozens of benchmarks with standardized, reproducible evaluation.
• Big-Bench
  Google’s massive collaborative benchmark with over 200+ diverse tasks.
• HELM (Holistic Evaluation of Language Models)
  Stanford’s comprehensive benchmarking suite covering accuracy, calibration, robustness, fairness, bias, and toxicity.
• LightEval
  Hugging Face’s lightweight and fast evaluation library optimized for large-scale benchmarking.
• EvalPlus
  Rigorous evaluation framework focused on code generation with extended test cases (HumanEval+, MBPP+).
• LiveBench
  Open-source contamination-free benchmark with regularly refreshed questions across multiple categories.
• DeepSWE
  High-resolution benchmark from Datacurve for coding agents, featuring zero-pollution, long-horizon tasks across 100+ languages with behavioral verifiers.
• LangChain / LangSmith Evaluators
  Production-grade evaluation tools with tracing and custom metric support.
• StructEval
  TMLR benchmark for structured-output generation and conversion across 2,035 examples, 18 text and renderable visual formats, and 44 task types. Paper · Leaderboard
• Dr. Bench
  Open-source benchmark for deep-research agents, evaluating long-form reports on 214 expert-curated tasks with semantic quality, topical focus, and retrieval-trustworthiness metrics. Paper
• Open LLM Leaderboard
  Official open-source codebase behind Hugging Face’s popular leaderboard. Fully customizable for running your own evaluations.

Additional Strong Open-Source Options

• MT-Bench — Multi-turn conversation benchmarking (from LMSYS).
• RAGAS — Specialized framework for evaluating Retrieval-Augmented Generation systems.
• PromptBench — Robustness and adversarial testing for prompts.
• ClawBench — Live-website browser-agent benchmark; 283 everyday tasks (V1 153 + V2 130) across 163 live platforms. Two-stage scoring: HTTP-request interception at per-task URL/method schema + LLM judge on intercepted payload. Paper · Live leaderboard.
• Many community forks and extensions of lm-evaluation-harness for domain-specific benchmarks (finance, legal, medical, etc.).
• SafetyBench — Safety and alignment evaluation suites.
• LLM-KG-Bench — Knowledge graph and structured reasoning evaluation.
• VIrAL (Viral Intelligence and Risk Assessment Link) — Comprehensive benchmark for Nucleotide Foundation Models (NFMs) evaluating viral genomics understanding and latent biosecurity risks.
• AgentBench — For evaluating autonomous AI agents.

Frameworks for building custom benchmarks: Use lm-evaluation-harness + LightEval + DeepEval combined with LangGraph for comprehensive, reproducible evaluation pipelines that can run locally or at scale.

🤝 How to Contribute

1. Fork the repo.
2. Add/edit entries in README.md (follow existing format).
3. Include: name, link, 1–2 sentence description, and whether it's SaaS or open-source.
4. Submit PR with a short explanation.

Star the repo if you find it useful! ⭐

⬆️ Back to Top

⚠️ Disclaimer

• This is a community-curated list — not exhaustive and not an endorsement.
• Benchmark scores can be misleading without understanding the methodology, contamination risks, and specific use cases. Always interpret results in context.
• Different benchmarks favor different model strengths — no single leaderboard tells the full story.

---

Made with ❤️ for AI researchers, LLM engineers, product teams, and open-source enthusiasts.
Let's make AI evaluation more transparent, rigorous, and reproducible.

⬆️ Back to Top

📈 Star History