rag-grounding-eval

A reproducible evaluation of RAG grounding fidelity across three frontier LLMs — Claude Sonnet 4.6, GPT-5.4, and Gemini 3.1 Pro — on a corpus of Indian public-company annual reports (FY23 + FY24).

The question behind the artifact: when retrieval is imperfect (as it usually is in production), which models stay disciplined — answering only what the context supports, and abstaining when it doesn't — and which models confabulate?

Thesis

Most public RAG benchmarks reward answering and under-penalize confident wrong answers. In production, the second failure mode matters more: a hallucinated number in a financial workflow is worse than a refusal. This eval measures three orthogonal properties — correctness, grounding, and calibration — and reports a composite Grounding Fidelity Score (GFS) with robustness checks across weightings.

What's evaluated

Corpus	FY23 + FY24 annual reports, 10 Indian listed companies (Reliance, HDFC Bank, Bharti Airtel, Adani Ports, ITC, Tata Motors, Ola Electric, Zomato, Nykaa, Paytm)
Retriever	BAAI/bge-m3 embeddings, ChromaDB, top-k=5, no re-ranker (fixed for all models)
Models	Claude Sonnet 4.6, GPT-5.4, Gemini 3.1 Pro
Questions	200 authored, 4 categories (see METHODOLOGY.md §3)
Runs	3× per (model, question) at temperature 0
Judges	2 cross-family LLM judges per response; 50-response human-labeled subset for Cohen's κ validation (target κ ≥ 0.75)

Full design in METHODOLOGY.md.

Reproducing the results

# 1. Clone and enter
git clone https://github.com/<user>/rag-grounding-eval && cd rag-grounding-eval

# 2. Install (uv recommended; ~30s)
uv sync

# 3. Configure
cp .env.example .env
# Fill in ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_API_KEY

# 4. Build corpus (downloads ~300MB of PDFs, chunks, embeds, indexes)
uv run rge corpus build

# 5. Dry-run cost estimate (no API calls)
uv run rge eval run --dry-run

# 6. Full eval (estimated cost <\$100; hard cap enforced)
uv run rge eval run

# 7. Regenerate tables + plots from data/results/
uv run rge report generate

Re-running with existing results in data/results/ is a no-op unless you pass --force. Budget ledger persists in data/results/.budget_ledger.json and survives restarts.

Repository layout

rag-grounding-eval/
├── README.md                # this file
├── METHODOLOGY.md           # eval design, rubric, statistical protocol
├── WRITEUP.md               # blog-ready results writeup (post-Milestone 7)
├── pyproject.toml
├── .env.example
├── LICENSE                  # MIT (code)
├── src/
│   ├── corpus/              # PDF download, chunking, sources.yaml
│   ├── retrieval/           # bge-m3 embedder, ChromaDB wrapper, retriever
│   ├── questions/           # Pydantic schemas, loader, validator, authoring CLI
│   ├── models/              # Anthropic / OpenAI / Google client adapters
│   ├── eval/                # runner, scorer, judge, budget tracker, stats
│   └── report/              # table + plot generators
├── data/
│   ├── corpus/              # PDFs (gitignored); sources.yaml tracked
│   ├── questions/           # questions.jsonl (CC-BY-4.0)
│   ├── results/             # raw eval outputs (part of the artifact)
│   └── human_labels/        # 50-response kappa-validation set
├── notebooks/               # exploration, judge validation, final analysis
├── configs/                 # frozen run configs (e.g., published.yaml)
└── tests/

Licensing

• Code: MIT License — see LICENSE.
• Dataset + writeup (data/questions/, WRITEUP.md): CC-BY-4.0. Headers in those files. If you build on the question set, cite this repo.

Citation

@misc{arora2026raggrounding,
  author       = {Arora, Hriday},
  title        = {rag-grounding-eval: A Reproducible Evaluation of RAG Grounding Fidelity Across Frontier LLMs},
  year         = {2026},
  howpublished = {\url{https://github.com/<user>/rag-grounding-eval}},
}

Status

Under active development. See milestones in the project thread. Current: Milestone 1 — scaffold.