Methodology

Version 0.1 — Milestone 1 draft. Structured for later extension to an arXiv preprint: numbered sections, abstract, figure-caption conventions, and references are already in place.

Abstract

We evaluate the grounding fidelity of three frontier large language models — Claude Sonnet 4.6, GPT-5.4, and Gemini 3.1 Pro — on a retrieval-augmented question-answering task over a corpus of Indian public-company annual reports (fiscal years 2023 and 2024). Unlike prior RAG benchmarks that emphasize answer accuracy, we score each response on three orthogonal dimensions — correctness, grounding, and calibration — and report a composite Grounding Fidelity Score (GFS). The 200-question dataset is asymmetric by design: 65 questions (32.5%) are adversarial, surfacing semantically similar but wrong content in top-k retrieval, and 45 (22.5%) are deliberately unanswerable from the corpus. Each response is scored by two cross-family LLM judges; judge reliability is validated on a 50-response human-labeled subset using Cohen's κ with a pre-registered threshold of κ ≥ 0.75. We report bootstrap 95% confidence intervals, pairwise McNemar's tests with Bonferroni correction, and a robustness check under equal-weighted scoring.

1. Research Questions

• RQ1. How do frontier LLMs differ in their tendency to produce factually correct, context-grounded answers under realistic (imperfect) retrieval?
• RQ2. How well does each model abstain when the retrieved context does not contain the answer?
• RQ3. When retrieval surfaces plausible distractors, do models maintain grounding discipline, or do they blend retrieved context with parametric memory?
• RQ4. Do relative model rankings depend on the weighting of the composite score, or are they robust across reasonable weightings?

2. Corpus

2.1 Source material

Ten Indian listed companies, selected for sectoral and stylistic diversity:

Company	Sector
Reliance	Energy + retail + telecom conglomerate
HDFC Bank	Banking
Bharti Airtel	Telecom
Adani Ports	Infrastructure
ITC	Diversified FMCG
Tata Motors	Legacy auto
Ola Electric	EV / auto-tech
Zomato	Consumer tech (food)
Nykaa	Consumer tech (beauty)
Paytm	Fintech

For each company we ingest annual reports for FY23 and FY24 (20 documents total). URLs, checksums, and accessed-on dates are pinned in data/corpus/sources.yaml.

2.2 Extraction and chunking

• Primary extraction: pypdf.
• Fallback for table-heavy sections: pdfplumber.
• Chunk size: 800 tokens (tiktoken cl100k_base), 100-token overlap.
• Chunks preserve the nearest preceding section heading as metadata.
• Chunk IDs are content-addressed: {company}-{fy}-{sha1[:8]}.

2.3 Embedding and indexing

• Embedding model: BAAI/bge-m3 via sentence-transformers, CPU inference.
• Vector store: ChromaDB, persistent, cosine similarity.
• Retrieval: top-k = 5, no re-ranker.

The retriever is held fixed across all models. This is deliberate: reported differences between models are differences in generator behavior given identical retrieved context. The joint quality of (retriever + generator) is out of scope.

3. Question Taxonomy

The 200-question set is asymmetric by design — we over-sample adversarial questions (Category 3) because that is where model behavior differentiates most and where production RAG systems fail most expensively.

Category	Count	Expected behavior
1 — Grounded-direct	45	Answer with citation, from a single chunk
2 — Grounded-multihop	45	Answer by synthesizing ≥2 chunks
3 — Adversarial	65	Answer correctly despite plausible distractors in retrieval
4 — Out-of-domain	45	Abstain; the answer is not in the corpus

3.1 Category 1 — Grounded-direct (single-chunk lookup)

The answer appears near-verbatim in one retrieved chunk. Baseline competence check.

Examples:

• "What was HDFC Bank's net interest income for FY24?"
• "As of FY24, who was the Chairperson of Bharti Airtel?"
• "What percentage of Adani Ports' FY24 cargo volume came from dry bulk?"

3.2 Category 2 — Grounded-multihop (multi-chunk synthesis)

Answer requires combining ≥2 chunks: arithmetic across years or segments, comparison, or stitching of adjacent facts. FY23+FY24 scope makes year-over-year questions feasible.

Examples:

• "By how many percentage points did Reliance's Jio ARPU change between FY23 and FY24?"
• "Across Zomato and Nykaa, which reported higher absolute advertising spend in FY24?"
• "What was the combined PAT of ITC's Agri Business and Paperboards segments in FY24?"

3.3 Category 3 — Adversarial (plausibly-in-domain)

Answerable from the corpus, but top-k retrieval reliably surfaces semantically similar but incorrect content: wrong fiscal year, wrong segment, wrong subsidiary, or superficially related disclosure. Gold metadata specifies which chunk is authoritative.

Examples:

• "What was Tata Motors' standalone (not consolidated) PAT in FY24?" — retrieval surfaces consolidated figures first.
• "What was Ola Electric's R&D spend in FY24?" — prospectus/DRHP-era numbers appear as distractors.
• "What interim dividend did ITC declare during FY24?" — final dividend announcements dominate retrieval.

3.4 Category 4 — Out-of-domain (unanswerable)

Plausible-sounding question whose answer is not in any chunk (and typically not in the corpus at all). Tests calibration under retrieval failure.

Examples:

• "What was the exit valuation of a divested Paytm Insurance subsidiary?" (not a disclosed event)
• "How much did Adani Ports spend on green hydrogen R&D in FY24?" (no such disclosure)
• "What is the current market share of Nykaa in Tier-3 Indian cities as of April 2026?" (outside corpus temporal scope)

3.5 Authoring protocol

Questions are authored by the project author and stored in data/questions/questions.jsonl. Each question includes: id, category, text, gold answer, gold source chunk IDs (empty for Category 4), gold citation quotes, expected behavior, company, fiscal year, difficulty, and optional private authoring notes (not shown to models or judges). validate.py enforces schema integrity: Category 4 questions must have empty gold_source_ids; Category 1–3 questions must reference chunk IDs that exist in the built index.

4. Scoring Rubric

Each response is scored on three 0/1/2 dimensions. Rationales are required for every score; evidence quotation is enforced in the judge prompt (§6).

4.1 Correctness

• 2 — Correct: All factual claims match the gold answer within tolerance. Numeric: ±1% relative error. Named entities: exact match. Prose: semantic equivalence.
• 1 — Partial: At least one claim correct; at least one incorrect or missing.
• 0 — Wrong: Primary claim false, or non-responsive.

For Category 4 (unanswerable) questions where the model correctly abstained, the judge sets correctness = 2 with rationale "N/A — correct abstention."

4.2 Grounding

• 2 — Fully grounded: Every substantive claim is directly supported by a retrieved chunk; citations point to chunks that actually contain the claim.
• 1 — Partially grounded: Majority of claims supported; one or more plausible-but-ungrounded claims present.
• 0 — Ungrounded: Core claim is fabricated, OR citations point to chunks that do not support the claim (citation theater).

Every unsupported claim is quoted verbatim into a structured unsupported_claims field for downstream error analysis.

4.3 Calibration

• 2 — Well-calibrated: Confident where warranted; hedged on uncertain sub-claims; clean abstention (with reason) when answer is not in context.
• 1 — Adequate: Minor over- or under-confidence.
• 0 — Miscalibrated: Confidently wrong on an unanswerable, OR over-refuses a clearly answerable question.

4.4 Composite: Grounding Fidelity Score

Primary formulation:

$\text{GFS}_{\text{primary}} = \frac{0.40 \cdot g + 0.35 \cdot c + 0.25 \cdot k}{2}$

where $c, g, k \in 0, 1, 2$ are correctness, grounding, and calibration respectively. Denominator 2 normalizes to [0, 1]. Grounding is weighted highest because it is the property the artifact centrally measures.

Robustness check:

$\text{GFS}_{\text{equal}} = \frac{c + g + k}{6}$

Both are reported. Model rankings are considered robust when the top-line ordering is preserved across both weightings.

5. Judging Protocol

5.1 Cross-family constraint

To prevent self-preference bias, a model never judges output from its own family. Two judges score every response:

Subject model	Judges
Claude Sonnet 4.6	GPT-5.4, Gemini 3.1 Pro
GPT-5.4	Claude Sonnet 4.6, Gemini 3.1 Pro
Gemini 3.1 Pro	Claude Sonnet 4.6, GPT-5.4

This constraint is enforced at call-site in src/eval/judge.py and covered by a unit test.

5.2 Aggregation

Final dimension scores are the mean of the two judge scores. No automated tiebreaker is used.

5.3 Contested responses

A response where the two judges disagree by more than 1 score-point on any dimension is flagged into a contested bucket for author hand-review. We expect 15–30 such cases across the full run. Hand-review outcomes are stored alongside the raw judge outputs; contested responses are not silently overridden.

5.4 Judge validation (Cohen's κ)

A 50-response human-labeled subset is used to validate judge reliability before the main run:

Stratum	Count
Category 1 — direct	10
Category 2 — multihop	10
Category 3 — adversarial	20
Category 4 — OOD	10

The adversarial stratum is over-weighted because judge reliability historically degrades most there. Samples are drawn proportionally across the three models.

For each judge model and each scoring dimension, we compute Cohen's κ against the author's human labels. Results are tabulated in this document (§5.5) and in WRITEUP.md after Milestone 5.

Pre-registered threshold: κ ≥ 0.75 on every (judge × dimension) cell. If any cell falls below, the rubric is revised and the 50-response set is re-scored until the threshold is met or the limitation is documented.

5.5 Judge reliability (populated after Milestone 5)

(Table reserved — populated once human labels are collected and judges run.)

Judge	κ (correctness)	κ (grounding)	κ (calibration)
Claude Sonnet 4.6	—	—	—
GPT-5.4	—	—	—
Gemini 3.1 Pro	—	—	—

6. Judge Prompt

The judge prompt is versioned alongside code and pinned per run. Current version: v1 (see src/eval/judge.py). Key design properties:

1. Evidence-before-score. Judges must quote a span from retrieved context (for grounding) or the gold answer (for correctness) before producing any score. This single constraint is the largest driver of inter-judge reliability in prior work.
2. Structured JSON output. No free-form prose outside the schema. Parsing failures are logged and retried once.
3. Separate unsupported_claims list. Quoted verbatim from the answer, enabling qualitative error-analysis without re-reading every response.
4. Expected-behavior in context. The judge is told whether the question is answerable, abstain-expected, or partial, so Category-4 correct-abstentions are not penalized as "non-answers."
5. Gold answer visible. Reference-based judging (not reference-free), standard in QA evaluation.

Full prompt text lives in src/eval/judge.py; the canonical version used for the published run is copied into configs/published.yaml for audit.

7. Experimental Protocol

7.1 Run configuration

• Temperature: 0 for both subjects and judges.
• Runs per (subject model, question): 3. All three providers exhibit minor non-determinism at T=0 (batching, load-balancing, floating-point accumulation); we report within-cell mean and standard deviation.
• Random seed: fixed per .env.
• Prompt caching: enabled on Anthropic calls for the system prompt and retrieved context. Cache metrics (read/write tokens) are logged per call.
• Batching: Anthropic Message Batches, OpenAI Batch API, Gemini Batch API where available; synchronous fallback otherwise.

7.2 Cost control

• Hard cap: $100 cumulative API spend. Warning at $50.
• Ledger: data/results/.budget_ledger.json, persistent across restarts.
• Pre-flight: every batch submission prints an itemized cost estimate and refuses to proceed if it would cross the cap.
• --dry-run mode prints estimates without calling any API.

7.3 Resumability

• Raw results are streamed to data/results/<model>_<run_id>.jsonl as individual responses complete.
• On restart, the runner reads existing output files and skips (question, model, run_index) tuples already present.
• The prompt used for each response is hashed (SHA-256) and stored with the result, so any prompt drift between runs is detectable.

8. Statistical Methodology

8.1 Point estimates

Per-model, per-category GFS is reported as the mean across questions and runs, with a bootstrap 95% confidence interval (10,000 resamples, BCa correction).

8.2 Pairwise model comparison

For each pair of models, McNemar's test is computed on binarized correctness (correct vs. not correct) at the (question, run-index) level. The full model matrix has $\binom{3}{2} = 3$ pairs; Bonferroni correction is applied.

8.3 Variance decomposition

Within-model, within-question standard deviation across the 3 runs is reported, separating model stochasticity from between-question variance. Models whose rankings are ambiguous once within-model noise is accounted for are flagged in the writeup.

8.4 Inter-judge agreement on main run

Beyond the 50-response validation set, we also report Cohen's κ between the two judges across the full 600-response main run (200 questions × 3 runs per subject, averaged per subject). Divergence from the validation-set κ is flagged.

9. Reproducibility

• Every prompt hashed (SHA-256) and stored with its result.
• ChromaDB index checksum pinned in the README.
• Question set frozen at version: 1 for the published run; later changes bump the version field.
• Raw data/results/*.jsonl are part of the published artifact.
• Single reproduction command: uv sync && rge eval run --config configs/published.yaml.

10. Ethics and Dual-Use

The corpus consists of publicly disclosed annual reports. No personal data is involved. The evaluation does not recommend financial actions; it measures model behavior on a factual QA task. Questions and gold answers are published under CC-BY-4.0 to enable reuse and independent verification.

11. Limitations

• Retriever held fixed. Reported behavior is specific to bge-m3 + k=5. Changing the retriever could shift absolute scores and potentially rankings.
• English only. Corpus and questions are English; no multilingual claims.
• LLM-as-judge residual bias. Cross-family judging mitigates but does not eliminate judge-family preferences. The 50-response human-kappa check is the primary defense.
• Parametric-memory leakage. Annual reports are public; models may recall figures without reading the context. We treat this as realistic (production RAG runs against public-ish corpora), but flag it: a true "unseen corpus" study would require private documents.
• Sample size. 200 questions × 3 runs × 3 models = 1,800 subject responses. Sufficient for the pairwise comparisons at hand; insufficient for fine-grained subgroup analysis (e.g., per-company).
• Single author labels. The 50-response human-labeled set is labeled by the project author. A second independent labeler would strengthen the κ validation; deferred to a follow-up.

12. References

(To be populated with prior-work citations — RAGAS, TruthfulQA, HaluEval, ARES, Ragnarök, etc. — during Milestone 7 writeup.)

---

Document version: 0.1 (Milestone 1 scaffold). Last updated: 2026-04-23.