EnterpriseRAG-Bench Submission

September 7, 2026 · View on GitHub

This note records the RE-call EnterpriseRAG-Bench run prepared for public leaderboard review.

Public Artifacts

The repo intentionally does not include EnterpriseRAG questions, gold answers, document text, or generated evaluator document caches. Those come from the official EnterpriseRAG-Bench release.

Committed artifacts:

File	Purpose
`benchmarks/artifacts/enterprise_rag/re_call_voyage_splade_gpt4o.answers.jsonl`	500 answer rows for official evaluation.
`benchmarks/artifacts/enterprise_rag/re_call_voyage_splade_gpt4o.answers.manifest.json`	RE-call answer run provenance.
`benchmarks/artifacts/enterprise_rag/re_call_voyage_splade_gpt4o.judge_gpt54_mixed_default.summary.json`	Sanitized local score summary, official default evaluator, disclosed mixed GPT 5.4 judge settings.
`benchmarks/artifacts/enterprise_rag/re_call_voyage_splade_gpt4o.judge_gpt54_medium.no_correction.summary.json`	Sanitized local score summary, official evaluator, GPT 5.4 medium reasoning, no correction.
`benchmarks/artifacts/enterprise_rag/re_call_voyage_splade_gpt4o.no_correction.summary.json`	Sanitized local comparison score summary, GPT 4o judge, no correction.

Answer Configuration

The submitted answer file was generated by RE-call with:

Component	Value
Embeddings	`voyage:voyage-4-large`
Sparse retrieval	Postgres lexical plus SPLADE
SPLADE model	`prithivida/Splade_PP_en_v1`
Reranker	`voyage:rerank-2.5`
Generator	`openai/gpt-4o` through OpenRouter
`candidate_k`	`200`
`k`	`8`
`gap_threshold`	`0.5`
`max_context_chars`	`12000`

The answer manifest records:

Field	Value
RE-call revision	`858e1af6870a93aadec859a6d71b6ec807fcaf72`
Questions SHA256	`f9524b9157cd43aae36b99333a124738804306ea6d07f332d49faa6d3d147905`
Documents SHA256	`9d1174928696ad08bc15f3f104739519de633c1605a4ec2034e0e3c0087bc5cd`
Answer file SHA256	`05d01db6ee9350aaf9093b7bcac63fbbcdbfc4e7af3f2608b67cd8c8065c35ac`

Local Scores

The leaderboard ranks by average correctness.

Run	Judge	Evaluator options	Correctness	Completeness	Overall score	Recall
Default local score	`openai/gpt-5.4`, mixed reasoning disclosed	default correction and citation stripping	`65.60`	`53.48`	`48.03`	`77.48`
Official shaped local score	`openai/gpt-5.4`, medium reasoning	`--no-correction --skip-citation-stripping`	`63.80`	`53.23`	`46.16`	`77.34`
Comparison score	`openai/gpt-4o`	`--no-correction --skip-citation-stripping`	`63.20`	`63.79`	`54.85`	`77.34`

The default local score must be disclosed as mixed judge mode. Rows qst_0001 through qst_0214 used openai/gpt-5.4 medium reasoning through OpenRouter. Rows qst_0215 through qst_0500 used openai/gpt-5.4 through OpenRouter with reasoning disabled. The public answer file can be rerun by Onyx with any preferred homogeneous judge configuration.

Reproduction

Clone the official benchmark:

git clone --depth 1 https://github.com/onyx-dot-app/EnterpriseRAG-Bench.git
cd EnterpriseRAG-Bench
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Download the official questions.jsonl and all_documents.zip release artifacts. The hashes must match the manifest above.
Copy the RE-call answer file:

mkdir -p answer_evaluation
cp /path/to/re_call_voyage_splade_gpt4o.answers.jsonl \
  answer_evaluation/re_call_voyage_splade_gpt4o.answers.jsonl

Build the minimal evaluator document cache:

python /path/to/RE-call/scripts/enterprise_rag_build_eval_cache.py \
  --questions questions.jsonl \
  --answers answer_evaluation/re_call_voyage_splade_gpt4o.answers.jsonl \
  --documents-zip /path/to/all_documents.zip \
  --sources-out generated_data/sources \
  --index-out generated_data/uuid_index.json

If using OpenRouter for the official judge model, apply an OpenAI compatible endpoint patch to src/llm/openai_llm.py that sends chat completions to the OpenRouter base URL and passes reasoning controls through extra_body. For medium reasoning:

extra_body={"reasoning": {"effort": "medium"}}

For the budget controlled run, omit reasoning or set it to None. The RE-call helper script below does this with ENTERPRISE_RAG_JUDGE_REASONING=off.

Run the official evaluator:

ENTERPRISE_RAG_BENCH_DIR=/path/to/EnterpriseRAG-Bench \
RECALL_DIR=/path/to/RE-call \
ENTERPRISE_RAG_ANSWERS_FILE=/path/to/EnterpriseRAG-Bench/answer_evaluation/re_call_voyage_splade_gpt4o.answers.jsonl \
ENTERPRISE_RAG_RESULTS_FILE=/path/to/EnterpriseRAG-Bench/answer_evaluation/re_call_voyage_splade_gpt4o.judge_gpt54_medium.default_results.json \
ENTERPRISE_RAG_UPDATED_QUESTIONS_FILE=/path/to/EnterpriseRAG-Bench/answer_evaluation/re_call_voyage_splade_gpt4o.judge_gpt54_medium.default_questions_updated.jsonl \
ENTERPRISE_RAG_UUID_INDEX_FILE=/path/to/EnterpriseRAG-Bench/generated_data/uuid_index.json \
ENTERPRISE_RAG_SCORE_PARALLELISM=4 \
ENTERPRISE_RAG_JUDGE_MODEL=openai/gpt-5.4 \
scripts/enterprise_rag_score_openrouter.sh

For the declared budget controlled run:

ENTERPRISE_RAG_BENCH_DIR=/path/to/EnterpriseRAG-Bench \
RECALL_DIR=/path/to/RE-call \
ENTERPRISE_RAG_ANSWERS_FILE=/path/to/EnterpriseRAG-Bench/answer_evaluation/re_call_voyage_splade_gpt4o.answers.jsonl \
ENTERPRISE_RAG_RESULTS_FILE=/path/to/EnterpriseRAG-Bench/answer_evaluation/re_call_voyage_splade_gpt4o.judge_gpt54_reasoning_off.default_results.json \
ENTERPRISE_RAG_UPDATED_QUESTIONS_FILE=/path/to/EnterpriseRAG-Bench/answer_evaluation/re_call_voyage_splade_gpt4o.judge_gpt54_reasoning_off.default_questions_updated.jsonl \
ENTERPRISE_RAG_UUID_INDEX_FILE=/path/to/EnterpriseRAG-Bench/generated_data/uuid_index.json \
ENTERPRISE_RAG_SCORE_PARALLELISM=1 \
ENTERPRISE_RAG_JUDGE_MODEL=openai/gpt-5.4 \
ENTERPRISE_RAG_JUDGE_REASONING=off \
scripts/enterprise_rag_score_openrouter.sh

Submission boundary

The public repository carries the reproducibility guide and answer artifact. Contact details, submission drafts, and maintainer correspondence are intentionally kept out of the repository.