gage-eval Benchmark Guide

June 2, 2026 · View on GitHub

This document provides execution commands and detailed descriptions for benchmark configs that exist in the current gage-eval-main/ repository.

Common Evaluation Chain

flowchart LR
  Raw[Dataset] --> Pre[Preprocess] --> Sup[Support] --> Inf[Inference] --> Jdg[Judge] --> AE[AutoEval] --> Rep[Report]

Usage Overview

Run commands assume you are in the gage-eval-main/ repository root and use the core entry point run.py.

--config: Path to the YAML configuration defining the model, prompts, and dataset parameters.
--output-dir: Where logs and results are stored.
--run-id: Unique identifier for the specific evaluation run.

Supported Backend Engines

The framework supports multiple inference backends to ensure flexibility across different environments. Currently supported engines include, but are not limited to:

vLLM: Optimized for high-throughput serving of local models.
LiteLLM: A unified interface for calling various external LLM APIs (OpenAI, Anthropic, etc.).

To switch between backends, modify the backend_id parameter within the role_adapters section of your specific benchmark YAML file:

role_adapters:
  - adapter_id: dut_mmlu_pro_vllm_qwen
    role_type: dut_model
    backend_id: "vllm"  # Options: vllm, litellm, etc.

If you need a full evaluation, adjust max_samples/limit.

BizFinBench.v2 is the secend release of BizFinBench. It is built entirely on real-world user queries from Chinese and U.S. equity markets. It bridges the gap between academic evaluation and actual financial operations.

Authentic & Real-Time: 100% derived from real financial platform queries, integrating online assessment capabilities. Expert-Level Difficulty: A challenging dataset of 29,578 Q&A pairs requiring professional financial reasoning. Comprehensive Coverage: Spans 4 core business scenarios, 8 fundamental tasks, and 2 online tasks.

Parameter	Description	Supported Values
`needle_type`	Defines the complexity of the "Needle-in-a-Haystack" test by specifying how many distinct pieces of information (needles) are hidden within the context window.	`2needle`, `4needle`, `8needle`
`max_content_window`	Specifies the maximum token length of the "haystack" (the input text) used during evaluation. This defines the limit of the model's context capacity to be tested.	Integer (e.g., 32768, 128000)

Parameter	Description	Supported Values
`scenario`	Defines the specific evaluation task.	`codegeneration`, `codeexecution`, `testoutputprediction`
`release_version`	Specifies the dataset version to track performance over time.	`release_v1` through `release_v6`
`local_dir`	The directory path where the downloaded dataset is cached.	Local file path

Parameter	Description	Supported Values
`use_caption`	Determines whether to provide textual descriptions of the images to the model.	`true`, `false`
`use_ocr`	Enables or disables the inclusion of extracted Optical Character Recognition (OCR) text from the figures.	`true`, `false`
`shot_num`	The number of few-shot examples included in the prompt context.	Integer (e.g., 0, 3, 5)
`shot_type`	Defines the reasoning format for few-shot examples.	`'solution'` (Natural Language), `'code'` (Program-of-Thought)

Parameter	Description	Supported Values
`pre_encode_video`	Whether to fetch the video URL and encode it as a base64 data URL before sending to the backend.	`true`, `false`
`include_subtitles`	Whether to include subtitle text in the prompt.	`true`, `false`

Parameter	Description	Supported Values
`path`	Path to the local MMAU-Pro JSONL file (schema aligned with `gamma-lab-umd/MMAU-Pro`).	Valid JSONL file path
`audio_path_root`	Root directory where the referenced audio files (e.g. `data/*.wav`) are stored.	Valid directory path
`audio_index`	Index of the audio segment to be used when multiple audio paths are provided per sample.	Non-negative integer (default `0`)
`model_path` / `tokenizer_path`	Local path to the audio-capable model and tokenizer (e.g. Qwen2-Audio family) used by the vLLM backend.	Model directory containing `config.json` and tokenizer files

Parameter	Description	Supported Values
`question_set_path`	ForecastBench question set JSON.	`FORECASTBENCH_QUESTION_SET_PATH` or a valid file path
`resolution_set_path`	Matching ForecastBench resolution set JSON.	`FORECASTBENCH_RESOLUTION_SET_PATH` or a valid file path
`source_filter`	Source subset included in the run.	Default: `polymarket`
`resolved_only`	Keep only resolved rows for static scoring.	Default: `true`
`include_market_baseline_in_prompt`	Include the freeze-time market baseline in the prompt.	Default: `true`
model output	Probability parsed from `0.42`, JSON, or a plain number.	Clamped to `[0, 1]`; parse failures count as `parse_error`