---

GAGE is a unified, extensible evaluation framework for large language models, multimodal models, audio models, diffusion models, agents, and game environments. It provides one evaluation engine for datasets, model backends, metrics, arena runtimes, structured outputs, and replayable artifacts.

Game Arena Showcase

Why GAGE?

• Fast evaluation engine: Run local smoke tests, model-backed jobs, and larger benchmark batches through the same pipeline shape.
• Unified evaluation surface: Datasets, backends, role adapters, metrics, and output contracts are configured instead of hand-wired per benchmark.
• Game and agent sandboxing: Game Arena, AgentKitV2, AppWorld, SWE-bench-style agent tasks, GUI interaction, and tool-augmented workflows share the same run/output model.
• External harness integration: Delegate task-batch benchmarks to Harbor, then import trial evidence back into standard GAGE samples, metrics, reports, and raw artifacts.
• Replayable GameKit runtime: Gomoku, Tic-Tac-Toe, Doudizhu, Mahjong, PettingZoo Space Invaders, Retro Mario, and ViZDoom now emit structured arena traces plus arena_visual sessions.
• Operational visibility: Runs write summary.json, sample outputs, logs, visual artifacts, and a static report pack so failures can be inspected after the fact.

Design Overview

Core design philosophy: everything is a step, everything is configurable.

Architecture Design

Orchestration Design

Game Arena Design

AgentKitV2 Design

External Harness Design

Quick Start

1. Installation

# If you are in the mono-repo root:
cd gage-eval-main

# Python 3.10+ recommended
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

For Game Arena LLM configs, use the *_openai_gamekit.yaml variants and export OPENAI_API_KEY. The model defaults to gpt-5.4; set GAGE_GAME_ARENA_LLM_MODEL to override it, or set OPENAI_API_BASE for an OpenAI-compatible endpoint.

2. Run a Basic Demo

python run.py \
  --config config/run_configs/demo_echo_run_1.yaml \
  --output-dir runs \
  --run-id demo_echo

3. View Reports

Default output structure:

runs/<run_id>/
  events.jsonl
  samples.jsonl
  summary.json
  samples/
    <namespace>/
      <sample_id>.json
  report_pack/
    report.html
    report_context.json
    report_context.md
    prompt.txt
    diagnostics.json
    assets_manifest.json

Open runs/<run_id>/report_pack/report.html for the execution-aware report: primary metrics, key findings, scenario profiles, evidence links, media previews, diagnostics, and reason-code explanations. See Run Reports.

Advanced Configurations

Scenario	Config Example	Description
GameArena Human-vs-AI	config/custom/doudizhu/doudizhu_human_visual_gamekit.yaml	Browser-controlled Doudizhu match against LLM players
GameArena Pure Human Control	config/custom/retro_mario/retro_mario_human_visual_gamekit.yaml	Browser-controlled real-time Retro Mario session
AgentKitV2 Tau2	config/custom/manual_e2e/agentkit_v2_tau2_local_lmstudio.yaml	Native per-sample local-process Tau2 1-case smoke run
AgentKitV2 SWE-bench Pro	config/custom/manual_e2e/agentkit_v2_swebench_pro_docker_lmstudio_smoke1_qutebrowser.yaml	Native Docker-backed SWE-bench Pro smoke run
External Harness Harbor	config/custom/external_harness_kits/harbor_terminal_bench2_lmstudio_1case.yaml	Delegates a Terminal-Bench 2.0 task to Harbor and imports results
AgentKitV2 AppWorld	config/custom/appworld/appworld_official_jsonl.yaml	AppWorld sandbox evaluation through the native AgentKitV2 path
Text	config/custom/aime24/aime2024_chat.yaml	AIME, GPQA, Math500, and related text benchmarks
Multimodal	config/custom/mathvista/chat.yaml	MathVista and related multimodal benchmarks
LLM Judge	config/custom/examples/single_task_local_judge_qwen.yaml	Local LLM judge example

Roadmap

• Agent evaluation: Continue hardening AgentKitV2 and External Harness trace import, failure diagnostics, and reproducible live smoke configs.
• Game Arena expansion: Grow the GameKit catalog and keep browser control, replay, and output contracts consistent.
• Gage-Client: Add a client tool for configuration management, failure diagnostics, and benchmark onboarding.
• Distributed inference: Support multi-node task sharding and load balancing for large runs.
• Benchmark expansion: Continue adding benchmark configs, metrics, and troubleshooting guidance.

Status

This project is in internal validation; APIs, configs, and docs may change rapidly.