Standardized Sample Spec (Schema & Alignment)

English | 中文

This document describes the standardized Sample structure used by gage-eval, so that dataset onboarding, preprocessing, inference, judging, metrics, and reporting can share a single contract.

1. Standardized Sample

1.1 Goals & principles

• OpenAI-style first: messages is the primary input for text, multimodal, and multi-turn tasks.
• Explicit task type: use task_type so templates/metrics can be selected automatically.
• Options vs. answers: keep candidates in options, keep ground-truth in references.
• Built-in few-shot: store reproducible few-shot in few_shot_examples.
• Trajectory alignment: store process references in golden_trajectories (for agent-style evaluation).
• Sandbox as top-level: describe execution environments via sandbox instead of burying it in metadata.
• Debias support: use unconditioned_input for PMI-like debiasing.
• Mappable to current code: preprocessors can map legacy fields (choices/metadata/label/...) into this schema.
• Extension-friendly: keep task tags and slicing labels in metadata / data_tag.
• Inputs are read-only: put large/raw objects in raw_assets; never directly render prompts from raw blobs.
• Per-sample eval controls: use eval_config for sample-level judge/metric overrides.
• Deprecate legacy prompt fields: do not keep model_prompt_tmpl/model_prompt_placeholder.

1.2 Structure overview

Lifecycle

flowchart TD
  RawRecord[RawRecord] --> Preprocess[Preprocess]
  Preprocess --> Sample[Sample]
  Sample --> Inference[Inference]
  Sample --> Judge[Judge]
  Inference --> PredictResult[PredictResult]
  Judge --> EvalResult[EvalResult]
  EvalResult --> Report[Report]

Top-level fields below do not include nested fields inside metadata.

Field	Type	Required	Notes
schema_version	string	yes	Schema version
id	string	yes	Unique sample id
task_type	string	no	Task/question type
messages	list	yes	OpenAI-style messages
options	list	no	Multiple-choice candidates
references	list	yes	Ground-truth answers
label	string	no	String alias of references[0]
few_shot_examples	list	no	In-sample few-shot items
golden_trajectories	list	no	Process reference trajectories
sandbox	object	no	Execution environment config
metadata	object	no	Task/runtime metadata
data_tag	object	no	Slicing labels for statistics
raw_assets	object	no	Raw/large assets store
tools	list	no	OpenAI tool definitions
tool_choice	string/object	no	Tool choice strategy
sampling_params	object	no	Sampling parameters
generation_params	object	no	Generation parameters
eval_config	object	no	Per-sample evaluation control
unconditioned_input	string/list	no	Debiasing input
predict_result	list	no	Runtime inference results
eval_result	object	no	Runtime evaluation results

1.3 Field details

1.3.1 messages and content

messages follows the OpenAI multimodal message format; content is a list of segments.

type	Field	Notes
text	text	Text
image_url	image_url.url	Image URL or local path
audio_url	audio_url.url	Audio URL or local path
video_url	video_url.url	Video URL or local path
file_url	file_url.url	Document URL or local path

Rules:

• messages is the primary entry. If the raw record only contains prompt/text/question, the preprocessor should generate messages.
• Multimodal resources may use relative paths; preprocessing is responsible for path resolution and encoding as needed.

1.3.2 task_type

task_type makes the task/question type explicit so prompt renderers and metrics can make better defaults.

Suggested values:

task_type	Use case
multiple-choice	Multiple-choice
short-answer	Short QA
dialogue	Multi-turn conversation
audio-translation	Audio translation
image-qa	Image QA
video-qa	Video understanding
doc-qa	Document understanding
text-to-image	Text-to-image
image-to-image	Image-to-image
text-to-audio	Text-to-audio
text-to-video	Text-to-video
code-generation	Code generation
agent	Agent tool-calling

Notes:

• If omitted, renderers may infer it from task config or use defaults.
• If legacy data uses question_type, preprocessors should map it to task_type.

1.3.3 options and references

options rules:

• options is a list; each item contains id and content.
• List order is the display order (helps reduce position bias).

references rules:

• references is a list and only contains final answers.
• Text tasks can use string arrays; multimodal tasks can use structured objects.
• label is a single-string alias, equivalent to references[0] (or a normalized string form of it).

Option item:

Field	Type	Required	Notes
id	string	yes	Option identifier
content	string	yes	Option text

Reference item:

Field	Type	Required	Notes
answer	string/list	yes	Answer content
meta	object	no	Answer metadata

Recommended answer formats:

• For text, prefer [{"type":"text","text":"..."}].
• A plain string shorthand is allowed and can be normalized during preprocessing.
• For multimodal generation, use image_url/audio_url/video_url/file_url segments.
• Keep label as string; for multimodal generation it can be a path/id.

1.3.4 few_shot_examples

• few_shot_examples stores per-sample few-shot to guarantee reproducibility.
• Each item is a “compact Sample”, typically only keeping messages/options/references/label/tools/tool_choice.
• Preprocessing can render references[0] (or label) into an assistant message and prepend them to the main messages.
• Nested few_shot_examples is not allowed.
• Do not include runtime/big fields like predict_result/eval_result/raw_assets/sandbox.

Few-shot item:

Field	Type	Required	Notes
messages	list	yes	Few-shot messages
options	list	no	Few-shot options
references	list	yes	Few-shot ground-truth
label	string	no	Alias of references[0]
tools	list	no	Tools definitions
tool_choice	string/object	no	Tool choice

Rendering sketch:

flowchart LR
  FewShotExamples[FewShotExamples] --> PromptRender[PromptRender]
  Messages[Messages] --> PromptRender
  PromptRender --> Inference[Inference]

1.3.5 golden_trajectories

• golden_trajectories is a list of reference trajectories (each is a full trace).
• Trajectory messages use the same structure as messages, and may include tool_calls and tool messages.
• Default evaluations often compare only references; process-level evaluation must enable dedicated metrics.

1.3.6 sandbox

sandbox describes sample-level execution environment preparation, suitable for code execution and agent tasks.

Field	Type	Notes
image	string	Container image
files	object	Files to mount/copy
setup	string	Init script
env	object	Env vars

Rules:

• Executors should read sandbox for environment setup.
• In files, keys are relative target paths; values are URIs or local paths.
• If an executor does not support it, preprocessors may map it into metadata.execution.

Preparation sketch:

flowchart TD
  Sample[Sample] --> Sandbox[SandboxPrepare]
  Sandbox --> Executor[Executor]
  Executor --> Run[RunStep]

1.3.7 metadata and data_tag

• metadata is for task/runtime metadata (source ids, split, difficulty, etc.).
• data_tag is for slicing labels used by statistics and reporting (domain/language/category, etc.).

1.3.8 tools and tool_choice

• tools follows the OpenAI tool schema.
• tool_choice controls the tool selection strategy.
• For agent tasks, preprocessors should keep tool definitions with the sample so that evaluations are reproducible.

1.3.9 raw_assets and “read-only additions”

• raw_assets stores raw/large objects (documents, images, audio, video, etc.).
• raw_assets is not used directly for prompt rendering; preprocessing must convert it into messages/media segments.

1.3.10 eval_config

eval_config enables sample-level overrides for judging and metrics (e.g., different judge settings, extra constraints).

In the current implementation, arena reads eval_config for per-sample game controls (such as retry_illegal and max_turns) in src/gage_eval/role/adapters/arena.py.

1.3.11 unconditioned_input

• unconditioned_input is used by debiasing methods such as PMI.
• It can be a string or a list, depending on the task and metric implementation.

1.4 Runtime result fields

At runtime, the framework writes results back to the Sample object:

• predict_result: inference outputs (often messages/text/tool calls depending on backend)
• eval_result: judge outputs and/or metric outputs

predict_result example

{
  "predict_result": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": [{"type": "text", "text": "B"}]
      },
      "raw_response": {},
      "usage": {"total_tokens": 123},
      "latency_ms": 1200
    }
  ]
}

eval_result example

{
  "eval_result": {
    "overall": {"score": 1.0, "passed": true},
    "metrics": {
      "exact_match": {"score": 1.0},
      "bleu": {"score": 0.72}
    },
    "judge": {
      "model": "gpt-4o-mini",
      "verdict": "correct",
      "reason": "Answer matches the reference"
    }
  }
}

1.5 Mapping to the current implementation

The standardized Sample is designed to be compatible with the current gage-eval implementation through preprocessing mappings.

Typical mappings:

• legacy question/prompt/text -> messages via normalize_messages
• legacy choices/options -> normalized choices + metadata.option_map via normalize_options/map_question_option_answer
• legacy answer/label -> label and metadata.correct_choice (optional references)
• multimodal legacy fields -> messages[*].content[*] segments + inputs.multi_modal_data

Code excerpt: writing predict_result

def append_predict_result(sample: Dict[str, Any], model_output: Optional[Dict[str, Any]]) -> None:
    # Skip empty outputs
    if not isinstance(model_output, dict) or not model_output:
        return
    predict_result = sample.setdefault("predict_result", [])
    if not isinstance(predict_result, list):
        predict_result = sample["predict_result"] = []
    entry = copy.deepcopy(model_output)
    entry.setdefault("index", len(predict_result))
    if "message" not in entry:
        entry["message"] = _build_message(entry)
    predict_result.append(entry)

1.6 Example cases

Below are example Sample JSON snippets for common task types.

1.6.1 Text QA

{
  "schema_version": "v1",
  "id": "qa_0001",
  "task_type": "short-answer",
  "messages": [
    {
      "role": "user",
      "content": [{"type": "text", "text": "What is the capital of France?"}]
    }
  ],
  "references": [{"answer": [{"type": "text", "text": "Paris"}]}],
  "label": "Paris",
  "metadata": {"dataset": "demo"}
}

1.6.2 Multiple-choice

{
  "schema_version": "v1",
  "id": "mc_0001",
  "task_type": "multiple-choice",
  "messages": [
    {
      "role": "user",
      "content": [{"type": "text", "text": "Which one is a mammal?"}]
    }
  ],
  "options": [
    {"id": "A", "content": "Shark"},
    {"id": "B", "content": "Dolphin"}
  ],
  "references": [{"answer": [{"type": "text", "text": "B"}]}],
  "label": "B",
  "metadata": {"option_map": {"A": "Shark", "B": "Dolphin"}}
}

1.6.3 Multi-turn dialogue

{
  "schema_version": "v1",
  "id": "dlg_0001",
  "task_type": "dialogue",
  "messages": [
    {"role": "user", "content": [{"type": "text", "text": "Explain recursion."}]},
    {"role": "assistant", "content": [{"type": "text", "text": "Recursion is ..."}]},
    {"role": "user", "content": [{"type": "text", "text": "Give an example."}]}
  ],
  "references": [{"answer": [{"type": "text", "text": "..." }]}]
}

1.6.4 Image QA

{
  "schema_version": "v1",
  "id": "imgqa_0001",
  "task_type": "image-qa",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "image_url", "image_url": {"url": "inputs/apple_0001.png"}},
        {"type": "text", "text": "What fruit is shown in the image?"}
      ]
    }
  ],
  "references": [{"answer": [{"type": "text", "text": "apple"}]}],
  "label": "apple"
}

1.6.5 Audio translation

{
  "schema_version": "v1",
  "id": "aud_0001",
  "task_type": "audio-translation",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "audio_url", "audio_url": {"url": "inputs/hello.wav"}},
        {"type": "text", "text": "Translate the audio into English."}
      ]
    }
  ],
  "references": [{"answer": [{"type": "text", "text": "Hello"}]}],
  "label": "Hello"
}

1.6.6 Video understanding

{
  "schema_version": "v1",
  "id": "vid_0001",
  "task_type": "video-qa",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "video_url", "video_url": {"url": "inputs/demo.mp4"}},
        {"type": "text", "text": "Describe what happens in the video."}
      ]
    }
  ],
  "references": [{"answer": [{"type": "text", "text": "..."}]}]
}

1.6.7 Document understanding

{
  "schema_version": "v1",
  "id": "doc_0001",
  "task_type": "doc-qa",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "file_url", "file_url": {"url": "inputs/paper.pdf"}},
        {"type": "text", "text": "Summarize section 2."}
      ]
    }
  ],
  "references": [{"answer": [{"type": "text", "text": "..."}]}]
}

1.6.8 Code generation & execution

{
  "schema_version": "v1",
  "id": "code_0001",
  "task_type": "code-generation",
  "messages": [
    {
      "role": "user",
      "content": [{"type": "text", "text": "Write Python code to compute Fibonacci numbers."}]
    }
  ],
  "sandbox": {
    "image": "python:3.10",
    "setup": "python -V"
  },
  "references": [{"answer": [{"type": "text", "text": "..."}]}]
}

1.6.9 Text-to-image

{
  "schema_version": "v1",
  "id": "t2i_0001",
  "task_type": "text-to-image",
  "messages": [
    {
      "role": "user",
      "content": [{"type": "text", "text": "Generate an image of an apple on a table."}]
    }
  ],
  "references": [
    {"answer": [{"type": "image_url", "image_url": {"url": "refs/apple_0001.png"}}]}
  ],
  "label": "refs/apple_0001.png"
}

1.6.10 Image-to-image

{
  "schema_version": "v1",
  "id": "i2i_0001",
  "task_type": "image-to-image",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "image_url", "image_url": {"url": "inputs/sketch_0001.png"}},
        {"type": "text", "text": "Colorize the sketch."}
      ]
    }
  ],
  "references": [
    {"answer": [{"type": "image_url", "image_url": {"url": "refs/color_0001.png"}}]}
  ],
  "label": "refs/color_0001.png"
}

1.6.11 Text-to-audio

{
  "schema_version": "v1",
  "id": "t2a_0001",
  "task_type": "text-to-audio",
  "messages": [
    {
      "role": "user",
      "content": [{"type": "text", "text": "Generate an audio clip of ocean waves."}]
    }
  ],
  "references": [{"answer": [{"type": "audio_url", "audio_url": {"url": "refs/ocean_0001.wav"}}]}],
  "label": "refs/ocean_0001.wav"
}

1.6.12 Text-to-video

{
  "schema_version": "v1",
  "id": "t2v_0001",
  "task_type": "text-to-video",
  "messages": [
    {
      "role": "user",
      "content": [{"type": "text", "text": "Generate a short video of a running dog."}]
    }
  ],
  "references": [{"answer": [{"type": "video_url", "video_url": {"url": "refs/dog_0001.mp4"}}]}],
  "label": "refs/dog_0001.mp4"
}

1.6.13 Agent tool calling & golden trajectories

{
  "schema_version": "v1",
  "id": "agent_0001",
  "task_type": "agent",
  "messages": [
    {"role": "user", "content": [{"type": "text", "text": "Search the weather and summarize."}]}
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }
  ],
  "tool_choice": "auto",
  "golden_trajectories": [
    [
      {
        "role": "assistant",
        "tool_calls": [
          {"id": "call_1", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\":\"Paris\"}"}}
        ]
      },
      {"role": "tool", "tool_call_id": "call_1", "content": "Sunny, 22C"},
      {"role": "assistant", "content": [{"type": "text", "text": "Paris is sunny at 22C."}]}
    ]
  ],
  "references": [{"answer": [{"type": "text", "text": "Paris is sunny at 22C."}]}]
}

Agent execution flow

sequenceDiagram
  participant User
  participant Model
  participant Tool
  User->>Model: messages and tools
  Model->>Tool: tool_call
  Tool-->>Model: tool_result
  Model-->>User: final_answer

2. Usage in gage-eval

2.1 Producing standardized Samples

• Preprocessors can emit a dict or a Sample dataclass. DataManager will validate and normalize before yielding dicts to the runtime.
• The default envelope validator requires schema_version, id, and messages (src/gage_eval/assets/datasets/validation.py).
• Use normalize_sample and merge_multimodal_inputs to standardize messages and media fields.

Minimal validation config example:

datasets:
  - dataset_id: demo
    loader: jsonl
    params:
      path: /path/to/data.jsonl
    schema:
      mode: warn

2.2 Writing runtime outputs

• Inference and arena steps append outputs via append_predict_result (src/gage_eval/evaluation/sample_envelope.py).
• Judge outputs are merged via update_eval_result, and downstream metrics resolve them through resolve_model_output/resolve_judge_output.

2.3 Multiple-choice compatibility

• Current runtime utilities normalize choices; if you maintain options, map them into choices and metadata.option_map during preprocessing.
• label and metadata.correct_choice are commonly used by multiple-choice metrics and judge adapters.