Agent Learning Track

Use the Agent Learning Track to evaluate methods that learn reusable agent improvements from previous task trajectories, including agentic memory, skills, and prompt optimization.

If you want to directly evaluate an agent on all tasks without an agent learning method, use the Main Track instead.

What This Track Measures

The Agent Learning Track uses the same simulator, domain tools, judges, and metrics as the Main Track. It adds:

• 100 task trajectories per domain generated by the protocol-locked agent,
• 50 held-out test tasks per domain,
• an inference-time retrieval hook for reusable learnings.

The score asks whether learnings extracted from past trajectories improve agent performance.

Learning extraction is user-owned. It can be a memory store, skills library, prompt optimization artifact, vector index, or another local artifact, as long as your agent exposes relevant entries through the retrieve_learnings hook during evaluation. The retrieval hook is defined in Expose Retrieval.

Train And Test Data

Domain	Train trajectories	Test tasks
Travel	100	50
Customer Support	100	50
Shopping Assistant	100	50

Train trajectories live at:

../datasets/train_task_trajectories/<domain>/<task_id>.json

Use these trajectories to generate your reusable learnings. Test task definitions and task environments are used by the benchmark at evaluation time and must not be used as oracle inputs for learning extraction.

1. Install

STATE-Bench supports Python 3.12+ and uses uv. From a fresh checkout, sync dependencies and create your local environment file:

uv sync
cp .env.example .env

STATE-Bench makes LLM calls in three places:

• the locked user simulator,
• the locked judge,
• the agent under test.

The simulator and judge are fixed by the benchmark protocol. Your agent model and learning method are configurable.

2. Configure Clients

Locked simulator and judge

Every official run requires the protocol-locked GPT-5.4 evaluation client. Configure it first:

• Locked Evaluation Client

Agent under test

If your learning-enabled agent uses an Azure AI Foundry model or OpenAI model through the built-in standard tool-calling StateBenchAgent, configure the built-in agent client:

• Built-in StateBenchAgent Client (no code change needed; only configure env variables)

If your learning method evaluates models from a different provider or uses a custom tool-calling agent, follow the instructions to extend the base classes:

• Custom Client + Agent

After you build the custom harness, come back to this guide for the retrieval hook and run command.

3. Build Learnings

Build your learnings artifact from the train trajectories. STATE-Bench does not prescribe the artifact format and does not score the artifact directly. It scores the held-out test trajectories produced when your agent uses that artifact.

4. Expose Retrieval

Built-in StateBenchAgent

Subclass StateBenchAgent and implement retrieve_learnings(query, top_k=3) -> list[str] under the repo-root agents/ folder:

# agents/my_memory_agent.py
import json
from pathlib import Path

from state_bench.agents.state_bench import StateBenchAgent


class MyMemoryAgent(StateBenchAgent):
    learnings_path = Path("<path_to_learnings.json>")

    def retrieve_learnings(self, query: str, top_k: int = 3) -> list[str]:
        learnings = json.loads(self.learnings_path.read_text())
        return learnings[:top_k]

When the subclass defines retrieve_learnings, StateBenchAgent adds the retrieval tool, routes calls to your method, forces the benchmark-configured top_k, and validates that the result is list[str].

Details: Memory Hook: Built-in StateBenchAgent.

Custom BaseAgent

If you are using a custom client + agent, first implement the custom harness in Custom Client + Agent, then expose retrieval from your BaseAgent using the custom memory hook:

• Memory Hook: Custom BaseAgent

Memory retrieval tools must be read-only. Domain tools are still owned and executed by STATE-Bench.

5. Run Tasks

Run one domain at a time:

uv run python -m state_bench.scripts.run_batch \
  --domain <domain> \
  --agent-class <YourMemoryAgent> \
  --agent-model-name <model-name> \
  --num-runs 5 \
  --retrieve-learnings-top-k 3 \
  --num-workers <parallel-workers> \
  --output-dir outputs/<domain>/

If your agent model uses a reportable reasoning level, add:

  --agent-model-reasoning-level <reasoning-level>

For a custom client + agent, also add:

  --agent-client-class <YourClient>

Use these values:

Argument	Value
--domain	Required. travel, customer_support, or shopping_assistant
--agent-class	Your learning-enabled agent class under repo-root agents/
--agent-client-class	Required only for a custom client + agent
--agent-model-name	The model name to report in trajectories and metrics, such as gpt-5.1 or claude-sonnet-4.5
--agent-model-reasoning-level	Optional reasoning level, such as low, medium, or high; omit if not applicable
--num-runs	5 for official submissions
--retrieve-learnings-top-k	3 for official submissions
--num-workers	Parallel task workers; tune for your provider rate limits
--output-dir	outputs/<domain>/ for the standard layout

run_batch writes scored trajectories under:

outputs/<domain>/run1/<task_id>.json
outputs/<domain>/run2/<task_id>.json
...
outputs/<domain>/run5/<task_id>.json

For the full CLI reference and worker guidance, see run_batch.

6. Report Cost Per Task

Cost reporting is optional but strongly encouraged. Custom agents should compute provider-specific cost themselves and call self.add_cost_usd(...); STATE-Bench aggregates the reported dollars without interpreting token buckets. Embedding or offline artifact-building costs are not included in the official public metrics unless your agent reports them during the benchmark run. Details: Reporting Avg. Cost Per Task.

7. Compute Metrics

After a domain finishes, produce its standardized metrics file:

uv run python -m state_bench.scripts.compute_metrics \
  --domain <domain> \
  --results-dir outputs/<domain>/ \
  --num-runs 5 \
  --output-dir outputs/<domain>/

Metrics default to the protocol test split and fail if any expected test task is missing or unscored. Details: Compute Metrics.

Repeat the run and metrics steps for travel, customer_support, and shopping_assistant for a complete submission.

8. Submit

Package the scored trajectories and metrics for each completed domain, then open a submission issue. Details: Submit Results.

Official Run Settings

For protocol-compliant Agent Learning Track submissions:

• use only datasets/train_task_trajectories/ for offline learning extraction,
• do not use held-out test task definitions or test environments as oracle inputs for learnings,
• use the locked GPT-5.4 simulator and judge client,
• do not edit simulator prompts, judge prompts, domain tools, task files, environment files, or protocol files,
• run --num-runs 5,
• run --retrieve-learnings-top-k 3,
• ensure retrieve_learnings returns list[str],
• keep learning retrieval read-only,
• include metrics.json and scored trajectories for each submitted domain.