Standalone Evaluation Pipeline

T3RL provides a standalone evaluation script (t3rl/test/eval_bfcl.py) that runs BFCL multi-turn evaluation without any slime dependency. This makes it easy to evaluate checkpoints on any machine with an SGLang server.

Overview

The eval pipeline reuses the same core components as training — BFCLEnv, BFCLGymAdapter, and parse_response — but replaces slime's InteractionDriver with a lightweight async loop that talks directly to SGLang's /generate HTTP endpoint via aiohttp.

eval_bfcl.py
  -> eval_single_entry()
       -> tokenizer.apply_chat_template()
       -> POST /generate (SGLang via aiohttp)
       -> parse_response()  [t3rl/response_parsing.py]
       -> BFCLGymAdapter(eval_mode=True).step()
       -> collect user_turn_scores
  -> print_report()  -- per-category accuracy table

Usage

1. Start an SGLang Server

python -m sglang.launch_server \
    --model-path /path/to/model \
    --port 30000 \
    --mem-fraction-static 0.85

2. Run Evaluation

python -m t3rl.test.eval_bfcl \
    --data-path data/processed/bfcl/bfcl_test.jsonl \
    --sglang-url http://localhost:30000 \
    --model-path /path/to/model \
    --config configs/bfcl/qwen3.yaml

Or use the shell wrapper:

SGLANG_URL=http://localhost:30000 bash scripts/test/run_t3rl_eval.sh

Key CLI Options

Option	Default	Description
--concurrency	32	Max concurrent requests to SGLang
--temperature	0.001	Sampling temperature (aligned with BFCL official)
--max-new-tokens	2048	Max tokens per generation step
--max-num-steps	30	Max total interaction steps per entry
--max-entries	None	Process only first N entries (for debugging)
--output	None	Save per-entry results to JSONL

Eval Mode vs Train Mode

BFCLGymAdapter accepts an eval_mode flag that changes behavior in two ways:

1. Step-limit handling: In eval mode, exceeding the step limit hard-terminates the episode (the turn is marked as failed). In train mode, the turn advances with a zero score so the agent can continue learning from subsequent turns.

2. Irrelevance scoring: In eval mode, turns where the agent's response is irrelevant (neither a valid tool call nor a meaningful answer) are excluded from scoring — this aligns with the official BFCL evaluation protocol. In train mode, these turns receive a zero score.

Accuracy Metric

The eval uses binary accuracy per entry:

• An entry passes (accuracy = 1.0) only if every scored user turn is correct
• If any turn fails, the entire entry is marked as failed (accuracy = 0.0)

This is stricter than the training reward (progress), which gives partial credit.

Async Design

The eval pipeline is fully async for high throughput:

• Uses asyncio.Semaphore to control concurrency (default 32 concurrent requests)
• aiohttp.TCPConnector with connection pooling
• asyncio.as_completed for streaming results with a live progress bar
• 300-second timeout per request to handle long multi-turn episodes

Output

Console Report

A per-category accuracy table is printed to stdout:

--------------------------------------------------------------
Category                         Correct    Total   Accuracy
--------------------------------------------------------------
gorilla_openfunctions_v1_...           8       10     80.00%
...
--------------------------------------------------------------
OVERALL                               85      100     85.00%
--------------------------------------------------------------

Status breakdown: {'completed': 95, 'force_quit': 3, 'error': 2}

Per-entry JSONL

With --output, each entry's result is saved as a JSONL line:

{
  "id": "multi_turn_base_0",
  "data_source": "gorilla_openfunctions_v1_test_multiple_function",
  "accuracy": 1.0,
  "user_turn_scores": [1.0, 1.0, 1.0],
  "status": "completed",
  "num_turns": 3
}