BFCL Multi-Turn Reference

Reference documentation for the BFCL multi-turn evaluation protocol, based on 3rdparty/gorilla/berkeley-function-call-leaderboard/bfcl_eval/ source code. Covers dataset structure, scoring semantics, execution engine internals, and known edge cases.

Version note: Multi-turn evaluation was introduced in BFCL V3. The current codebase uses BFCL_v4 file prefixes (V4 added agentic/memory/web_search categories on top of V3, but the multi-turn evaluation logic is unchanged).

---

Architecture Overview

bfcl generate (inference phase)
  -> load_dataset_entry(category)         # load BFCL_v4_*.json
  -> build_handler(model_name)            # select handler via MODEL_CONFIG_MAPPING
  -> handler.inference(test_case)         # multi-turn inference loop
      -> _pre_query_processing_FC/prompting()
      -> for each turn:
          for each step (max 20):
            model API call -> decode response -> execute calls -> add results to history
      -> write result JSONL

bfcl evaluate (evaluation phase)
  -> load result + ground_truth
  -> multi_turn_runner()
      -> decode model output -> execute both model & GT calls
      -> state_checker() + response_checker() per turn
      -> binary pass/fail per entry
  -> generate_leaderboard_csv()

---

Dataset Structure

Dataset entry (BFCL_v4_multi_turn_*.json):

{
  "id": "multi_turn_base_42",
  "question": [                           // outer = turns, inner = messages
    [{"role": "user", "content": "..."}],
    [{"role": "user", "content": "..."}]
  ],
  "function": [{...tool definitions...}],
  "initial_config": {                     // initializes simulated environment state
    "GorillaFileSystem": {"root": {...}},
    "TwitterAPI": {"tweets": {...}, ...}
  },
  "involved_classes": ["GorillaFileSystem", "TwitterAPI"],
  "missed_function": {"3": ["sort"]},     // miss_func variant: hide sort() at turn 3
  "excluded_function": [...]              // excluded from available tools
}

Ground truth (possible_answer/BFCL_v4_multi_turn_*.json):

{
  "id": "multi_turn_base_42",
  "ground_truth": [
    ["cd(folder='document')", "mkdir(dir_name='temp')"],   // turn 0
    ["grep(file_name='report.pdf', pattern='budget')"],    // turn 1
    [],                                                     // turn 2 (miss_func: should not call)
    ["diff(file_name1='a.pdf', file_name2='b.pdf')"]       // turn 3
  ]
}

---

Scoring Logic

Core function: multi_turn_checker() in eval_checker/multi_turn_eval/multi_turn_checker.py

Each turn performs two checks:

1. state_checker — Compares all public attributes of the model instance and GT instance using Python == exact matching
2. response_checker — Verifies that GT execution results are an unordered sub-multiset of the model's accumulated execution results

Key design decisions:

• Accumulated responses: response_checker compares against all accumulated results across turns, not just the current turn. If the model correctly executes a turn-N+1 GT call early at turn N, it still passes.
• Order-independent: method_invoke_order_checker() is implemented but commented out — function call order does not affect scoring.
• Binary scoring: Each test entry is pass/fail only, no partial credit.
• Empty GT turns: Turns where GT is [] (miss_func/miss_param) skip checking but still execute to maintain state synchronization.

Scoring formula:

• Category accuracy = correct_entries / total_entries
• Multi-Turn Overall = arithmetic mean of the four subcategories (unweighted)
• Global Overall = weighted: [non_live: 10%, live: 10%, irrelevance: 10%, multi_turn: 30%, agentic: 40%]

---

Execution Engine

execute_multi_turn_func_call() in multi_turn_utils.py:

1. Dynamically imports and instantiates API classes based on involved_classes
2. Stores instances in Python globals() (key: {model_name}_eval_{entry_id}_{class_name}_instance)
3. Calls instance._load_scenario(initial_config[class_name]) to initialize state
4. Regex-transforms bare function calls get_info("x") into instance.get_info("x")
5. Executes via Python eval()
6. Result serialization: dict -> json.dumps, others -> str()

Note: The globals() storage makes BFCL evaluation not thread-safe. T3RL's BFCLEnv fixes this by using instance-level dicts.

---

API Backends

Eight simulated API backends power the multi-turn environment:

Class	Stateful	Auth Required	Core State
GorillaFileSystem	Yes	No	Virtual file directory tree
MathAPI	No (only stateless)	No	None
MessageAPI	Yes	Yes (login)	inbox, user_map
TwitterAPI	Yes	Yes (authenticate)	tweets, following_list
TicketAPI	Yes	Yes (login)	ticket_queue
TradingBot	Yes	Yes (login)	orders, portfolio
TravelAPI	Yes	Yes (authenticate)	bookings, credit_cards
VehicleControlAPI	Yes	No	engine/door/climate state

Deterministic randomness: APIs that generate random IDs (Message, Trading, Travel) use initial_config["random_seed"] to initialize random.Random, ensuring model and GT produce identical IDs.

---

Subcategories

Category	Description
multi_turn_base	Standard multi-turn; all tools available
multi_turn_miss_func	Some functions hidden at certain turns; model should recognize and not call them
multi_turn_miss_param	Some parameters missing; model should recognize and not call the function
multi_turn_long_context	Long-context variant (API responses injected with hundreds of distractor items: stock tickers, flight routes, etc.)

---

Known Edge Cases

1. multi_turn_irrelevance_checker() is never called — The function exists but is not wired into the evaluation pipeline. A model incorrectly calling functions when GT is empty is not penalized (only the reverse — model empty when GT is non-empty — is checked).
2. method_invoke_order_checker() is commented out — Call order does not affect scoring.
3. is_evaL_run typo — The capital L in the parameter name is a historical typo in the BFCL codebase.
4. VehicleControlAPI _load_scenario uses camelCase — Scenario keys use "engineState" rather than "engine_state".
5. TravelAPI get_flight_cost pricing — Price multiplier is determined by date digit sum: sum(digits) % 2 == 0 -> 2x, otherwise 1x.
6. Long-context injection — Hundreds of fake data items (stock tickers, flight routes) are injected into API responses to distract the model.
7. Accumulated response comparison — Functions the model correctly calls early are still counted as correct in later turns.