SearchMaster: Grounded and Regulated Self-Play for Search Agents

Wentao Tan, Qiong Cao, Jiaqi Wang, and Nan Duan

This repository hosts the official implementation and artifacts for SearchMaster, a self-play framework for training LLM-based search agents from tasks they generate, solve, and verify in a local search environment.

Overview

Search agents need training data that contains both challenging multi-hop search tasks and effective tool-use trajectories. Existing pipelines often rely on human-written questions, expert demonstrations, or stronger teacher models. SearchMaster instead trains a single model through a grounded and regulated self-play loop:

• A Proposer generates search tasks from unlabeled seed documents.
• A Solver answers those tasks through browser-tool rollouts.
• A frozen Verifier checks task validity and answer correctness.
• Verified Proposer and Solver rollouts are jointly optimized with GRPO.

SearchMaster focuses on three common failure modes in search-agent self-play:

• Pseudo multi-hop tasks: questions appear complex but are answerable from one document.
• Shallow difficulty signals: success rate alone cannot tell whether a task requires deep search.
• Over-opening drift: the agent may repeatedly open documents instead of searching for targeted evidence.

To address them, SearchMaster introduces:

• Evidence-Chain Generator (ECG): grounds task generation in explicit cross-document evidence chains.
• Search-Depth Reward (SDR): rewards tasks whose easiest successful solution still requires multi-step search.
• Over-Opening Penalty (OOP): discourages excessive document opening relative to search actions.

Main Results

SearchMaster improves a Qwen3.5-9B backbone across six deep-search benchmarks, raising average accuracy from 38.19% to 51.52%.

Method	BrowseComp-Plus	BrowseComp	GAIA	SEAL-0	WebWalkerQA	XBench	Avg.
Qwen3.5-9B	30.12	20.93	50.49	26.13	41.47	60.00	38.19
SearchMaster	60.24	28.75	57.28	35.14	59.71	68.00	51.52

All numbers report accuracy (%). SearchMaster is trained only in a local search environment, without human-labeled QA pairs or expert demonstrations.

Method

At each self-play iteration, SearchMaster samples unlabeled seed documents from a local corpus and runs the following loop:

1. The Proposer explores the search environment and builds an evidence chain.
2. It converts the chain into a candidate multi-hop question and reference answer.
3. Filters remove invalid, shallow, unsupported, or tool-free-answerable tasks.
4. The Solver attempts each retained task through independent search rollouts.
5. A frozen Verifier judges answer correctness.
6. SDR and OOP assign rewards that favor deep, efficient search behavior.
7. Proposer and Solver rollouts are jointly optimized with GRPO.

The shared policy learns both to generate better search tasks and to solve them with stronger evidence-seeking behavior.

Repository Plan

The following artifacts are planned for release:

• Self-play training code
• Local search environment setup
• Self-generated training data
• Trained SearchMaster checkpoints

Citation

@misc{tan2026searchmastergroundedregulatedselfplay,
      title={SearchMaster: Grounded and Regulated Self-Play for Search Agents}, 
      author={Wentao Tan and Qiong Cao and Jiaqi Wang and Nan Duan},
      year={2026},
      eprint={2608.01822},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2608.01822}, 
}