---

Timeline

• [2026/08/29] We release VideoSearcher-8B, VideoSearcher-Data, the project page, and the full training and evaluation code! Both stages are here: supervised cold start in sft/, BiSPO reinforcement learning at the repo root.
• [2026/08/21] VideoSearcher has been accepted to EMNLP 2026 Main Conference! See you at the conference!
• [2026/07/03] We release the VideoSearcher paper!

---

Overview

Video understanding is moving beyond closed-context perception toward open-world evidence exploration, a paradigm formalized as Video Deep Research (VDR). Existing multimodal search agents primarily target static images, and the current VDR benchmark relies on text-centric retrieval that discards crucial visual information.

VideoSearcher is a closed-loop agentic framework that gives Vision-Language Models multi-tool reasoning for VDR. It unifies temporal localization, spatial focusing, and multimodal search within a single reasoning trajectory, letting the agent progressively ground visual clues, retrieve relevant evidence, and synthesize an answer.

To optimize knowledge-intensive reasoning trajectories we propose BiSPO (Bi-branch Sequence Policy Optimization), a reinforcement learning algorithm that decouples tool-invocation optimization from answer-accuracy optimization, providing stable learning signals for both evidence-grounded reasoning and purposeful tool use.

We also release VideoSearch-QA, the first benchmark designed to evaluate open-world video information grounding and multimodal search-based reasoning.

This repository covers both training stages. The supervised cold start lives in sft/; the RL stage is at the repo root. Run SFT first, then point the RL scripts at its checkpoint.

---

The agent's tool suite

The video is decoded at 1 fps with the frame index watermarked into each frame's top-left corner. The agent starts from 64 uniformly-sampled frames and five tools. Each turn it emits a <think> block followed by either one tool call or a final <answer>, and the tool result is folded back into its context. The agent gets at most 5 exploration loops before it must answer.

Tool	Capability	What it does
choose_frames	Temporal localization	Narrows to a frame interval and returns 8 uniformly-sampled frames from it
find_frame	Temporal localization	Locks onto a single frame index for close examination
zoom_in	Spatial focusing	Magnifies a normalized bbox of the locked frame
image_search	Multimodal search	Reverse-image-searches a cropped region to identify an unknown entity
web_search	Textual search	Queries the web / a local Wikipedia index for external knowledge

Tool schemas and hyperparameters: config/tool_config/tools_video_train.yaml.

---

Results

VideoSearcher-8B averages 57.66% across eight search-oriented benchmarks, +15.71 points over its Qwen3-VL-8B backbone.

It also holds up on general video understanding, so the agentic training does not trade away base video capability:

Ablations. The bi-branch objective versus single-branch GRPO / GSPO:

And the contribution of the localization and retrieval tools:

Case studies — two successes and one failure

A failure mode worth knowing about: noisy retrieval feedback propagating into a wrong final answer.

---

Repository layout

VideoSearcher/
├── sft/                         # Stage 1: supervised cold start
│   ├── setup.sh                 #   clones LLaMA-Factory @ pinned commit + patches it
│   ├── configs/                 #   8B and 4B SFT recipes
│   └── patches/                 #   our 3-line diff to upstream
├── scripts/                     # Stage 2: RL
│   ├── train_8b.sh              #   8B backbone (main result)
│   ├── train_4b.sh              #   4B backbone
│   ├── train_videosearcher.sh   #   shared training core (all knobs)
│   └── merge_rl_ckpt.sh         #   FSDP shards -> HuggingFace format
├── config/tool_config/          # tool schemas for train / validation
├── verl/                        # vendored verl fork (see "What we changed")
├── web_search_server/           # self-hosted web_search backend
├── video_dr_gen/                # tool backend: prompts, bbox ops, image search
├── inference/                   # evaluation harness for VideoSearch-QA
├── data/                        # dataset manifests (data itself is on 🤗)
├── assets/                      # figures used in this README
├── .env.example                 # every credential and endpoint, documented
└── requirements.txt

video_dr_gen/ serves double duty: it holds the agent's system prompt and the bbox/crop/image-search primitives that inference/eval.py calls, and it is also the pipeline that generated the SFT trajectories in VideoSearcher-Data. Use it to build trajectories over your own videos.

What we changed inside verl/

verl/ is a fork of volcengine/verl. Almost all of it is upstream; the VideoSearcher contributions are:

Path	Contribution
verl/tools/{choose_frames,find_frame,zoom_in}_tool.py	The three vision tools
verl/tools/video_image_search_tool.py	Crop → upload → reverse image search
verl/tools/video_web_search_tool.py	Web search with SFT-aligned prompting
verl/workers/actor/bispo_actor.py, verl/trainer/ppo/bispo_algos.py	BiSPO two-branch objective
verl/workers/reward_manager/metis.py	Reward: accuracy, format, grounding, tool cost
verl/experimental/agent_loop/tool_agent_loop.py	Multi-turn agentic rollout
verl/utils/dataset/rl_dataset_json_v2.py	Video-trajectory dataset loader

---

Installation

Requires Python 3.10+, CUDA 12.1+, and 8×80GB GPUs per node for the 8B configuration.

git clone https://github.com/Stephen-gzk/VideoSearcher.git
cd VideoSearcher

conda create -n videosearcher python=3.10 -y
conda activate videosearcher

pip install -r requirements.txt
pip install -e verl/

A prebuilt image is also provided:

docker build -t videosearcher:latest .

---

Configuration

All configuration is by environment variable. Copy the template and fill in what you need:

cp .env.example .env

.env is gitignored. Never commit credentials.

Credentials, and how to get them

Variable	Service	How to register	Required?
SERPER_API_KEY	Serper — Google Search & Lens API, backs web_search and image_search	Sign up, copy the key from the dashboard. Free tier ≈ 2,500 queries.	Yes, unless offline stubs are on
OSS_ACCESS_KEY_ID OSS_ACCESS_KEY_SECRET OSS_BUCKET_NAME OSS_ENDPOINT	Object storage — image_search must expose a cropped region at a public URL before it can be reverse-searched	See Object storage below	Yes, unless offline stubs are on
LLM_JUDGE_URL JUDGE_MODEL	The LLM judge that scores answers, in training and evaluation	Any OpenAI-compatible server you host — no external account needed. See Tool services.	Yes for meaningful rewards/scores
OPENAI_API_KEY OPENAI_BASE_URL	Only for scoring hosted closed models (GPT-5.x, Gemini) as evaluation baselines	platform.openai.com/api-keys	Optional
WANDB_API_KEY	Weights & Biases run tracking	wandb.ai/authorize. Or set WANDB_MODE=offline, or TRAINER_LOGGER="['console']" to skip it.	Optional
HF_TOKEN	HuggingFace, for gated models and uploads	huggingface.co/settings/tokens	Optional
TAVILY_API_KEY	Tavily — alternative web-search backend	Sign up for a free key	Optional
JINA_API_KEY	Jina Reader — cleaner page-content extraction	Free tier available	Optional

Object storage

image_search works by cropping a region of a frame, uploading it, and sending the resulting public HTTPS URL to the reverse-image-search API. Any S3-compatible bucket with public read access will do.

The reference implementation uses Alibaba Cloud OSS (intl.aliyun.com/product/oss):

1. Create an account and an OSS bucket, with ACL set to public read.
2. Create a RAM user with AliyunOSSFullAccess, and generate an AccessKey pair.
3. Put the AccessKey ID/Secret, the bucket name, and the regional endpoint (e.g. oss-cn-shanghai.aliyuncs.com) into .env.

To use AWS S3 or another provider instead, swap the upload call in verl/verl/tools/video_image_search_tool.py — it is a single put_object and the rest of the tool only needs the returned URL.

Running without any paid API

For a smoke test of the training loop, stub out the network tools:

export IMAGE_SEARCH_OFFLINE_TEST=true
export WEB_SEARCH_OFFLINE_TEST=true

Both return empty results instantly. Training will run end-to-end but the reported numbers are meaningless — set both to false for any real run.

---

Tool services

web_search is backed by two services you host yourself. Start both before training.

1. Wikipedia retrieval server (Search-R1)

Provides the local dense-retrieval corpus (e5 embeddings + FAISS).

git clone https://github.com/PeterGriffinJin/Search-R1.git
cd Search-R1

Follow Search-R1's README to fetch the wiki-18 corpus and its prebuilt e5_Flat.index (a flat FP32 index over 21M passages — budget tens of GB of disk), then edit file_path at the top of retrieval_launch.sh to point at them and start the server:

bash retrieval_launch.sh    # serves on :8000

Verify it is not just listening but actually serving the index:

curl -f --noproxy '*' -X POST http://127.0.0.1:8000/retrieve \
  -H "Content-Type: application/json" \
  -d '{"queries":["test"],"topk":1,"return_scores":true}'

"return_scores": true is required — some builds return HTTP 500 without it, which is easy to misread as "the service is down".

2. Web search server

Retrieval plus LLM summarization of the retrieved pages. Lives in this repo.

cd web_search_server
pip install -r requirements.txt

export SERPER_API_KEY=<your key>
export SUMMARIZER_BASE_URL=http://127.0.0.1:8123/v1   # your sglang/vLLM server
export SUMMARIZER_API_KEY=EMPTY                        # sglang ignores the value
export SUMMARIZER_MODEL=qwen3-32b                      # a key in llm_config.json
export LOCAL_DATABASE_ADDRESS=127.0.0.1:8000

python server.py --port 8001

web_search_server/llm_config.json is a registry mapping short names to concrete models and providers. SUMMARIZER_MODEL (and text_search_llm_model in the tool configs) must be one of its keys — not a raw model id. Add an entry to use your own:

"my-summarizer": {
  "model_name": "Qwen/Qwen3-32B",       // what your server was launched with
  "api_key_var": "SUMMARIZER_API_KEY",
  "base_url_var": "SUMMARIZER_BASE_URL",
  "provider": "openai"                   // or "azure"
}

Test the full path — retrieval and summarization, since only the latter touches the LLM:

curl --noproxy '*' -m 120 -X POST http://127.0.0.1:8001/search \
  -H "Content-Type: application/json" \
  -d '{"query":"Eiffel Tower height","top_k":3,"retrieval_mode":"local",
       "local_database_address":"127.0.0.1:8000","prompt_type":"mmsearch_r1",
       "enable_thinking":false}'

A real summary means success. Failed to generate final summary means the server cannot reach the summarizer LLM.

3. Judge / summarizer model

Any OpenAI-compatible server. For example, with sglang:

bash inference/serve_summarizer.sh   # serves a judge model on :8123

If you use a proxy: clear http_proxy/https_proxy before launching sglang. It sends a warm-up request to its own bind address at startup; if that gets hijacked by a proxy it returns 502 and sglang kills itself ~2 minutes in — presenting as "started fine, then died for no reason".

---

Data

Training trajectories and the VideoSearch-QA benchmark are on HuggingFace:

huggingface-cli download Stephengzk/VideoSearcher-Data --repo-type dataset --local-dir data/

TRAIN_FILES and VAL_FILES point at dataset manifests, not at the samples themselves. A manifest maps a split name to its annotation file, its media root, the reward functions to apply, and the system prompt / output-format template the agent is trained against:

{
  "videosearch_qa_train": {
    "annotation": "data/VideoSearcher-Data/videosearch_qa_train.jsonl",
    "root":       "data/VideoSearcher-Data",
    "repeat_time": 1,
    "reward_fn":  ["llm_score", "format_score"],
    "input_template": {
      "name": "general",
      "arguments": {
        "system_prompt": "# Role\nYou are an advanced Video DeepResearch ...",
        "format_instruction": "",     // empty: the contract lives in system_prompt
        "add_image_path": false       // don't inline frame paths into the prompt
      }
    }
  }
}

Several splits can be listed in one manifest; the loader concatenates them, applying each split's own repeat_time and template. Worked examples with the full production system prompt: data/videosearch_qa_train.example.json · data/videosearch_qa_test.example.json.

Each line of the referenced .jsonl is one QA sample carrying the video id, the question, the reference answer, and the frame directory that choose_frames / find_frame index into.

---

Training

Stage 1 — supervised cold start

Run sft/ first. It teaches the base model the tool-call format and produces the checkpoint that RL starts from — RL from a cold base model is much weaker, because the reward signal is uninformative until the model reliably emits valid tool calls.

cd sft && bash setup.sh          # clones + patches LLaMA-Factory
cd LLaMA-Factory
llamafactory-cli train configs/videosearcher_sft_8b.yaml

See sft/README.md for the full recipe.

Stage 2 — reinforcement learning with BiSPO

# 8B (main result)
MODEL_PATH=/path/to/videosearcher-8b-sft bash scripts/train_8b.sh

# 4B
MODEL_PATH=/path/to/videosearcher-4b-sft bash scripts/train_4b.sh

Multi-node:

# head node
NODE_RANK=0 NNODES=2 MODEL_PATH=... bash scripts/train_8b.sh
# each worker
NODE_RANK=1 NNODES=2 MASTER_ADDR=<head_ip> MODEL_PATH=... bash scripts/train_8b.sh

Training resumes automatically from the latest checkpoint (trainer.resume_mode=auto), so a crashed run can simply be relaunched with the same command.

Algorithm toggles

ENABLE_BISPO=true (the default) enables the full method. The individual components can be ablated:

Variable	Default	Effect
ENABLE_BISPO	true	Master switch — turns on both branches below
ENABLE_TOOL_BRANCH	follows ENABLE_BISPO	The decoupled tool-invocation branch
ENABLE_GSPO	follows ENABLE_BISPO	Sequence-level importance ratios
W_ACC / W_TOOL	1.0 / 0.15	Weights of the accuracy and tool branches
ENABLE_SEARCH_PENALTY	true	Marginal-cost table discouraging redundant searches
ENABLE_BN	false	Batch-normalized advantages

Setting ENABLE_BISPO=false reduces the run to a plain GRPO baseline.

Paper ↔ code naming

Algorithm names match the paper (BiSPOActor, bispo_algos.py, adv_estimator=bispo). Two names inherited from upstream remain:

Paper	Code	Why
Composite reward manager	metis reward manager	Named after Metis, which this component is adapted from. Upstream calls the objective HDPO; we call it BiSPO.
VideoSearch-QA	video_deepresearch in some file and dataset names	The project's working name during development.

Exporting a checkpoint

RL checkpoints are sharded FSDP state. Convert to HuggingFace format before serving or evaluating:

bash scripts/merge_rl_ckpt.sh <checkpoint_dir> <output_dir>

---

Evaluation

The eval harness fetches JS-rendered pages via Playwright, which needs a browser downloaded once (pip install alone is not enough):

playwright install chromium

Then:

bash inference/serve_model.sh <merged_model_dir>   # serve the policy
bash inference/serve_summarizer.sh                 # serve the judge/summarizer
bash inference/eval.sh                             # run VideoSearch-QA

inference/eval.sh is the worked example — edit the exports at the top rather than passing everything on the command line. It expects:

Variable	Purpose
MODEL_BASE_URL	OpenAI-compatible server hosting the policy under test
LLM_JUDGE_URL / JUDGE_MODEL	The judge. If unset, llm_score silently falls back to exact-match and your numbers will not be comparable
SUMMARIZER_BASE_URL / SUMMARIZER_MODEL	Condenses pages for the web_search tool
SERPER_API_KEY	Web and reverse-image search
VIDEO_DR_SOURCE_ROOT	Tool backend; defaults to the shipped video_dr_gen/

inference/eval.py drives the same five tools as training, so the tool services above must be running.

To score against hosted closed models (GPT-5.x, Gemini) as baselines, use --model-client proprietary with OPENAI_API_KEY and OPENAI_BASE_URL. The default --model-client openai targets your own MODEL_BASE_URL server and costs nothing.

--judge-client and --judge-temperature are accepted but ignored; the judge is selected purely by LLM_JUDGE_URL / JUDGE_MODEL.

---

Acknowledgements

VideoSearcher builds directly on excellent open-source work:

• verl (Apache-2.0, ByteDance) — the RL training framework this repo forks.
• Metis (Apache-2.0, Accio-Lab) — our two-branch objective and dual-channel reward manager are adapted from Metis's HDPO implementation. See NOTICE for the specific files.
• SenseNova-MARS (MIT) — the multimodal agentic search pipeline our tool-agent rollout and reward structure are derived from.
• Search-R1 — the local Wikipedia retrieval server used by web_search.
• LLaMA-Factory (Apache-2.0) — the SFT stage.
• Qwen3-VL — the base vision-language models.
• sglang — rollout inference.

Upstream licenses are preserved in NOTICE and in the vendored subdirectories.

---

Citation

@article{gao2026videosearcher,
  title   = {VideoSearcher: Empowering Video Deep Research with Multi-Tool
             Agentic Reasoning via Reinforcement Learning},
  author  = {Gao, Zhenkun and Bao, Yicheng and Peng, Jinlong and Li, Xueheng and
             Huang, Theo and Liu, Bangwei and Li, Kunquan and Gan, Zhenye and
             Hu, Tao and Xie, Chengjun and Yang, Mingqian and He, Xuanhua and
             Zhang, Zhizhong and Tan, Xin and Wang, Chengjie and Xie, Yuan},
  journal = {arXiv preprint arXiv:2607.02927},
  year    = {2026}
}

License

Released under the Apache License 2.0. Vendored components remain under their original licenses — see NOTICE.