ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents

July 1, 2026 · View on GitHub

This repository contains the official Python implementation of the framework described in the paper "ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents."

News

[01/20/2026] Our paper is available on arXiv.
[01/17/2026] GitHub repository created. Code release is coming soon.

Reward-guided search methods have demon- strated strong potential in enhancing tool-using agents by effectively guiding sampling and ex- ploration over complex action spaces. As a core design, those search methods utilize process reward models (PRMs) to provide step-level rewards, enabling more fine-grained monitor- ing. However, there is a lack of systematic and reliable evaluation benchmarks for PRMs in tool-using settings. In this paper, we intro- duce ToolPRMBenchs, a large-scale benchmark specifically designed to evaluate PRMs for tool- using agents. ToolPRMBench is built on top of several representative tool-using benchmarks and converts agent trajectories into step-level test cases. Each case contains the interac- tion history, a correct action, a plausible but incorrect alternative, and relevant tool meta- data. We respectively utilize offline sampling to isolate local single-step errors and online sampling to capture realistic multi-step failures from full agent rollouts. A multi-LLM verifica- tion pipeline is proposed to reduce label noise and ensure data quality. We conduct exten- sive experiments across large language models, general PRMs, and tool-specialized PRMs on ToolPRMBench. The results reveal clear differ- ences in PRM effectiveness and highlight the potential of specialized PRMs for tool-using.

Figure 1: Comparison between ToolPRMBench and other RewardBench.

Contribution

⭐ New Benchmark. We propose ToolPRMBench, a large-scale benchmark carefully constructed for systematic evaluation of PRMs in tool-using settings.

⭐ Comprehensive Evaluation. We benchmark a series of LLMs, gen- eral PRMs, and tool-using specialized PRMs in ToolPRMBench, building a comprehensive PRMs leaderboard in the tool-using scenario.

⭐ Further Insights. We conduct further analysis, including meta-evaluation and cost analysis, providing findings and insights for future reward-guided trajectory searching in tool-using.

main
Figure 2: The overview pipeline of ToolPRMBench, including trajectory sampling, data verification & filtering, and ToolPRM training.

Repository Structure

ToolPRMBench/
├── data/                      # ToolPRMBench evaluation & training data (v3) + data card
├── results/                   # Precomputed per-model / per-dataset result files
├── scripts/                   # One-command launch scripts
│   ├── run_toolprm.sh         #   ToolPRM family (served via vLLM)
│   └── run_specialized_prms.sh#   General / specialized PRM baselines
├── third_party/
│   └── skywork-o1-prm-inference/  # Vendored Skywork PRM helper code
├── toolprmbench/              # Python package
│   ├── core.py                #   Shared data IO, ordering, scoring, eval loop
│   ├── generation.py          #   API / vLLM backends (keys read from env)
│   ├── models.py              #   Registry of all benchmarked models
│   ├── evaluate.py            #   Unified CLI dispatcher
│   └── evaluators/            #   One module per model family
├── requirements.txt
└── pyproject.toml

Installation

pip install -e .            # core package
pip install -e ".[local]"   # + torch / transformers / vllm for local models
pip install -e ".[api]"     # + openai / anthropic / gemini / together clients

API keys. No credentials are stored in this repository. Hosted-API and vLLM-served evaluators read keys from the environment: OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, TOGETHER_API_KEY, and TOOLPRM_BASE_URL (the local vLLM endpoint, e.g. http://localhost:8001/v1).

Dataset

The benchmark data lives in data/ (see the data card). Each case pairs a correct action (action_chosen) with a plausible-but-incorrect alternative (action_rejected); a PRM is correct when it prefers the former.

Split	Source	# cases
`prmbench_GTA.json`	GTA	118
`prmbench_bfcl.json`	BFCL	111
`prmbench_tooltalk.json`	ToolTalk	86
`prmbench_ToolSandbox.json`	ToolSandbox	130

Override the location with --data_dir or TOOLPRMBENCH_DATA_DIR.

Models

Tool-using specialized PRMs (ToolPRM family)

Trained for this benchmark and released at huggingface.co/wjldw:

Model	Weights	Evaluator
ToolPRM-Base-v4	https://huggingface.co/wjldw/ToolPRM-Base-v4	`toolprm`
ToolPRM-GRPO-v4	https://huggingface.co/wjldw/ToolPRM-GRPO-v4	`toolprm`
ToolPRM-GRPO-synthesis	https://huggingface.co/wjldw/ToolPRM-GRPO-synthesis	`toolprm`
ToolPRM-CoT-v4	https://huggingface.co/wjldw/ToolPRM-CoT-v4	`toolprm_cot`

Baselines (general / specialized PRMs and LLM judges)

Model	Weights	Evaluator
Skywork-o1-Open-PRM	https://huggingface.co/Skywork/Skywork-o1-Open-PRM-Qwen-2.5-7B	`skywork`
RM-R1	https://huggingface.co/gaotang/RM-R1-Qwen2.5-Instruct-7B	`rmr1`
RRM-7B	https://huggingface.co/Reward-Reasoning/RRM-7B	`rrm`
Nemotron-GenRM	https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-GenRM	`nemotron`
ToolACE-8B	https://huggingface.co/Team-ACE/ToolACE-8B	`toolace`
xLAM-7b-r	https://huggingface.co/Salesforce/xLAM-7b-r	`xlam`
Qwen3-Coder	https://huggingface.co/Qwen/Qwen3-Coder-Next	`llm_judge`

List everything registered at any time:

python -m toolprmbench.evaluate --list

Evaluation

All evaluators share one entry point. Results are written to results/prmbench_<dataset>_<model>.json and the accuracy is printed.

By registry key (fills in the evaluator and Hugging Face id automatically):

python -m toolprmbench.evaluate --model-key RM-R1 --dataset bfcl

Explicitly (any model id / local path):

python -m toolprmbench.evaluate --evaluator xlam \
    --model Salesforce/xLAM-7b-r --dataset ToolSandbox

ToolPRM family (serve with vLLM first, then evaluate):

python -m vllm.entrypoints.openai.api_server \
    --model wjldw/ToolPRM-Base-v4 --port 8001 --trust-remote-code &
export TOOLPRM_BASE_URL=http://localhost:8001/v1
python -m toolprmbench.evaluate --evaluator toolprm \
    --model wjldw/ToolPRM-Base-v4 --dataset bfcl

Batch runs over every dataset:

bash scripts/run_toolprm.sh            # ToolPRM family (handles serving)
bash scripts/run_specialized_prms.sh   # all baseline PRMs

Citation

If our repo helped you out, we'd love it if you gave us a citation! Thanks for supporting our work!

@article{li2026toolprmbench,
      title={ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents}, 
      author={Dawei Li and Yuguang Yao and Zhen Tan and Huan Liu and Ruocheng Guo},
      year={2026},
      eprint={2601.12294},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2601.12294}, 
}