AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments

📃 Paper • 🤗 Dataset

🔔 News

• 🎉 AgentGym2 is released, including the benchmark, evaluation code, and tools for non-idealized agent evaluation in real-world digital environments.
• 🤖 AgentGym2 focuses on complex tool usage, data analysis, and deep search, and evaluates agents with a basic, composable toolbox plus full internet access.

🌟 Introduction

Existing agent benchmarks often evaluate models in simplified, idealized settings: tool interfaces are pre-packaged, intermediate steps are assumed solved, and inputs are relatively clean and well-specified. In contrast, AgentGym2 evaluates language agents under de-idealized, real-world conditions, where tools must often be discovered, pipelines must be composed from scratch, and information can be noisy, ambiguous, or underspecified.

AgentGym2 is built on realistic digital environments and provides agents with a basic yet composable toolbox instead of pre-selected, task-specific tools. This design encourages agents to proactively explore the environment, discover additional tools when needed, and execute complete end-to-end workflows under uncertainty.

📦 AgentGym2 Benchmark

AgentGym2 spans 437 tasks across 27 domains with ZH/EN requests. It covers three core scenarios:

• Complex Tool Use: agents must discover and compose tools in realistic web environments.
• Data Analysis: agents must explore raw data, understand cleaning or transformation requirements, and complete end-to-end analysis.
• Deep Search: agents must retrieve, synthesize, and verify information across multiple sources while handling ambiguous or noisy user queries.

Each query may be paired with multimodal attachments, including documents, images, audio, or video. All tools are encapsulated into a unified interface and can be invoked in parallel when appropriate, supporting scalable and reproducible evaluation.

Statistics	Number
Total queries	437
- Complex Tool Use	182
- Data Analysis	57
- Deep Search	198
Language	ZH / EN
Total attachments	652
Attachment modalities	Text, Image, Audio, Video
Basic tool types	27

🛠 Framework

AgentGym2 adopts a layered design with four major components:

• Environment: the realistic digital runtime that supports task execution, including web browsing, file systems, code execution, and internet access.
• Tools: general-purpose interfaces exposed by environments. The toolbox covers web browsing, information retrieval, file processing, multimodal understanding, and code execution.
• Agent: the LLM-based agent that receives observations, plans and reasons over its trajectory, and decides which tools to call.
• Configuration: user request, initial state, and verification mechanism. A single request may involve multiple environments and multimodal inputs.

This modular design decouples agents from underlying environments and tools, making evaluation rigorous, extensible, and easier to scale.

📊 Main Results

Below is the main performance table on AgentGym2. Within each model group, bold indicates the best score.

🛠 Usage & Quick Start

This project contains the agentenv python package and the integrated environments.

Setup agentenv pacakage

from PyPI

pip install agentenv

from Source

git clone --recursive https://github.com/hotdog-zz/Agentgym2
cd ./AgentGym2

cd agentenv
pip install -e .

Depending on which environments you want to use, cd into the corresponding agentenv-* and agenttool-camel folder and follow the README.md inside.

Evaluation Examples

• See agentenv/examples/multiprocessing/openai_eval_multiprocessing_toolkit_launch_from_python.py

🔑 Key Takeaways

• Even strong contemporary systems such as GPT-5 and Claude-4.5-Sonnet still struggle on AgentGym2, indicating that today’s agents remain far from robust production use in non-idealized settings.
• Open-source models still lag significantly behind the best proprietary systems, while stronger agentic post-training can bring substantial gains.
• Data Analysis is especially challenging, highlighting that end-to-end wrangling and analysis, rather than only high-level reasoning, remain major bottlenecks.
• In practice, tool discovery is a strong prerequisite for success, but downstream abilities such as tool composition and verification also remain critical.

🌐 Relevant Repositories

• AgentGym: a versatile repository for evolving and evaluating LLM-based agents across diverse environments.
• AgentGym-RL: a reinforcement learning extension of AgentGym for training LLM agents through multi-turn interaction.

📧 Contact

• zhxi22@m.fudan.edu.cn

🔖 Citation

@misc{xi2026agentgym2benchmarkinglargelanguage,
      title={AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments}, 
      author={Zhiheng Xi and Dingwen Yang and Jiaqi Liu and Jixuan Huang and Honglin Guo and Baodai Huang and Tinggang Chen and Qi Zhang and Zhonghang Lu and Chenyu Liu and Jiajun Sun and Jiazheng Zhang and Dingwei Zhu and Xin Guo and Junzhe Wang and Zhihao Zhang and Yuming Yang and Junjie Ye and Minghe Gao and Dongrui Liu and Jiaming Ji and Guohao Li and Tao Gui and Qi Zhang and Xuanjing Huang},
      year={2026},
      eprint={2607.05174},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2607.05174}, 
}

@misc{xi2025agentgymrltrainingllmagents,
      title={AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning}, 
      author={Zhiheng Xi and Jixuan Huang and Chenyang Liao and Baodai Huang and Honglin Guo and Jiaqi Liu and Rui Zheng and Junjie Ye and Jiazheng Zhang and Wenxiang Chen and Wei He and Yiwen Ding and Guanyu Li and Zehui Chen and Zhengyin Du and Xuesong Yao and Yufei Xu and Jiecao Chen and Tao Gui and Zuxuan Wu and Qi Zhang and Xuanjing Huang and Yu-Gang Jiang},
      year={2025},
      eprint={2509.08755},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2509.08755}, 
}

@misc{xi2024agentgym,
      title={AgentGym: Evolving Large Language Model-based Agents across Diverse Environments}, 
      author={Zhiheng Xi and Yiwen Ding and Wenxiang Chen and Boyang Hong and Honglin Guo and Junzhe Wang and Dingwen Yang and Chenyang Liao and Xin Guo and Wei He and Songyang Gao and Lu Chen and Rui Zheng and Yicheng Zou and Tao Gui and Qi Zhang and Xipeng Qiu and Xuanjing Huang and Zuxuan Wu and Yu-Gang Jiang},
      year={2024},
      eprint={2406.04151},
      archivePrefix={arXiv},
      primaryClass={cs.AI}
}