Framework Design

• Agentic Learning for Tool-Use
  • MCP Tool-Calling: Thanks to the tool-use ability of the Qwen3 model and the support for MCP tools in the qwen_agent codebase, we have integrated this into the RL training process. Users only need to provide an MCP config file to start training with just one click!
  • Efficient Parallel Tool-Calling: Asynchronous parallel tool-calling implemented with asyncio ensures that tool-calling is no longer the bottleneck in RL training! A loss mask is also provided to train only on the model-generated content, greatly improving convergence speed!
  • Low-Cost Tool Environment Setup: With a fully decoupled codebase design, you can build your own tool-calling environment at extremely low cost! We provide a deep search example in main_tutorial.md referencing Search-R1, and will later provide a WebUI-based environment setup.
• Diverse Reward Computation Methods
  • Rule-based Rewards: Largely following verl’s reward computation, but introducing asynchronous parallel reward calculation via asyncio (mainly to support reward computation using tools, such as in NL2SQL tasks).
  • Model Judge-based Rewards: The QwQ-32B model has strong reasoning abilities and can evaluate each response via reasoning. RLFactory supports distributed deployment and calling under such requirements, achieving highly efficient reward rollouts.

Why RL training for Multi-turn Tool Use?

• Industry Application Needs: Tasks like deep search, computer use, and travel planning are essentially multi-turn tool-calling problems, urgently needing a dedicated, plug-and-play RL post-training framework to enhance model capabilities on vertical tasks.
• Agent Evolution Trend: LLM Agents will evolve from calling tools, to calling LLMs as tools, and even calling other agents to form Agentic Graphs. Building this framework for Agentic post-training is a forward-looking necessity.
• Qwen3 Model Advantages: Although Qwen3's overall performance is still behind SOTA models like Deepseek-R1, its tool-use and instruction-following abilities are strong enough to skip the SFT stage and directly perform RL training for domain-specific tasks, bringing more opportunities.

Concept Clarification

• To help users understand what RLFactory can do, here are clarifications for two key questions:
• Q: What is a tool?
• A: Broadly, tools supported by RLFactory are “not generated by the currently trained model”—they can be programs, other models, or even other agents.
  • Program form: Various search interfaces (input query–output result), code interpreters (input code–output execution result), calculators (input formula–output calculation result)
  • Model form: Other open/closed-source models (e.g., using GPT-4o for document summarization, input Prompt–output Response)
  • Agent form: A collection of programs and models (e.g., a literature review agent, input topic–output review result)
• Q: Why do we need multi-turn tool-calling?
  • A: For a given tool-use task, manual orchestration requires a lot of human design. With RLFactory, you can train an end-to-end agent model that decides which tools to call, how to call them, when to stop, and how to organize outputs—all through reasoning. This is a key trend for future LLM applications. The interaction process is illustrated below:
    
  • End-to-End Agent Model Workflow
    • Step 1: Input the original Prompt, call the Training Model to output a Response
    • Step 2: Post-process the model output to parse tool names and parameters (if no tool is parsed, treat as loop termination and output the model response)
    • Step 3: Run the tool(s) according to the parsed names and parameters (possibly in parallel), and post-process the tool results
    • Step 4: Add the post-processed tool results to the Prompt, call the model again until termination

Different layers for different roles (uses or hardcore developers)

• The RLFactory framework allows users to implement RL training for Multi-Turn Tool Use with minimal code. The following diagram categorizes the core modules of the framework:
  • Application Layer: Users need to construct their own Env, provide a tool config file mcp_tools.pydata, and define how to compute the reward function (compute_score)
  • Component Layer: Users do not need to modify, but can customize or add components and features if needed (e.g., new ToolManager)
  • Basic Layer: Users do not need to modify at all, including the Multi-Turn Tool Use interaction logic, Model Judge infrastructure, and the native RL training mechanism in verl
    
• The diagram below uses the GRPO algorithm as an example to show the role of each key RLFactory module in the RL process. Compared to original verl RL training, RLFactory mainly improves on low-code tool-calling environments and efficient reward function design.
  
• Tool-Calling Environment: Core files are in the envs folder. For details, see docs/rl_factory/en/tools.md
  • Tool Definition: Based on qwen_agent, supports custom tools (inheriting from BaseTool), MCP toolsets, and built-in tools (e.g., code_interpreter)
  • Tool Parsing: Register your own manager in the tool_manager folder to implement your own tool parsing logic, or use our qwen3_manager.py for parsing, launching, and flexible calling of all three types of tools
  • Tool Calling: Register your own Env in the envs folder to implement your overall tool-calling logic and reward computation logic. The step function provides default tool-calling logic, inputs a batch of responses, and outputs tool-calling results, truly decoupling tool-calling from the LLM as an environment!
  • Multi-turn Tool Calling & Training: In envs/utils/tool_utils.py, implements prompt concatenation, loss_mask definition and calculation for multi-turn tool-calling, accurately enabling multi-turn tool-calling based on LLM!
• Reward Computation: Core content is in the envs folder. For details, see docs/rl_factory/en/rewards.md
  • Rule-based: Based on verl, implements asynchronous parallel reward calculation with asyncio, mainly for compatibility with future parallel evaluation
  • Model Judge: Adds a reward_rollout_wg and creates a dedicated resource pool, using vllm for distributed deployment and calling of LRM models like QwQ, and generates model Judge results in Env for evaluating the original response
  • Tool Verification: Adds a verify_tool method in Env, which can be called during reward calculation to evaluate the original response. For NL2SQL tasks, the model's final output is code—RLFactory can execute this SQL statement to determine the correctness, also using asynchronous parallelism to improve efficiency (you can also use this to call APIs for Model Judge)