PettingLLMs is an open-source framework for on-policy reinforcement learning with multi-agent large language models. It currently powers two lines of work:

• 🆕 Metaagent-X — Breaking the Ceiling of Automatic Multi-Agent Systems via End-to-End Reinforcement Learning.  📄 arXiv:2605.14212 — an end-to-end framework that trains agentic models which can both self-design and self-execute their own MAS, jointly optimizing the meta-designer and the executor.
• Stronger-MAS — On-Policy Reinforcement Learning for Collaborative LLMs.  📄 arXiv:2510.11062 — Agent- and Turn-wise Group Relative Policy Optimization (AT-GRPO) for training collaborative LLM agents in a fixed multi-agent system (MAS), with fine-grained per-agent / per-turn credit assignment and role-specialized policies.

---

1. 🆕 Metaagent-X — End-to-End Trainable Automatic MAS

📄 Paper (arXiv:2605.14212)

🪄 Model

🏆Project Page

Multi-agent systems have shown clear advantages over single-agent approaches across medical decision-making, scientific discovery, financial trading, software engineering, and hardware design. Recent work increasingly turns to meta-agents that automatically design and instantiate the MAS flow best suited to each task; in parallel, agentic RL and self-evolving paradigms are turning LLMs into interactive, continuously improving decision-makers. Yet existing automatic MAS remain only partially adaptive — they either search over MAS structures at test time, or optimize only the designer while freezing the downstream executor. Metaagent-X is our latest framework that closes this gap: it trains agentic models which can self-design and self-execute their MAS end-to-end. Task-conditioned auto-MAS designs are instantiated, executed, grouped, and collected for role-aware policy updates of both the designer and the executor — so the executor is no longer a hard ceiling on the meta-designer, and the designer can induce specialized execution behaviors from its counterpart.

This addresses two fundamental limitations of prior automatic MAS:

1. Parameter-level disjunction. Designer and executor are coupled only through prompt-level interactions at inference time, with no optimization signal that updates the underlying policy from downstream execution outcomes.
2. Vague co-evolution dynamics. How designer and executor co-evolve under joint training — and where each role's improvement comes from — remains unclear in practice.

Results

Across six math and code benchmarks and two base models, Metaagent-X outperforms single-agent and automatic-MAS baselines by up to 21.7%. Ablations show that (1) both the designer and the executor keep improving throughout training across tasks and domains, and (2) effective co-evolution follows a stagewise process in which the two components benefit from decoupled optimization.

Quick Start (Metaagent-X)

# Interactive browser demo.
# This serves Mercury7353/MetaAgent-X with vLLM, opens a web UI, and lets users
# enter math/code queries while inspecting MAS design and execution traces.
bash scripts/evaluate/autoevol/serve_ui.sh

# If the model is already served on this machine or another host, only start the UI:
START_VLLM=false HOST=127.0.0.1 PORT=8300 bash scripts/evaluate/autoevol/serve_ui.sh

# One-shot CLI demo that writes an HTML report instead of serving a UI:
QUESTION="Find the value of x if 2x + 3 = 17. Answer with a single number." \
bash scripts/evaluate/autoevol/serve_demo.sh

# Eval-first benchmark run on the released model
bash scripts/evaluate/autoevol/eval_first_open_model.sh

# Training example: shared-policy co-training with hierarchical M*N rollouts
# and stage-wise alternate learning rates.
bash scripts/train/autoeval/example_cotrain_autoeval.sh

The interactive UI is served at http://127.0.0.1:8899 by default. Each run stores its artifacts under outputs/autoeval_interactive/, including mas_design.py, executable mas.py, execution.log, index.html, retry attempts, and the workflow visualization. The UI shows math/code examples, the model's MAS design, execution pipeline, AgentNode traces, full logs, and final result. The auto-MAS environment, designer/executor agents, and reward functions live under pettingllms/multi_agent_env/autoevol/, with configs in pettingllms/config/autoevol/.

---

2. Stronger-MAS / AT-GRPO

📄 Paper (arXiv:2510.11062)

AT-GRPO (Agent- and Turn-wise Group Relative Policy Optimization) trains collaborative LLM agents across diverse tasks within a fixed MAS topology.

Highlights

• AT-GRPO algorithm for fine-grained agent and turn-wise credit assignment.
• Agent-specific policies via LoRA or fully independent models.
• Multi-level rewards: process, agent, and global/team signals.
• Multimodal examples (e.g., Qwen2.5VL) for vision + language tasks.
• Seamless switch between single-agent and multi-agent training flows.

Feature Snapshot

Capability	PettingLLMs	AgentLightning / VERL (typical)
Agent-specific LoRA & models (per-agent adapters or different base models)	✅	❌ (one shared model)
Multi-level rewards (process + agent + global/team)	✅	❌ (mostly global only)
Fine-grained grouping (turn/phase/role/tool-call)	✅	❌ (often one-task = one-group)
Multimodal (see Qwen2.5VL examples)	✅	❌

Supported modes

• ✅ Single-agent RL training
• ✅ Multi-agent RL training (one role-sharing policy)
• ✅ Multi-agent RL training (role-specialized policies using different LoRA adapters or different LLMs)

Agent Specification Levels

Level	Specification Type	Architecture Components	Trajectory Flow	Description
L1	Shared Policy (agent-specific prompt)	1 base model + distinct prompts	Shared trajectory	All agents share the same base model; roles are defined via different system prompts.
L2	Agent-specific Policy (agent-specific LoRA)	1 base model + LoRA adapters	Per-agent trajectory	Agents share a base model but use lightweight, role-specific LoRA adapters for specialization.
L3	Agent-specific Model (full weights)	Independent models (Model 1, Model 2, Model 3...)	Per-agent trajectory	Each agent runs a separate model instance for maximal specialization.

MAS Design Options

Category	Design Paradigm	Key Features & Support	Best For
A	Graph-based agent	Flexible topology; integrates with frameworks like AutoGen, Ag2, LangChain.	Complex, non-linear workflows needing external agent ecosystems.
B	Turn-based agent (finite-state machine)	Fine-grained control; customizable sequential execution.	Scenarios requiring precise operation order and state transitions.
C	AFlow Co-Evolve [experiment]	Automated design via a lightweight MAS-designer.	Experimental setups where the system self-optimizes agent structures.

---

📰 News

• [2026.04] 🧠 Metaagent-X released — end-to-end RL for self-designing and self-executing automatic MAS; +21.7% over baselines across six math/code benchmarks.
• [2025.12] ✅ Roadmap milestone delivered: more environments (Verilog design, web search, robotics, database query, scientific discovery), multimodal support, and agentic framework integrations (AutoGen, LangGraph, LlamaIndex).
• [2025.10] 🚀 GitHub repository open-sourced and publicly available.
• [2025.10] 🎉 AT-GRPO (Stronger-MAS) paper released! Check out our arXiv preprint.
• [2025.10] 🔥 Support for different LoRA adapters per agent role—efficient role-specialized training.
• [2025.09] 🌍 Multi-environment support added: Game (Sudoku, Sokoban), Code (APPS, CodeContests), Math (AIME, OlympiadBench).
• [2025.08] 🤖 Multi-agent framework implementation: supports both shared single model and role-specific models.

📦 Installation

git clone https://github.com/pettingllms-ai/PettingLLMs.git
cd PettingLLMs
bash setup.bash

🎯 Quick Start

1) Dataset preparation

# Code tasks (APPS, CodeContests, LiveCodeBench)
python scripts/dataprocess/load_code.py

# Math tasks (AIME24/25, OlympiadBench)
python scripts/dataprocess/load_math.py

# Game/Planning tasks (Sokoban, Sudoku)
python scripts/dataprocess/load_sokoban.py

Datasets are saved to datasets/code/, datasets/math/, and datasets/sudoku_environments/.

2) Training

# Metaagent-X: shared-policy co-training with M*N hierarchical rollouts
bash scripts/train/autoeval/example_cotrain_autoeval.sh

# AT-GRPO: fixed multi-agent system on math tasks
bash scripts/train/math/math_L1_prompt.sh

Other AT-GRPO training scripts live in scripts/train/:

• code_single_policy.sh, code_two_policy.sh (code)
• plan_path_single.sh, plan_path_two_policy.sh (planning)
• sokoban_two_policy.sh, sokodu_single.sh (games)

3) Evaluation

Edit scripts/evaluate/evaluate.sh to set your model path and config:

MODEL_PATHS=("/path/to/your/model")
CONFIG_NAME="math_single_policy"

Then run:

bash scripts/evaluate/evaluate.sh

For MetaAgent-X, the eval-first entry point defaults to the released model:

bash scripts/evaluate/autoevol/eval_first_open_model.sh

📚 Citation

If you find PettingLLMs useful for your research or projects, please cite the relevant paper:

@misc{zhang2026metaagentxbreakingceiling,
      title={MetaAgent-X : Breaking the Ceiling of Automatic Multi-Agent Systems via End-to-End Reinforcement Learning}, 
      author={Yaolun Zhang and Yujie Zhao and Nan Wang and Yiran Wu and Jiayu Chang and Yizhao Chen and Qingyun Wu and Jishen Zhao and Huazheng Wang},
      year={2026},
      eprint={2605.14212},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2605.14212}, 
}

@article{zhao2025stronger,
  title={Stronger Together: On-Policy Reinforcement Learning for Collaborative LLMs},
  author={Zhao, Yujie and Hu, Lanxiang and Wang, Yang and Hou, Minmin and Zhang, Hao and Ding, Ke and Zhao, Jishen},
  journal={arXiv preprint arXiv:2510.11062},
  year={2025}
}

🔗 Acknowledgements

This work was primarily conducted by Yujie Zhao during her summer internship at Intel Corporation. We gratefully acknowledge Intel's support and resources.

• VERL: VERL: Efficient RL Training for LLMs — efficient distributed RL training infrastructure.
• RLLM: RLLM: Reinforcement Learning with Language Models — foundational RL algorithms for LLMs.

📌 License

Released under the MIT license. See LICENSE for details.