Group-Reflective Self-Distillation for Agentic Reinforcement Learning

Policy-native reflection and turn-level credit assignment from verified on-policy experience

Reflect on verified rollouts. Contrast success with failure. Distill the difference into turn-level credit.

News  ·  Quickstart  ·  Overview  ·  Method  ·  Results  ·  Layout  ·  Reproduce  ·  Launchers  ·  Checkpoints  ·  Verification

🔥 News

• 2026.07.30 — We have open-sourced the official GRSD implementation, and the paper is now available on arXiv.

⚡ Quickstart

# 1. environment
conda create -n grsd python=3.10 -y && conda activate grsd
pip install vllm==0.11.0 && pip install -e .

# 2. one task (ALFWorld)
pip install gymnasium==0.29.1 stable-baselines3==2.6.0 alfworld
export ALFWORLD_DATA="$PWD/data/alfworld" && alfworld-download -f
bash examples/data_preprocess/prepare_alfworld.sh

# 3. train
bash examples/run_grsd_alfworld_qwen3_local.sh

🔎 Overview

GRSD lets an agent learn fine-grained credit from its own verified experience. For each prompt, the policy reflects on every rollout, contrasts successful and failed reflections into compact group-level DO / AVOID guidance, and uses that guidance as privileged context for turn-level self-distillation.

Terminal rewards tell an agent whether a trajectory succeeded, but not which of its many decisions deserve credit. GRSD keeps the stable GRPO objective and adds a policy-native signal for that missing credit assignment.

🧠 Method

_{GRSD pipeline. Privileged guidance is used during training only; inference uses the plain task prompt.}

1. Policy-native self-reflection. For every verified rollout, the rollout policy identifies outcome-critical decisions and extracts a trajectory-level reflection skill. An optional OpenAI-compatible endpoint supplies only a scalar rubric score for the reflection objective; it never generates or revises reflection content.
2. Group-level guidance construction. A stop-gradient policy snapshot contrasts reflection-derived skills from successful and failed rollouts of the same prompt and synthesizes concise, outcome-discriminative DO / AVOID guidance.
3. Turn-level self-distillation. The same sampled actions are evaluated under plain and guidance-augmented contexts. Their aligned likelihood difference refines the trajectory-level GRPO advantage into turn-specific credit without changing its verifier-determined sign.

The policy-native implementation is the official GRSD variant and is selected by default. The legacy verl.trainer.main_grsd entry point is retained as the external-reflection ablation (GRSD_VARIANT=external) for comparisons and backward compatibility.

📊 Results

The paper evaluates three long-horizon environments and three instruction-tuned backbones. Values below are the reported per-environment averages; ALFWorld and SearchQA are percentages, while WebShop reports normalized score / success rate.

_{Performance overview. GRSD delivers strong aggregate results across environments and model scales.}

📈 Training dynamics

On ALFWorld with Qwen3-1.7B, GRSD continues to improve after the baselines begin to plateau and reaches the highest validation success rate. It also maintains a smaller absolute mean teacher-student gap than Skill-SD and SDAR, consistent with more compatible guidance and smoother late-stage optimization.

🪞 Reflection strategy

Policy-native reflections are compared with external-model reflection generation. The official GRSD pipeline achieves higher late-stage success while maintaining a smaller-magnitude teacher-student gap, indicating guidance that better matches the policy's capability boundary.

Overall success rate

Mean teacher-student gap

🎛️ Hyperparameter sensitivity

The default settings use a moderate reflection-loss weight and self-distillation strength. The paper sweeps both to expose the stability / credit-shaping trade-off.

Reflection-loss weight alpha

Self-distillation strength lambda

🌐 Cross-domain generalization

GRSD transfers beyond the in-domain ALFWorld training distribution and improves the Unseen task types, with particularly clear gains on Cool and Pick2.

🗂️ Repository layout

GRSD/
├── agent_system/                # ALFWorld, SearchQA, and WebShop environments
├── examples/
│   ├── data_preprocess/         # task download and Parquet preparation
│   ├── search/retriever/        # local SearchQA retrieval service
│   └── run_grsd_*_local.sh      # complete 3 task x 3 backbone matrix
├── verl/trainer/ppo/            # GRSD trainers, reflection, and advantage logic
├── skills/                      # task skill prompts and mappings
├── docs/grsd/                   # figures used in the paper and this README
├── scripts/                     # checkpoint conversion and merging utilities
└── tests/                       # focused GRSD regression tests

🛠️ Reproduce

The public launchers use repository-relative defaults. Set MODEL_ROOT, DATA_ROOT, MODEL_PATH, DATA_DIR, CHECKPOINT_DIR, and PYTHON_BIN to adapt them to your machine; no absolute path from the development environment is required.

conda create -n grsd python=3.10 -y
conda activate grsd

pip install vllm==0.11.0
pip install flash-attn==2.7.4.post1 --no-build-isolation --no-cache-dir
pip install -e .

pip install gymnasium==0.29.1 stable-baselines3==2.6.0 alfworld
export ALFWORLD_DATA="$PWD/data/alfworld"
alfworld-download -f
bash examples/data_preprocess/prepare_alfworld.sh

pip install -e agent_system/environments/env_package/search/third_party
pip install gym==0.26.2
bash examples/data_preprocess/prepare_search.sh

bash examples/search/retriever/retrieval_launch.sh

cd agent_system/environments/env_package/webshop/webshop
./setup.sh -d all
cd ../../../../../
bash examples/data_preprocess/prepare_webshop.sh

export JUDGE_API_KEY=your_key
export JUDGE_API_BASE=https://your-openai-compatible-endpoint/v1
export JUDGE_MODEL=your_judge_model

🚀 Launchers

There is one official GRSD launcher for every combination of the three tasks and three backbones. Each defaults to GRSD_VARIANT=grsd and can be overridden without editing the script.

Run from the repository root, for example:

bash examples/run_grsd_alfworld_qwen3_local.sh
bash examples/run_grsd_search_qwen2.5_3b_local.sh
bash examples/run_grsd_webshop_qwen2.5_7b_local.sh

The paper defaults are summarized below; every setting can be overridden with an environment variable or appended Hydra arguments.

To reproduce the external-reflection ablation:

GRSD_VARIANT=external bash examples/run_grsd_alfworld_qwen3_local.sh

GRSD_VARIANT=reflect and GRSD_VARIANT=original remain accepted as legacy aliases for grsd and external.

💾 Checkpoints and evaluation

Training outputs default to checkpoints/. Use scripts/model_merger.py to merge FSDP or Megatron shards into a Hugging Face model directory. Evaluation uses the task-specific environment wrappers and the same plain prompt format used by training.

✅ Verification

Run the focused GRSD regression suite before a long GPU job:

python -m unittest tests.trainer.ppo.test_grsd_advantage
python -m compileall -q verl agent_system examples/data_preprocess
for script in examples/run_grsd_*local.sh examples/run_grsd_reflect_*local*.sh; do bash -n "$script"; done

The launchers also validate required data/model paths, CUDA visibility, and (when enabled) the judge or SearchQA retriever endpoint before starting Ray.

When a run fails for environment reasons, scripts/diagnose.py reports the OS, hardware, Python, pip, and network state in one pass:

python scripts/diagnose.py

🙏 Acknowledgements

GRSD builds on veRL, verl-agent, ALFWorld, WebShop, and Search-R1. We also acknowledge the public SDAR repository, which provided an important implementation reference for agentic self-distillation. We thank the authors and contributors of these projects.

📝 Citation

@misc{zheng2026groupreflectiveselfdistillationagenticreinforcement,
      title={Group-Reflective Self-Distillation for Agentic Reinforcement Learning},
      author={Binbin Zheng and Zijun Xie and Guanqun Zhao and Enlei Gong and Xing Ma and Xiaoliang Fu and Zeyu Chen},
      year={2026},
      eprint={2607.28076},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2607.28076},
}