🤖🗺️ Reinforced Reasoning for Embodied Planning

August 16, 2025 · View on GitHub

Reinforced Reasoning for Embodied Planning

Di Wu, Jiaxin Fan*, Junzhe Zang*, Guanbo Wang, Wei Yin, Wenhao Li†, Bo Jin†

Tongji MAIL-TAII Lab, Tongji University, Tsinghua University, Bank of Communications

🧐 Overview

Multi-step embodied planning tasks challenge agents to translate first-person visual observations and natural-language goals into coherent, sequential actions that dynamically affect their environment. Unlike static tasks like visual QA or math, these tasks require continuous spatial reasoning, temporal consistency, and adaptation across multiple decision turns.

However, current vision-language models (VLMs), even those with strong static capabilities, often fail in such settings—producing redundant, incoherent, or short-sighted action sequences.

To address this gap, we propose:

🔁 Reinforced Reasoning for Embodied Planning A two-stage fine-tuning pipeline that equips VLMs with multi-step reasoning and long-horizon planning abilities in dynamic environments.

🔧 Our Pipeline

  1. Supervised Fine-tuning (SFT): We distill ~4k high-quality multi-step trajectories from a powerful teacher model into Qwen2.5-VL-7B to initialize commonsense and structured reasoning.

  2. Reinforcement Fine-tuning (RFT): Using a rule-based reward tailored for multi-step decision quality, we apply GRPO to align model outputs with long-horizon planning objectives.

🧪 Results

We evaluate on the Embench benchmark (EB-ALFRED & EB-Habitat) and find that our 7B model:

  • Outperforms 70B+ open-source baselines,
  • Surpasses GPT-4o-mini in planning efficiency and coherence,
  • Generalizes well to unseen embodied environments.

🗒️ News

  • 2025-08-16 : 📦 Update relevant data and RFT model
  • 2025-05-28 : 🚀 We released our initial arXiv paper on Reinforced Reasoning for Embodied Planning.
  • 2025-06-06 : 🧠 Github Page Released.
  • 2025-06-17 : 🛠️ Published the github page and related data.

🏗️ Model and Dataset

We release all key components of our two-stage training pipeline, including datasets and models for both Supervised Fine-tuning (SFT) and Reinforcement Fine-tuning (RFT).

ComponentDescriptionDownload
🧩 SFT Dataset~4.2k Gemini-2.0 distilled multi-step plans🤗 Hugging Face
🧩 RFT Dataset~43.9k ALFRED-derived episodes with reward annotations🤗 Hugging Face
🧠 SFT ModelQwen2.5-VL-7B fine-tuned with SFT only🤗 Hugging Face
🧠 RFT ModelQwen2.5-VL-7B further optimized via GRPO🤗 Hugging Face

⚡️ Simple Usage

STAGE 1. 🧠 Supervised Fine-tuning (SFT)

Step 1. Environment Setup

You can directly follow the instructions in Llama_factory to complete the SFT stage and find more detailed guidance.

# You can setup conda...
cd SFT_training
pip install -e ".[torch,metrics]"

Step 2. Prepare Dataset

Download our SFT Dataset, follow instructions in llama_factory_Data_Readme.md to prepare the dataset (What you mainly need to do is adding our dataset to dataset_info.json)

Step 3. Launch Training

You need config the yaml file before launching

# Lora
llamafactory-cli train examples/train_lora/qwen2vl_lora_sft.yaml

# merge Lora after training
llamafactory-cli export examples/merge_lora/qwen2vl_lora_sft.yaml

# Full Parameters
llamafactory-cli train examples/train_full/qwen2vl_full_sft.yaml

After training, you will get the SFT model for the next RL stage.

STAGE 2. 🎯 Reinforcement Fine-tuning (RFT)

Step 1. Environment Setup

# You can setup conda...
cd RFT training
pip install -e .[vllm]
pip install flash_attn --no-build-isolation

Step 2. Prepare Dataset

Download our RFT Dataset, place the RFT dataset outside the RFT training folder, because the training process of RFT requires packing the folder for communication.

Step 2. Launch RFT Training

Modify RFT.sh before launching

cd RFT training
bash RFT.sh

The rule-based reward function is in \RFT training\examples\scripts\reward_func_embench_alfred_nonlinear.py

The RFT code is modified based on OpenRLHF and MM-Eureka. You can refer to these two repositories for more information about the underlying codebase.

📊 Evaluation on Embench

To ensure fairness, we did not make any modifications to the benchmark. You only need to follow the setup instructions provided in Embench, and then directly evaluate using the models we have trained.

📈 Performance

In-domin Result

Out-of-domin Result

🙏 Acknowledgement

This project builds upon the excellent work of:

✏️ Citation

If you find this repo useful, please cite:

@article{wu2025reinforced,
  title={Reinforced Reasoning for Embodied Planning},
  author={Wu, Di and Fan, Jiaxin and Zang, Junzhe and Wang, Guanbo and Yin, Wei and Li, Wenhao and Jin, Bo},
  journal={arXiv preprint arXiv:2505.22050},
  year={2025}
}

Enjoy reasoning in embodied worlds 🤖🗺️!

```