Shuai Zhen¹, Yanhua Yu^1✉, Ruopei Guo², Nan Cheng², Yang Deng³,

¹Beijing University of Posts and Telecommunications ²China Mobile Group Design Institute ³Singapore Management University

Overview

(a) The pipeline of STEP-HRL: Local progress policy is responsible for producing a compact summary of local interaction history within each subtask. Specifically, the local progress policy $\pi^p_\theta$ depends on previous progress $p^k_{t-1}$, current subtask $g_k$, executed action $a^k_{t-1}$ and the resulting observation $o^k_t$ to the generate updated local progress $p^k_t$. The low-level policy $\pi^l_\theta$ combines $p^k_t$ with observation $o^k_t$ and subtask $g_k$ to generate primitive actions. When current subtask $g_k$ terminates, its final local progress $\hat{p}_k$ is forwarded to the high-level policy $\pi^h_\theta$. Conditioned on the task instruction $c_i$, completed subtasks $G_k$, final local progress $\hat{p}_k$ and the initial observation $o^{k+1}_0$ of next subtask, $\pi^h_\theta$ generates the subsequent subtask.
(b) The structure of our model: Three different policies share the same parameters, but equipped with different critic network respectively for offline RL training.

Prerequisites ⚙️

Virtual Environment

conda create -n step-hrl python=3.10 -y
conda activate step-hrl
pip install -r requirements.txt

Benchmark Setup

STEP-HRL is evaluated on two benchmarks: ScienceWorld and AlfWorld. Please follow the official installation instructions for each environment.

Experiments 🔬

Dataset Preperation

The processed dataset can be downloaded from Google Drive, then place it in the root directory, e.g. ./dataset/alfworld and ./dataset/scienceworld. Then run following command to get training data.

python env/scienceworld/convert_data_memory.py --half -1
python env/alfworld/convert_data_memory.py --half -1

Behavioral Cloning

Run BC training with the following script with corresponding config in ./config/step_bc.yaml:

deepspeed --num_gpus 8 train_bc.py

Reinforcement Learning

*For AlfWorld, the improvement is very limited since the tasks are relatively simple. Behavior cloning (BC) alone can already achieve scores around 97.

Run BC training with the following script with corresponding config in ./config/step_collection.yaml:

deepspeed --num_gpus 8 step_data_collection.py

Then find corresponding file in dataset/scienceworld/collect_data and perform:

python env/scienceworld/convert_data_memory.py --half -1 -f <file_path>

Then run ORL training with the following script with corresponding config in ./config/step_rl.yaml:

deepspeed --num_gpus 8 train_rl.py

Evaluation

Set evaluation setting in ./config/eval_step.yaml , then run:

deepspeed --num_gpus 8 eval_step.py

Acknowledgements

This repository builds upon the implementation of GLIDER. We thank the authors for their excellent work and releasing their code.