XSkill: Cross Embodiment Skill Discovery

October 27, 2023 · View on GitHub

1,2Mengda Xu, 1,Zhenjia Xu, 1Cheng Chi, 2,3Manuela Veloso, 1Shuran Song

1Columbia University, 2JP Morgan AI Research,3CMU

CoRL 2023

Project Page|arxiv

This repository contains code for training and evaluating XSkill in both simulation and real-world settings. Teaser Image

🚀 Installation

Follow these steps to install XSkill:

  1. Create and activate the conda environment: 
    cd xskill
conda env create -f environment.yml
conda activate xskill
pip install -e .

📦 Simulation Dataset

To set up the simulation dataset:

  1. Create a new directory for datasets under XSkill: 
    mkdir datasets
cd datasets
wget https://xskill.cs.columbia.edu/data/kitchen.zip
unzip kitchen.zip
  2. Set the base_dev_dir config/simulation/create_kitchen_datase.yaml to your working directory. Run the following command to generate the cross-embodiment kitchen data and the training mask: 
    cd scripts
python create_all_kitchen_dataset.py
python extract_kitchen_info.py

🌐 Real World Dataset

To Download the real world kitchen dataset:

mkdir datasets
cd datasets
wget https://xskill.cs.columbia.edu/data/real_kitchen_data.zip

🚴‍♂️ Training

Simulation

  1. Run the skill discovery script: 
    python scripts/skill_discovery.py
  2. Label the dataset using the learned prototype by the trained model. 
    python scripts/label_sim_kitchen_dataset.py
  3. Execute the skill transfer and composing script. Replace the pretrain_path and pretrain_ckpt in cfg/simulation/skill_transfer_composing.yaml 
    python scripts/skill_transfer_composing.py

Real World

  1. Execute the real-world skill discovery script:

    python scripts/realworld/skill_discovery.py

  2. Label the real-world dataset:

    python scripts/realworld/label_real_kitchen_dataset.py

  3. 📊 Visualization

    Open the provided Jupyter notebook viz_real.ipynb to visualize the learned prototypes:

BibTeX

@inproceedings{
     xu2023xskill,
     title={{XS}kill: Cross Embodiment Skill Discovery},
     author={Mengda Xu and Zhenjia Xu and Cheng Chi and  Manuela Veloso and Shuran Song},
     booktitle={7th Annual Conference on Robot Learning},
     year={2023},
     url={https://openreview.net/forum?id=8L6pHd9aS6w}
     }

License

This repository is released under the MIT license.

Acknowledgement