VLMPC & Traj-VLMPC

This is the official repo for our papers:

1. VLMPC: Vision-Language Model Predictive Control for Robotic Manipulation, RSS2024

    Wentao Zhao*, Jiaming Chen*, Ziyu Meng, Donghui Mao, Ran Song†, Wei Zhang

2. Vision-Language Model Predictive Control for Manipulation Planning and Trajectory Generation

    Jiaming Chen*, Wentao Zhao*, ,Ziyu Meng, Donghui Mao, Ran Song†, Wei Pan , Wei Zhang

We provide the implementation of VLMPC in Language-Table environment.

Installation

conda create -n vlmpc python=3.10
conda activate vlmpc

pip install -r requirements.txt

Note: Add your Openai API key to the vlmpc.py.

Quickstart

We provide the trained checkpoints of video prediction model and detector, download them for quick start:

python main.py --checkpoint_file path/to/video_prediction_model/checkpoint --task push_corner --zoom 0.03 --num_samples 50 --plan_freq 3 --det_path path/to/detector/checkpoint

Citation

@inproceedings{zhao2024vlmpc,
    title={VLMPC: Vision-Language Model Predictive Control for Robotic Manipulation},
    author={Zhao, Wentao and Chen, Jiaming and Meng, Ziyu and Mao, Donghui and Song, Ran and Zhang, Wei},
    booktitle={Robotics: Science and Systems},
    year={2024},
    }

Acknowledgements

• Environment is based on Language-Table.
• The implementation of DMVFN-Act video prediction model is based on DMVFN.
• PySOT for lightweight visual tracking.