GVF-TAPE -- Video Generation Models

October 22, 2025 · View on GitHub

Generative Visual Foresight Meets Task-Agnostic Pose Estimation in Robotic Table-Top Manipulation [CoRL 2025]

[project page] [Arxiv]

Chuye Zhang1 Xiaoxiong Zhang1 Wei Pan1 Linfang Zheng†2,3 Wei Zhang†1,2

1Southern University of Science and Technology, 2LimX Dynamics, 3The University of HongKong

This codebase is official implementation of GVF-TAPE, it contains codes to train a flow-based video generation model.

Setting Environment

You can set up the environment with conda

git clone https://github.com/Xiaoxiongzzzz/GVF-TAPE-Video-Generation.git
cd GVF-TAPE-Video-Generation
conda env create -f environment.yml
conda activate videogenerator

Dataset Structure

The pytorch dataset class is defined in ./flowdiffusion/dataset.py.

This class is based on LIBERO dataset with some modification:

  • Agentview_rgb in LIBERO dataset is upside down, you need to invert it.
  • If you need generate RGB-D videos, then augmenting this dataset with depth information is needed. It's can be achieved by using VideoDepthAnything or other model.
  • Preprocess the task description to text embedding and save it to dataset. */flowdiffusion/preprocess_dataset.py script could do this.

You can also write your dataset class based on our implementation.

Training

You can specify the dataset path and output path in */flowdiffusion/train_rectified_flow.py and train easily:

accelerate launch flowdiffusion/train_rectified_flow.py

In our experiments, sample step = 3 is enough for LIBERO.

Ackownledgement

This codebase is based on AVDC, thanks for their excellent code.

Contact Xiaoxiong Zhang if you have any questions and suggestions.

Other parts of our codes will be released soon.