Video Diffusion Models are Training-free Motion Interpreter and Controller

Zeqi Xiao  Yifan Zhou  Shuai Yang  Xingang Pan 

Installation

Install the environments by

conda create -n moft python==3.8
conda activate moft
pip install -r requirements.txt

Downloads checkpoints from Animatediff, LoRA, and SD-1.5. Put them into the following structures:

models/
├── DreamBooth_LoRA
│   ├── realisticVisionV20_v20.safetensors
├── Motion_Module
│   ├── mm_sd_v15_v2.ckpt
├── stable-diffusion-v1-5

Run process.ipynb

🔗 Citation

If you find our work helpful, please cite:

@inproceedings{
xiao2024video,
title={Video Diffusion Models are Training-free Motion Interpreter and Controller},
author={Zeqi Xiao and Yifan Zhou and Shuai Yang and Xingang Pan},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=ZvQ4Bn75kN}
}