VideoTuna

🤗🤗🤗 Videotuna is a useful codebase for text-to-video applications.
🌟 VideoTuna is the first repo that integrates multiple AI video generation models including text-to-video (T2V), image-to-video (I2V), text-to-image (T2I), and video-to-video (V2V) generation for model inference and finetuning (to the best of our knowledge).
🌟 VideoTuna is the first repo that provides comprehensive pipelines in video generation, from fine-tuning to pre-training, continuous training, and post-training (alignment) (to the best of our knowledge).

🔆 Features

🌟 All-in-one framework: Inference and fine-tune various up-to-date pre-trained video generation models.
🌟 Continuous training: Keep improving your model with new data.
🌟 Fine-tuning: Adapt pre-trained models to specific domains.
🌟 Human preference alignment: Leverage RLHF to align with human preferences.
🌟 Post-processing: Enhance and rectify the videos with video-to-video enhancement model.

🔆 Updates

• [2025-04-22] 🐟 Supported inference for Wan2.1 and Step Video and fine-tuning for HunyuanVideo T2V, with a unified codebase architecture.
• [2025-02-03] 🐟 Supported automatic code formatting via PR#27. Thanks @samidarko!
• [2025-02-01] 🐟 Migrated to Poetry for streamlined dependency and script management (PR#25). Thanks @samidarko!
• [2025-01-20] 🐟 Supported fine-tuning for Flux-T2I.
• [2025-01-01] 🐟 Released training for VideoVAE+ in the VideoVAEPlus repo.
• [2025-01-01] 🐟 Supported inference for Hunyuan Video and Mochi.
• [2024-12-24] 🐟 Released VideoVAE+: a SOTA Video VAE model—now available in this repo! Achieves better video reconstruction than NVIDIA’s Cosmos-Tokenizer.
• [2024-12-01] 🐟 Supported inference for CogVideoX-1.5-T2V&I2V and Video-to-Video Enhancement from ModelScope.
• [2024-12-01] 🐟 Supported fine-tuning for CogVideoX.
• [2024-11-01] 🐟 🎉 Released VideoTuna v0.1.0!
  Initial support includes inference for VideoCrafter1-T2V&I2V, VideoCrafter2-T2V, DynamiCrafter-I2V, OpenSora-T2V, CogVideoX-1-2B-T2V, CogVideoX-1-T2V, Flux-T2I, and training/fine-tuning of VideoCrafter, DynamiCrafter, and Open-Sora.

🔆 Get started

1.Prepare environment

(1) If you use Linux and Conda (Recommend)

conda create -n videotuna python=3.10 -y
conda activate videotuna
pip install poetry
poetry install

• ↑ It takes around 3 minitues.

Optional: Flash-attn installation

Hunyuan model uses it to reduce memory usage and speed up inference. If it is not installed, the model will run in normal mode. Install the flash-attn via:

poetry run install-flash-attn 

• ↑ It takes 1 minitue.

Optional: Video-to-video enhancement

poetry run pip install "modelscope[cv]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

• If this command ↑ get stucked, kill and re-run it will solve the issue.

(2) If you use Linux and Poetry (without Conda):

poetry config virtualenvs.in-project true # optional but recommended, will ensure the virtual env is created in the project root
poetry config virtualenvs.create true # enable this argument to ensure the virtual env is created in the project root
poetry env use python3.10 # will create the virtual env, check with `ls -l .venv`.
poetry env activate # optional because Poetry commands (e.g. `poetry install` or `poetry run <command>`) will always automatically load the virtual env.
poetry install

poetry run install-flash-attn

poetry run pip install "modelscope[cv]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

(3) If you use MacOS

docker compose build videotuna

export HOST_UID=$(id -u)
export HOST_GID=$(id -g)

docker compose run --remove-orphans videotuna poetry env use /usr/local/bin/python
docker compose run --remove-orphans videotuna poetry run python -m pip install --upgrade pip setuptools wheel
docker compose run --remove-orphans videotuna poetry install
docker compose run --remove-orphans videotuna poetry run pip install "modelscope[cv]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

docker compose run --remove-orphans videotuna poetry add wheel

docker compose run --remove-orphans videotuna poetry run pip freeze

docker compose run --remove-orphans videotuna poetry run format

docker compose run -it --remove-orphans videotuna bash

2.Prepare checkpoints

• Please follow docs/checkpoints.md to download model checkpoints.
• After downloading, the model checkpoints should be placed as Checkpoint Structure.

3.Inference state-of-the-art T2V/I2V/T2I models

Run the following commands to inference models: It will automatically perform T2V/T2I based on prompts in inputs/t2v/prompts.txt, and I2V based on images and prompts in inputs/i2v/576x1024.

T2V

I2V

T2I

4. Finetune T2V models

(1) Prepare dataset

Please follow the docs/datasets.md to try provided toydataset or build your own datasets.

(2) Fine-tune

All training commands were tested on H800 80G GPUs.
T2V

I2V

T2I

5. Evaluation

We support VBench evaluation to evaluate the T2V generation performance. Please check eval/README.md for details.

Contribute

Git hooks

Git hooks are handled with pre-commit library.

Hooks installation

Run the following command to install hooks on commit. They will check formatting, linting and types.

poetry run pre-commit install
poetry run pre-commit install --hook-type commit-msg

Running the hooks without commiting

poetry run pre-commit run --all-files

Acknowledgement

We thank the following repos for sharing their awesome models and codes!

• Wan2.1: Wan: Open and Advanced Large-Scale Video Generative Models.
• HunyuanVideo: A Systematic Framework For Large Video Generation Model.
• Step-Video: A text-to-video pre-trained model with 30 billion parameters and the capability to generate videos up to 204 frames.
• Mochi: A new SOTA in open-source video generation models
• VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models
• VideoCrafter1: Open Diffusion Models for High-Quality Video Generation
• DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors
• Open-Sora: Democratizing Efficient Video Production for All
• CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
• VADER: Video Diffusion Alignment via Reward Gradients
• VBench: Comprehensive Benchmark Suite for Video Generative Models
• Flux: Text-to-image models from Black Forest Labs.
• SimpleTuner: A fine-tuning kit for text-to-image generation.

Some Resources

• LLMs-Meet-MM-Generation: A paper collection of utilizing LLMs for multimodal generation (image, video, 3D and audio).
• MMTrail: A multimodal trailer video dataset with language and music descriptions.
• Seeing-and-Hearing: A versatile framework for Joint VA generation, V2A, A2V, and I2A.
• Self-Cascade: A Self-Cascade model for higher-resolution image and video generation.
• ScaleCrafter and HiPrompt: Free method for higher-resolution image and video generation.
• FreeTraj and FreeNoise: Free method for video trajectory control and longer-video generation.
• Follow-Your-Emoji, Follow-Your-Click, and Follow-Your-Pose: Follow family for controllable video generation.
• Animate-A-Story: A framework for storytelling video generation.
• LVDM: Latent Video Diffusion Model for long video generation and text-to-video generation.

🍻 Contributors

📋 License

Please follow CC-BY-NC-ND. If you want a license authorization, please contact the project leads Yingqing He (yhebm@connect.ust.hk) and Yazhou Xing (yxingag@connect.ust.hk).

😊 Citation

@software{videotuna,
  author = {Yingqing He and Yazhou Xing and Zhefan Rao and Haoyu Wu and Zhaoyang Liu and Jingye Chen and Pengjun Fang and Jiajun Li and Liya Ji and Runtao Liu and Xiaowei Chi and Yang Fei and Guocheng Shao and Yue Ma and Qifeng Chen},
  title = {VideoTuna: A Powerful Toolkit for Video Generation with Model Fine-Tuning and Post-Training},
  month = {Nov},
  year = {2024},
  url = {https://github.com/VideoVerses/VideoTuna}
}

Star History