Turbo4DGen: Ultra-Fast Acceleration for 4D Generation (ICML 2026)

May 22, 2026 · View on GitHub

Yuanbin Man, Ying Huang, Zhile Ren, Miao Yin

Turbo4DGen accelerates diffusion-based multi-view 4D generation by exploiting redundancy in the spatio-camera-motion (SCM) attention chain at three granularities (block, token, and chain) through (1) a rolling cache that reuses block-level attention outputs across denoising steps, (2) semantic-aware token pruning guided by spatial cross-attention, and (3) a V_ASR-driven adaptive scheduler that bypasses the intermediate SCM chains when features become temporally stable.

Turbo4DGen teaser

Method

Drop-in replacement for the SV4D denoising UNet — same checkpoint, same weight shapes, baseline or accelerated.

Block attention reuse. RollingCache keys per-layer (spatial/camera/motion) attention outputs by {stage}_{layer_id}, plus one block-level slot for the post-decoder feature. Caching/pruning steps read them back so camera and motion attention skip QKV recompute.
Semantic-aware token pruning. The spatial cross-attention map prunes camera/motion Q and K; pruned slots are refilled from the cache (not zeros) to preserve spatiotemporal coherence. Ratio: spa_pruning_ratio (default 0.2).
Adaptive SCM chain bypass. Per-step cosine similarity between fresh and cached attention is averaged into V_ASR (Eq. 12). Once V_ASR > α, the intermediate UNet is skipped — only initial conv and the final-resolution SCM block run, with the cached block-level feature carrying the deep contribution.

Setup

conda env create -f turbo4d.yml
conda activate turbo4d
pip install -r requirements/pt2.txt

Download the SV4D and SV3D checkpoints into checkpoints/:

sv4d.safetensors from https://huggingface.co/stabilityai/sv4d
sv3d_u.safetensors from https://huggingface.co/stabilityai/sv3d
sv3d_p.safetensors (optional, for --sv3d_version sv3d_p)

Inference

Accelerated (Turbo4DGen):

python scripts/sampling/simple_video_sample_4d.py \
  --input_path assets/sv4d_videos/test_video2.mp4 \
  --output_folder outputs/turbo4d \
  --num_steps 20

Baseline (vanilla SV4D, for A/B comparison):

python scripts/sampling/simple_video_sample_4d.py \
  --input_path assets/sv4d_videos/test_video2.mp4 \
  --output_folder outputs/baseline \
  --model_config scripts/sampling/configs/sv4d_baseline.yaml

Training (Bypass-Aware Fine-Tuning)

To recover paper-level quality at the most aggressive bypass setting, fine-tune SV4D so the model is robust to chain bypass during inference. We add a two-step rollout to the EDM loss:

With probability --bypass_prob, run a no-grad dense forward at a higher σ to populate the rolling cache.
Run the graded forward at the target σ. The UNet's bypass guard fires and gradients flow through the bypass branch, teaching the model to compose stale cached features with fresh residuals.

python trainer.py \
  --dataset /path/to/objaverse_dy_4d \
  --cond_dir /path/to/cond \
  --bypass_aware --bypass_prob 0.5 \
  --epochs 4 --batch 1 --lr 1e-5 \
  --outdir checkpoints/bypass_aware

Generalization Beyond SV4D

Because the three accelerators target the SCM attention chain rather than any UNet-specific module, they apply equally to DiT-based 4D generators. We additionally port Turbo4DGen to TrajectoryCrafter (DiT); qualitative comparisons appear on the project page.

Citation

@inproceedings{man2026turbo4dgen,
  title     = {Turbo4DGen: Ultra-Fast Acceleration for 4D Generation},
  author    = {Man, Yuanbin and Huang, Ying and Ren, Zhile and Yin, Miao},
  booktitle = {Forty-third International Conference on Machine Learning},
  year      = {2026}
}

Acknowledgments

This codebase is built on top of Stable Video 4D (SV4D) by Stability AI (Xie et al., 2025). We thank the authors for releasing the SV4D weights and reference implementation; our accelerator is a drop-in replacement of the SV4D denoising UNet that preserves their checkpoint format and conditioning pipeline.

We also gratefully use SV3D (Voleti et al., 2024) for anchor-view generation, and the ObjaverseDy dataset (Deitke et al., 2022) for fine-tuning.

License

Code is released under the same terms as the upstream SV4D repository (LICENSE-CODE). SV4D model weights are subject to Stability AI's research license.