Bernini

← Back to main README

The full Bernini pipeline combines an MLLM-based semantic planner (built on Qwen2.5-VL) with the Wan2.2 DiT renderer. The planner decomposes complex instructions and plans semantic changes in latent space before diffusion rendering, which gives stronger instruction following on complex generation/editing requests than the renderer-only Bernini-R.

｜ Checkpoint	Renderer base	Planner base	Notes
ByteDance/Bernini-Diffusers	Wan2.2-T2V-A14B	Qwen2.5-VL-7B-Instruct	Cotrain starts with a near-zero initialized connector.
ByteDance/Bernini-Diffusers-v2	Wan2.2-T2V-A14B	Qwen2.5-VL-7B-Instruct	Warm up the connector for thousands of steps before cotraining.

Benchmarks for both models are in the main README.

Download weights

Full Bernini uses the packaged Bernini-Diffusers layout, which collects the Bernini checkpoint, the Qwen2.5-VL planner assets, and the Wan2.2 diffusion components under one directory:

ByteDance/Bernini-Diffusers/
  bernini/                    # Bernini checkpoint
  mllm/                       # semantic planner (Qwen2.5-VL)
  t5_text_encoder/
  t5_tokenizer/
  vae/
  scheduler/
  transformer_config.json     # diffusion decoder configs (no Wan base
  transformer_2_config.json   # transformer weights needed)

Download it from Hugging Face:

pip install -U "huggingface_hub"
hf download ByteDance/Bernini-Diffusers \
    --local-dir ByteDance/Bernini-Diffusers

Pass the directory directly as --config. The run scripts default to ByteDance/Bernini-Diffusers; to use another location, set:

export BERNINI_CONFIG=/path/to/Bernini-Diffusers

Run

Make sure the environment is set up first — see Installation, which includes the required VeOmni install.

The recommended way to run Bernini is through the ready-to-run launch scripts under scripts/bernini/. These scripts wrap the single-GPU image path and the multi-GPU video path with the appropriate default case files, sampling hyperparameters, and Ulysses sequence-parallel settings.

Inputs are described by case files under assets/testcases/; see the case-file format.

Run scripts

scripts/bernini/ provides one ready-to-run script per task, each with the recommended sampling hyperparameters (--guidance_mode vae_txt_vit_wapg, --omega_*, --vit_*, ...) and a default case file. Default output is 480p / 16 fps, 81 frames for video tasks.

bash scripts/bernini/run_t2i.sh    # text-to-image
bash scripts/bernini/run_i2i.sh    # image editing
bash scripts/bernini/run_t2v.sh    # text-to-video
bash scripts/bernini/run_v2v.sh    # video editing
bash scripts/bernini/run_rv2v.sh   # reference + video editing
bash scripts/bernini/run_r2v.sh    # reference-to-video

Each script reads these environment variables:

Variable	Default	Meaning
CASE_PATH	a bundled example case	case JSON to run
BERNINI_CONFIG	ByteDance/Bernini-Diffusers	model directory
NPROC_PER_NODE	8	number of processes
ULYSSES	8	Ulysses sequence-parallel degree

Example override:

CASE_PATH=assets/testcases/v2v/v2v_case2.json \
BERNINI_CONFIG=/path/to/Bernini-Diffusers \
NPROC_PER_NODE=8 ULYSSES=8 \
bash scripts/bernini/run_v2v.sh

Gradio demo

# 8 GPUs, 8-way Ulysses sequence parallel
torchrun --nproc-per-node 8 gradio_demo.py --ulysses 8 \
    --config ByteDance/Bernini-Diffusers --port 7860 --share

# Or the script launcher (honors BERNINI_CONFIG)
bash scripts/bernini/run_gradio.sh

See the Gradio demo notes in the main README for the UI behavior and prompt-enhancer setup.