LTX Video 2 Quickstart

{
  "enable_group_offload": true,
  "group_offload_type": "block_level",
  "group_offload_blocks_per_group": 1,
  "group_offload_use_stream": true
}

SimpleTuner includes experimental features that can significantly improve training stability and performance.

• Scheduled Sampling (Rollout): reduces exposure bias and improves output quality by letting the model generate its own inputs during training.

⚠️ These features increase the computational overhead of training.

Dataset considerations

Video datasets require careful setup. Create config/multidatabackend.json:

[
  {
    "id": "my-video-dataset",
    "type": "local",
    "dataset_type": "video",
    "instance_data_dir": "datasets/videos",
    "caption_strategy": "textfile",
    "resolution": 512,
    "video": {
        "num_frames": 61,
        "min_frames": 61,
        "frame_rate": 25,
        "bucket_strategy": "aspect_ratio"
    },
    "repeats": 10
  },
  {
    "id": "text-embeds",
    "type": "local",
    "dataset_type": "text_embeds",
    "default": true,
    "cache_dir": "cache/text/ltxvideo2",
    "disabled": false
  }
]

In the video subsection:

• num_frames: Target frame count for training.
• min_frames: Minimum video length (shorter videos are discarded).
• max_frames: Maximum video length filter.
• bucket_strategy: How videos are grouped into buckets:
  • aspect_ratio (default): Group by spatial aspect ratio only.
  • resolution_frames: Group by WxH@F format (e.g., 1920x1080@61) for mixed-resolution/duration datasets.
• frame_interval: When using resolution_frames, round frame counts to this interval.

LTX-2 supports video-only training without audio. To enable audio training, add an audio block to your video dataset configuration:

"audio": {
    "auto_split": true,
    "sample_rate": 16000,
    "channels": 1,
    "duration_interval": 3.0,
    "allow_zero_audio": false
}

When the audio section is present, SimpleTuner auto-generates an audio dataset from your video files and caches audio latents alongside video latents. Set audio.allow_zero_audio: true if your videos lack audio streams. Without an audio section, LTX-2 trains on video only and masks audio loss automatically.

See caption_strategy options and requirements in DATALOADER.md.

Directory setup

mkdir -p datasets/videos
</details>

# Place .mp4 / .mov files here.
# Place corresponding .txt files with same filename for captions.

Login

wandb login
huggingface-cli login

Executing the training

simpletuner train

Notes & troubleshooting tips

Out of Memory (OOM)

Video training is extremely demanding. If you OOM:

1. Reduce Resolution: Try 480p (480x854 or similar).
2. Reduce Frames: Drop validation_num_video_frames and dataset num_frames to 33 or 49.
3. Check Offload: Ensure --enable_group_offload is active.

Validation Video Quality

• Black/Noise Videos: Often caused by validation_guidance being too high (> 6.0) or too low (< 2.0). Stick to 5.0.
• Motion Jitter: Check if your dataset frame rate matches the model's trained frame rate (often 25fps).
• Stagnant/Static Video: The model might be undertrained or the prompt isn't describing motion. Use prompts like "camera pans right", "zoom in", "running", etc.

TREAD training

TREAD works for video too and is highly recommended to save compute.

Add to config.json:

{
  "tread_config": {
    "routes": [
      {
        "selection_ratio": 0.5,
        "start_layer_idx": 2,
        "end_layer_idx": -2
      }
    ]
  }
}

This can speed up training by ~25-40% depending on the ratio.

Lowest VRAM use config (7900XTX)

Field-tested config that prioritizes minimal VRAM usage on LTX Video 2.

{
  "base_model_precision": "int8-quanto",
  "checkpoint_step_interval": 100,
  "data_backend_config": "config/ltx2/multidatabackend.json",
  "disable_benchmark": true,
  "dynamo_mode": "",
  "evaluation_type": "none",
  "hub_model_id": "simpletuner-ltxvideo2-19b-t2v-lora-test",
  "learning_rate": 0.00006,
  "lr_warmup_steps": 50,
  "lycoris_config": "config/lycoris_config.json",
  "max_grad_norm": 0.1,
  "max_train_steps": 200,
  "minimum_image_size": 0,
  "model_family": "ltxvideo2",
  "model_flavour": "dev",
  "model_type": "lora",
  "num_train_epochs": 0,
  "offload_during_startup": true,
  "optimizer": "adamw_bf16",
  "output_dir": "output/examples/ltxvideo2-19b-t2v.peft-lora",
  "override_dataset_config": true,
  "ramtorch": true,
  "ramtorch_text_encoder": true,
  "report_to": "none",
  "resolution": 480,
  "scheduled_sampling_reflexflow": false,
  "seed": 42,
  "skip_file_discovery": "",
  "tracker_project_name": "lora-training",
  "tracker_run_name": "example-training-run",
  "train_batch_size": 2,
  "vae_batch_size": 1,
  "vae_enable_patch_conv": true,
  "vae_enable_slicing": true,
  "vae_enable_temporal_roll": true,
  "vae_enable_tiling": true,
  "validation_disable": true,
  "validation_disable_unconditional": true,
  "validation_guidance": 5,
  "validation_num_inference_steps": 40,
  "validation_num_video_frames": 81,
  "validation_prompt": "🟫 is holding a sign that says hello world from ltxvideo2",
  "validation_resolution": "768x512",
  "validation_seed": 42,
  "validation_using_datasets": false
}

Audio-Only Training

LTX-2 supports audio-only training where you train only the audio generation capability without video files. This is useful when you have audio datasets but no corresponding video content.

In audio-only mode:

• Video latents are automatically zeroed out (minimal 64x64 resolution to save memory)
• Video loss is masked (not computed)
• Only audio generation layers are trained

Audio-only mode is automatically detected when your dataset configuration contains only audio datasets (no video or image datasets). You can also explicitly enable it with audio.audio_only: true.

Audio-only dataset configuration

[
  {
    "id": "my-audio-dataset",
    "type": "local",
    "dataset_type": "audio",
    "instance_data_dir": "datasets/audio",
    "caption_strategy": "textfile",
    "audio": {
      "sample_rate": 16000,
      "channels": 2,
      "duration_interval": 3.0,
      "truncation_mode": "beginning"
    },
    "repeats": 10
  },
  {
    "id": "text-embeds",
    "type": "local",
    "dataset_type": "text_embeds",
    "default": true,
    "cache_dir": "cache/text/ltxvideo2",
    "disabled": false
  }
]

Key audio settings:

• channels: Must be 2 (stereo) for the LTX-2 audio VAE
• duration_interval: Bucket audio into intervals (e.g., 3.0 seconds). Important for memory management - long audio files create many video frames even though they're zeros
• truncation_mode: How to handle audio longer than the bucket duration (beginning, end, or random)

Supported audio formats

SimpleTuner supports common audio formats (.wav, .flac, .mp3, .ogg, .opus, etc.) as well as container formats that may contain audio-only content (.mp4, .mpeg, .mkv, .webm). Container formats are automatically extracted using ffmpeg.

LoRA targets for audio training

When audio data is detected in your datasets, SimpleTuner automatically adds audio-specific modules to the LoRA targets:

• audio_proj_in - Audio input projection
• audio_proj_out - Audio output projection
• audio_caption_projection.linear_1 - Audio caption projection layer 1
• audio_caption_projection.linear_2 - Audio caption projection layer 2

This happens automatically for both audio-only training and joint audio+video training.

If you want to override the LoRA targets manually, use --peft_lora_target_modules with a JSON list of module names.

Place your audio files in the instance_data_dir with corresponding .txt caption files.

Validation workflows (T2V vs I2V)

• T2V (text-to-video): Leave validation_using_datasets: false and use validation_prompt or validation_prompt_library.
• I2V (image-to-video): Set validation_using_datasets: true and point eval_dataset_id at a validation split that provides a reference image. Validation will switch to the image-to-video pipeline and use that image as the conditioner.
• S2V (audio-conditioned): With validation_using_datasets: true, point eval_dataset_id at a dataset with s2v_datasets (or the default audio.auto_split behavior). Validation will load cached audio latents automatically.
• Spatial upscale validation: Set ltx2_validation_pipeline_mode: "spatial-upscale" to validate with the LTX-2 spatial upscaler pass. Stage 1 runs at half resolution, the upsampler doubles video latents, and stage 2 re-denoises at the requested resolution. validation_audio_only stays on the single-stage path.

Validation adapters (LoRAs)

Lightricks provides several LoRAs that can be applied during validation via validation_adapter_path (single) or validation_adapter_config (multiple runs). These repos use nonstandard weight filenames, so include the filename via repo_id:weight_name. See the LTX-2 collection for the latest filenames and related assets: https://huggingface.co/collections/Lightricks/ltx-2

• Lightricks/LTX-2-19b-IC-LoRA-Canny-Control:ltx-2-19b-ic-lora-canny-control.safetensors
• Lightricks/LTX-2-19b-IC-LoRA-Depth-Control:ltx-2-19b-ic-lora-depth-control.safetensors
• Lightricks/LTX-2-19b-IC-LoRA-Detailer:ltx-2-19b-ic-lora-detailer.safetensors
• Lightricks/LTX-2-19b-IC-LoRA-Pose-Control:ltx-2-19b-ic-lora-pose-control.safetensors
• Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-In:ltx-2-19b-lora-camera-control-dolly-in.safetensors
• Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-Out:ltx-2-19b-lora-camera-control-dolly-out.safetensors
• Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-Left:ltx-2-19b-lora-camera-control-dolly-left.safetensors
• Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-Right:ltx-2-19b-lora-camera-control-dolly-right.safetensors
• Lightricks/LTX-2-19b-LoRA-Camera-Control-Jib-Down:ltx-2-19b-lora-camera-control-jib-down.safetensors
• Lightricks/LTX-2-19b-LoRA-Camera-Control-Jib-Up:ltx-2-19b-lora-camera-control-jib-up.safetensors
• Lightricks/LTX-2-19b-LoRA-Camera-Control-Static:ltx-2-19b-lora-camera-control-static.safetensors

Example validation_adapter_config:

{
  "validation_adapter_config": [
    { "label": "canny", "path": "Lightricks/LTX-2-19b-IC-LoRA-Canny-Control:ltx-2-19b-ic-lora-canny-control.safetensors" },
    { "label": "pose", "path": "Lightricks/LTX-2-19b-IC-LoRA-Pose-Control:ltx-2-19b-ic-lora-pose-control.safetensors" }
  ]
}

For faster validation, apply Lightricks/LTX-2-19b-distilled-lora-384:ltx-2-19b-distilled-lora-384.safetensors as a validation adapter and set validation_guidance: 1 plus validation_num_inference_steps: 8.