LoRA Training Guide for Anima using animatrainnetwork.py / animatrainnetwork.py を用いたAnima モデルのLoRA学習ガイド

This document explains how to train LoRA (Low-Rank Adaptation) models for Anima using anima_train_network.py in the sd-scripts repository.

1. Introduction / はじめに

anima_train_network.py trains additional networks such as LoRA for Anima models. Anima adopts a DiT (Diffusion Transformer) architecture based on the MiniTrainDIT design with Rectified Flow training. It uses a Qwen3-0.6B text encoder, an LLM Adapter (6-layer transformer bridge from Qwen3 to T5-compatible space), and a Qwen-Image VAE (16-channel, 8x spatial downscale).

Qwen-Image VAE and Qwen-Image VAE have same architecture, but official Anima weight is named for Qwen-Image VAE.

This guide assumes you already understand the basics of LoRA training. For common usage and options, see the train_network.py guide. Some parameters are similar to those in sd3_train_network.py and flux_train_network.py.

Prerequisites:

• The sd-scripts repository has been cloned and the Python environment is ready.
• A training dataset has been prepared. See the Dataset Configuration Guide.
• Anima model files for training are available.

2. Differences from train_network.py / train_network.py との違い

anima_train_network.py is based on train_network.py but modified for Anima. Main differences are:

• Target models: Anima DiT models.
• Model structure: Uses a MiniTrainDIT (Transformer based) instead of U-Net. Employs a single text encoder (Qwen3-0.6B), an LLM Adapter that bridges Qwen3 embeddings to T5-compatible cross-attention space, and a Qwen-Image VAE (16-channel latent space with 8x spatial downscale).
• Arguments: Uses the common --pretrained_model_name_or_path for the DiT model path, --qwen3 for the Qwen3 text encoder, and --vae for the Qwen-Image VAE. The LLM adapter and T5 tokenizer can be specified separately with --llm_adapter_path and --t5_tokenizer_path.
• Incompatible arguments: Stable Diffusion v1/v2 options such as --v2, --v_parameterization and --clip_skip are not used. --fp8_base is not supported.
• Timestep sampling: Uses the same --timestep_sampling options as FLUX training (sigma, uniform, sigmoid, shift, flux_shift).
• LoRA: Uses regex-based module selection and per-module rank/learning rate control (network_reg_dims, network_reg_lrs) instead of per-component arguments. Module exclusion/inclusion is controlled by exclude_patterns and include_patterns.

3. Preparation / 準備

The following files are required before starting training:

1. Training script: anima_train_network.py
2. Anima DiT model file: .safetensors file for the base DiT model.
3. Qwen3-0.6B text encoder: Either a HuggingFace model directory, or a single .safetensors file (uses the bundled config files in configs/qwen3_06b/).
4. Qwen-Image VAE model file: .safetensors or .pth file for the VAE.
5. LLM Adapter model file (optional): .safetensors file. If not provided separately, the adapter is loaded from the DiT file if the key llm_adapter.out_proj.weight exists.
6. T5 Tokenizer (optional): If not specified, uses the bundled tokenizer at configs/t5_old/.
7. Dataset definition file (.toml): Dataset settings in TOML format. (See the Dataset Configuration Guide.) In this document we use my_anima_dataset_config.toml as an example.

Model files can be obtained from the Anima HuggingFace repository.

Notes:

• The T5 tokenizer only needs the tokenizer files (not the T5 model weights). It uses the vocabulary from google/t5-v1_1-xxl.

4. Running the Training / 学習の実行

Execute anima_train_network.py from the terminal to start training. The overall command-line format is the same as train_network.py, but Anima specific options must be supplied.

Example command:

accelerate launch --num_cpu_threads_per_process 1 anima_train_network.py \
  --pretrained_model_name_or_path="<path to Anima DiT model>" \
  --qwen3="<path to Qwen3-0.6B model or directory>" \
  --vae="<path to Qwen-Image VAE model>" \
  --dataset_config="my_anima_dataset_config.toml" \
  --output_dir="<output directory>" \
  --output_name="my_anima_lora" \
  --save_model_as=safetensors \
  --network_module=networks.lora_anima \
  --network_dim=8 \
  --learning_rate=1e-4 \
  --optimizer_type="AdamW8bit" \
  --lr_scheduler="constant" \
  --timestep_sampling="sigmoid" \
  --discrete_flow_shift=1.0 \
  --max_train_epochs=10 \
  --save_every_n_epochs=1 \
  --mixed_precision="bf16" \
  --gradient_checkpointing \
  --cache_latents \
  --cache_text_encoder_outputs \
  --vae_chunk_size=64 \
  --vae_disable_cache

(Write the command on one line or use \ or ^ for line breaks.)

The learning rate of 1e-4 is just an example. Adjust it according to your dataset and objectives. This value is for alpha=1.0 (default). If increasing --network_alpha, consider lowering the learning rate.

If loss becomes NaN, ensure you are using PyTorch version 2.5 or higher.

Note: --vae_chunk_size, --vae_disable_cache, and --qwen_image_vae_2d are custom options in this repository to reduce memory usage (and, for --qwen_image_vae_2d, also speed up) the Qwen-Image VAE.

4.1. Explanation of Key Options / 主要なコマンドライン引数の解説

Besides the arguments explained in the train_network.py guide, specify the following Anima specific options. For shared options (--output_dir, --output_name, --network_module, etc.), see that guide.

Model Options [Required] / モデル関連 [必須]

• --pretrained_model_name_or_path="<path to Anima DiT model>" [Required]
  • Path to the Anima DiT model .safetensors file. The model config (channels, blocks, heads) is auto-detected from the state dict. ComfyUI format with net. prefix is supported.
• --qwen3="<path to Qwen3-0.6B model>" [Required]
  • Path to the Qwen3-0.6B text encoder. Can be a HuggingFace model directory or a single .safetensors file. The text encoder is always frozen during training.
• --vae="<path to Qwen-Image VAE model>" [Required]
  • Path to the Qwen-Image VAE model .safetensors or .pth file. Fixed config: dim=96, z_dim=16.

Model Options [Optional] / モデル関連 [オプション]

• --llm_adapter_path="<path to LLM adapter>" [Optional]
  • Path to a separate LLM adapter weights file. If omitted, the adapter is loaded from the DiT file when the key llm_adapter.out_proj.weight exists.
• --t5_tokenizer_path="<path to T5 tokenizer>" [Optional]
  • Path to the T5 tokenizer directory. If omitted, uses the bundled config at configs/t5_old/.

Anima Training Parameters / Anima 学習パラメータ

• --timestep_sampling=<choice>
  • Timestep sampling method. Choose from sigma, uniform, sigmoid (default), shift, flux_shift. Same options as FLUX training. See the flux_train_network.py guide for details on each method.
• --discrete_flow_shift=<float>
  • Shift for the timestep distribution in Rectified Flow training. Default 1.0. This value is used when --timestep_sampling is set to shift. The shift formula is t_shifted = (t * shift) / (1 + (shift - 1) * t).
• --sigmoid_scale=<float>
  • Scale factor when --timestep_sampling is set to sigmoid, shift, or flux_shift. Default 1.0.
• --qwen3_max_token_length=<integer>
  • Maximum token length for the Qwen3 tokenizer. Default 512.
• --t5_max_token_length=<integer>
  • Maximum token length for the T5 tokenizer. Default 512.
• --attn_mode=<choice>
  • Attention implementation to use. Choose from torch (default), xformers, flash, sageattn. xformers requires --split_attn. sageattn does not support training (inference only). This option overrides --xformers.
• --split_attn
  • Split attention computation to reduce memory usage. Required when using --attn_mode xformers.

Component-wise Learning Rates / コンポーネント別学習率

These options set separate learning rates for each component of the Anima model. They are primarily used for full fine-tuning. Set to 0 to freeze a component:

• --self_attn_lr=<float> - Learning rate for self-attention layers. Default: same as --learning_rate.
• --cross_attn_lr=<float> - Learning rate for cross-attention layers. Default: same as --learning_rate.
• --mlp_lr=<float> - Learning rate for MLP layers. Default: same as --learning_rate.
• --mod_lr=<float> - Learning rate for AdaLN modulation layers. Default: same as --learning_rate. Note: modulation layers are not included in LoRA by default.
• --llm_adapter_lr=<float> - Learning rate for LLM adapter layers. Default: same as --learning_rate.

For LoRA training, use network_reg_lrs in --network_args instead. See Section 5.2.

Memory and Speed / メモリ・速度関連

• --blocks_to_swap=<integer>
  • Number of Transformer blocks to swap between CPU and GPU. More blocks reduce VRAM but slow training. Maximum values depend on model size:
    • 28-block model: max 26 (Anima-Preview)
    • 36-block model: max 34
    • 20-block model: max 18
  • Cannot be used with --cpu_offload_checkpointing or --unsloth_offload_checkpointing.
• --unsloth_offload_checkpointing
  • Offload activations to CPU RAM using async non-blocking transfers (faster than --cpu_offload_checkpointing). Cannot be combined with --cpu_offload_checkpointing or --blocks_to_swap.
• --cache_text_encoder_outputs
  • Cache Qwen3 text encoder outputs to reduce VRAM usage. Recommended when not training text encoder LoRA.
• --cache_text_encoder_outputs_to_disk
  • Cache text encoder outputs to disk. Auto-enables --cache_text_encoder_outputs.
• --cache_latents, --cache_latents_to_disk
  • Cache Qwen-Image VAE latent outputs.
• --vae_chunk_size=<integer>
  • Chunk size for Qwen-Image VAE processing. Reduces VRAM usage at the cost of speed. Default is no chunking.
• --vae_disable_cache
  • Disable internal caching in Qwen-Image VAE to reduce VRAM usage.
• --compile and related options
  • Speed up training with per-block torch.compile. See the torch.compile for Anima guide for details.
• --qwen_image_vae_2d
  • Use the image-only 2D Qwen-Image VAE. The official (3D causal Conv3d) VAE weights are converted to equivalent 2D convolutions on load, so no separate weight file is needed and, for single images, the latents are numerically equivalent to the default 3D VAE. This is roughly 2x faster and uses about 1/3 of the peak VRAM for encode/decode (e.g., about 4.4 GB / 7.7 s -> about 1.4 GB / 4.5 s for 10 images at 1024x1024 on an RTX 3090). Recommended for latent caching. Note: with the 2D VAE the peak memory is dominated by full-resolution activations and the mid-block attention, so --vae_chunk_size has little further effect on the peak, and --vae_disable_cache is a no-op (the 2D VAE has no temporal cache).

Incompatible or Unsupported Options / 非互換・非サポートの引数

• --v2, --v_parameterization, --clip_skip - Options for Stable Diffusion v1/v2 that are not used for Anima training.
• --fp8_base - Not supported for Anima. If specified, it will be disabled with a warning.

4.2. Starting Training / 学習の開始

After setting the required arguments, run the command to begin training. The overall flow and how to check logs are the same as in the train_network.py guide.

5. LoRA Target Modules / LoRAの学習対象モジュール

When training LoRA with anima_train_network.py, the following modules are targeted by default:

• DiT Blocks (Block): Self-attention (self_attn), cross-attention (cross_attn), and MLP (mlp) layers within each transformer block. Modulation (adaln_modulation), norm, embedder, and final layers are excluded by default.
• Embedding layers (PatchEmbed, TimestepEmbedding) and Final layer (FinalLayer): Excluded by default but can be included using include_patterns.
• LLM Adapter Blocks (LLMAdapterTransformerBlock): Only when --network_args "train_llm_adapter=True" is specified.
• Text Encoder (Qwen3): Only when --network_train_unet_only is NOT specified and --cache_text_encoder_outputs is NOT used.

The LoRA network module is networks.lora_anima.

5.1. Module Selection with Patterns / パターンによるモジュール選択

By default, the following modules are excluded from LoRA via the built-in exclude pattern:

.*(_modulation|_norm|_embedder|final_layer).*

You can customize which modules are included or excluded using regex patterns in --network_args:

• exclude_patterns - Exclude modules matching these patterns (in addition to the default exclusion).
• include_patterns - Force-include modules matching these patterns, overriding exclusion.

Patterns are matched against the full module name using re.fullmatch().

Example to include the final layer:

--network_args "include_patterns=['.*final_layer.*']"

Example to additionally exclude MLP layers:

--network_args "exclude_patterns=['.*mlp.*']"

5.2. Regex-based Rank and Learning Rate Control / 正規表現によるランク・学習率の制御

You can specify different ranks (network_dim) and learning rates for modules matching specific regex patterns:

• network_reg_dims: Specify ranks for modules matching a regular expression. The format is a comma-separated string of pattern=rank.
  • Example: --network_args "network_reg_dims=.*self_attn.*=8,.*cross_attn.*=4,.*mlp.*=8"
  • This sets the rank to 8 for self-attention modules, 4 for cross-attention modules, and 8 for MLP modules.
• network_reg_lrs: Specify learning rates for modules matching a regular expression. The format is a comma-separated string of pattern=lr.
  • Example: --network_args "network_reg_lrs=.*self_attn.*=1e-4,.*cross_attn.*=5e-5"
  • This sets the learning rate to 1e-4 for self-attention modules and 5e-5 for cross-attention modules.

Notes:

• Settings via network_reg_dims and network_reg_lrs take precedence over the global --network_dim and --learning_rate settings.
• Patterns are matched using re.fullmatch() against the module's original name (e.g., blocks.0.self_attn.q_proj).

5.3. LLM Adapter LoRA / LLM Adapter LoRA

To apply LoRA to the LLM Adapter blocks:

--network_args "train_llm_adapter=True"

In preliminary tests, lowering the learning rate for the LLM Adapter seems to improve stability. Adjust it using something like: "network_reg_lrs=.*llm_adapter.*=5e-5".

5.4. Other Network Args / その他のネットワーク引数

• --network_args "verbose=True" - Print all LoRA module names and their dimensions.
• --network_args "rank_dropout=0.1" - Rank dropout rate.
• --network_args "module_dropout=0.1" - Module dropout rate.
• --network_args "loraplus_lr_ratio=2.0" - LoRA+ learning rate ratio.
• --network_args "loraplus_unet_lr_ratio=2.0" - LoRA+ learning rate ratio for DiT only.
• --network_args "loraplus_text_encoder_lr_ratio=2.0" - LoRA+ learning rate ratio for text encoder only.

.*(_modulation|_norm|_embedder|final_layer).*

6. Using the Trained Model / 学習済みモデルの利用

When training finishes, a LoRA model file (e.g. my_anima_lora.safetensors) is saved in the directory specified by output_dir. Use this file with inference environments that support Anima, such as ComfyUI with appropriate nodes.

7. Advanced Settings / 高度な設定

7.1. VRAM Usage Optimization / VRAM使用量の最適化

Anima models can be large, so GPUs with limited VRAM may require optimization:

Key VRAM Reduction Options

• --blocks_to_swap <number>: Swaps blocks between CPU and GPU to reduce VRAM usage. Higher numbers save more VRAM but reduce training speed. See model-specific max values in section 4.1.

• --unsloth_offload_checkpointing: Offloads gradient checkpoints to CPU using async non-blocking transfers. Faster than --cpu_offload_checkpointing. Cannot be combined with --blocks_to_swap.

• --gradient_checkpointing: Standard gradient checkpointing to reduce VRAM at the cost of compute.

• --cache_text_encoder_outputs: Caches Qwen3 outputs so the text encoder can be freed from VRAM during training.

• --cache_latents: Caches Qwen-Image VAE outputs so the VAE can be freed from VRAM during training.

• Using Adafactor optimizer: Can reduce VRAM usage:

  --optimizer_type adafactor --optimizer_args "relative_step=False" "scale_parameter=False" "warmup_init=False" --lr_scheduler constant_with_warmup --max_grad_norm 0.0
  

7.2. Training Settings / 学習設定

Timestep Sampling

The --timestep_sampling option specifies how timesteps are sampled. The available methods are the same as FLUX training:

• sigma: Sigma-based sampling like SD3.
• uniform: Uniform random sampling from [0, 1].
• sigmoid (default): Sample from Normal(0,1), multiply by sigmoid_scale, apply sigmoid. Good general-purpose option.
• shift: Like sigmoid, but applies the discrete flow shift formula: t_shifted = (t * shift) / (1 + (shift - 1) * t).
• flux_shift: Resolution-dependent shift used in FLUX training.

See the flux_train_network.py guide for detailed descriptions.

Discrete Flow Shift

The --discrete_flow_shift option (default 1.0) only applies when --timestep_sampling is set to shift. The formula is:

t_shifted = (t * shift) / (1 + (shift - 1) * t)

Loss Weighting

The --weighting_scheme option specifies loss weighting by timestep:

• uniform (default): Equal weight for all timesteps.
• sigma_sqrt: Weight by sigma^(-2).
• cosmap: Weight by 2 / (pi * (1 - 2*sigma + 2*sigma^2)).
• none: Same as uniform.
• logit_normal, mode: Additional schemes from SD3 training. See the sd3_train_network.md guide for details.

Visualizing the Timestep Distribution

To check how the above settings actually affect the sampled timesteps, two features are available (both shared with FLUX training):

• Training log: At the start of every run, a one-line summary of the timestep sampling configuration is logged. It explicitly states whether --discrete_flow_shift is applied for the chosen --timestep_sampling (only sigma and shift use it; with sigmoid, uniform, or flux_shift it is ignored). This makes it easy to notice when a shift value you set has no effect.
• --show_timesteps=<console|image>: Visualize the actual sampled-timestep distribution and the loss weighting for the current settings, then exit without training. console prints an ASCII histogram; image shows a matplotlib plot (requires matplotlib). The distribution is shown from noisy (top / left, t=1000) to clean (bottom / right, t=0).
• --show_timesteps_resolution=<H | H,W>: Image resolution (in pixels) assumed by --show_timesteps for resolution-dependent sampling such as flux_shift. A single value is used for both height and width; two comma-separated values are H,W (W,H gives the same result). Default 1024. This is useful because the --resolution argument is None when the dataset is configured via a .toml file.

Example:

--show_timesteps console --show_timesteps_resolution 1024

Caption Dropout

Caption dropout uses the caption_dropout_rate setting from the dataset configuration (per-subset in TOML). When using --cache_text_encoder_outputs, the dropout rate is stored with each cached entry and applied during training, so caption dropout is compatible with text encoder output caching.

If you change the caption_dropout_rate setting, you must delete and regenerate the cache.

Note: Currently, only Anima supports combining caption_dropout_rate with text encoder output caching.

7.3. Text Encoder LoRA Support / Text Encoder LoRAのサポート

Anima LoRA training supports training Qwen3 text encoder LoRA:

• To train only DiT: specify --network_train_unet_only
• To train DiT and Qwen3: omit --network_train_unet_only and do NOT use --cache_text_encoder_outputs

You can specify a separate learning rate for Qwen3 with --text_encoder_lr. If not specified, the default --learning_rate is used.

Note: When --cache_text_encoder_outputs is used, text encoder outputs are pre-computed and the text encoder is removed from GPU, so text encoder LoRA cannot be trained.

8. Other Training Options / その他の学習オプション

• --loss_type: Loss function for training. Default l2.

  • l1: L1 loss.
  • l2: L2 loss (mean squared error).
  • huber: Huber loss.
  • smooth_l1: Smooth L1 loss.

• --huber_schedule, --huber_c, --huber_scale: Parameters for Huber loss when --loss_type is huber or smooth_l1.

• --ip_noise_gamma, --ip_noise_gamma_random_strength: Input Perturbation noise gamma values.

• --fused_backward_pass: Fuses the backward pass and optimizer step to reduce VRAM usage. Only works with Adafactor. For details, see the sdxl_train_network.py guide.

• --weighting_scheme, --logit_mean, --logit_std, --mode_scale: Timestep loss weighting options. For details, refer to the sd3_train_network.md guide.

9. Related Tools / 関連ツール

networks/anima_convert_lora_to_comfy.py

A script to convert LoRA models to ComfyUI-compatible format. ComfyUI does not directly support sd-scripts format Qwen3 LoRA, so conversion is necessary (conversion may not be needed for DiT-only LoRA). You can convert from the sd-scripts format to ComfyUI format with:

python networks/convert_anima_lora_to_comfy.py path/to/source.safetensors path/to/destination.safetensors

Using the --reverse option allows conversion in the opposite direction (ComfyUI format to sd-scripts format). However, reverse conversion is only possible for LoRAs converted by this script. LoRAs created with other training tools cannot be converted.

python networks/convert_anima_lora_to_comfy.py path/to/source.safetensors path/to/destination.safetensors

10. Others / その他

Metadata Saved in LoRA Models

The following metadata is saved in the LoRA model file:

• ss_weighting_scheme
• ss_logit_mean
• ss_logit_std
• ss_mode_scale
• ss_timestep_sampling
• ss_sigmoid_scale
• ss_discrete_flow_shift