LingBot-Video

August 20, 2026 · View on GitHub

LingBot-Video is a flow-matching video generation model developed by the LingBot team; a single model handles text-to-video, image-to-video and text-to-image tasks.

Huge thanks to NancyFyong for the outstanding contribution to the integration of this model!

Installation

Before performing model inference and training, please install DiffSynth-Studio first.

git clone https://github.com/modelscope/DiffSynth-Studio.git
cd DiffSynth-Studio
pip install -e .

For more information on installation, please refer to Setup Dependencies.

Quick Start

Running the following code will load the Robbyant/lingbot-video-dense-1.3b model for inference. VRAM management is enabled, the framework automatically controls parameter loading based on available VRAM, requiring a minimum of 6GB VRAM.

import torch
import json
from diffsynth.utils.data import save_video, VideoData
from diffsynth.pipelines.lingbot_video import LingBotVideoPipeline, ModelConfig
from modelscope import dataset_snapshot_download

vram_config = {
    "offload_dtype": "disk",
    "offload_device": "disk",
    "onload_dtype": torch.float8_e4m3fn,
    "onload_device": "cpu",
    "preparing_dtype": torch.float8_e4m3fn,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}

pipe = LingBotVideoPipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="Robbyant/lingbot-video-dense-1.3b", origin_file_pattern="transformer/diffusion_pytorch_model.safetensors", **vram_config),
        ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern="*.safetensors", **vram_config),
        ModelConfig(model_id="Robbyant/lingbot-video-dense-1.3b", origin_file_pattern="vae/diffusion_pytorch_model.safetensors", **vram_config),
    ],
    processor_config=ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern=""),
    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 0.5,
)

dataset_snapshot_download(
    dataset_id="DiffSynth-Studio/diffsynth_example_dataset",
    local_dir="data/diffsynth_example_dataset",
    allow_file_pattern="lingbot_video/lingbot-video-dense-1.3b_t2v/*",
)
with open("data/diffsynth_example_dataset/lingbot_video/lingbot-video-dense-1.3b_t2v/t2v_example_1.json", "r", encoding="utf-8") as f:
    caption = json.load(f)

video = pipe(
    prompt=caption,
    negative_prompt=pipe.default_negative_prompt,
    height=480, width=832, num_frames=81,
    num_inference_steps=40, cfg_scale=3.0,
    seed=0,
)
save_video(video, "video.mp4", fps=15, quality=10)

Model Overview

Model IDInferenceLow VRAM InferenceFull TrainingFull Training ValidationLoRA TrainingLoRA Training Validation
Robbyant/lingbot-video-dense-1.3b: T2Vcodecodecodecodecodecode
Robbyant/lingbot-video-dense-1.3b: TI2Vcodecodecodecodecodecode
Robbyant/lingbot-video-dense-1.3b: T2Icodecode----
Robbyant/lingbot-video-moe-30b-a3b: T2Vcodecodecodecodecodecode
Robbyant/lingbot-video-moe-30b-a3b: TI2Vcodecodecodecodecodecode
Robbyant/lingbot-video-moe-30b-a3b: T2Icodecode----
Robbyant/lingbot-video-moe-30b-a3b: T2V + Refinementcodecode----
Robbyant/lingbot-video-moe-30b-a3b: TI2V + Refinementcodecode----

Model Inference

The model is loaded via LingBotVideoPipeline.from_pretrained, see Loading Models for details.

The input parameters for LingBotVideoPipeline inference include:

  • prompt: Structured-JSON caption (dict) or a plain string describing the content. LingBot-Video is trained on structured captions; the pipeline normalises a dict automatically. Released structured captions ship in the example dataset (see Prompt rewriting below).
  • negative_prompt: Negative prompt describing content that should not appear. pipe.default_negative_prompt ships the official T2V/V2V/TI2V negative prompt; pipe.default_negative_prompt_image is the T2I variant with temporal terms removed.
  • input_image: First-frame PIL image for image-to-video (TI2V). The frame is VAE-encoded to a clean latent pinned into the first temporal slot after every scheduler step, so the model only generates the frames that follow. Leave None for T2V / V2V / T2I.
  • input_video: Input video (a list of frames or a VideoData) for video-to-video generation, used together with denoising_strength.
  • denoising_strength: Denoising strength in [0, 1], default 1.0. Lower values keep more of the input video structure. Only effective when input_video is provided.
  • height: Video / image height, default 480. Must be a multiple of 16.
  • width: Video / image width, default 480. Must be a multiple of 16.
  • num_frames: Number of frames, default 81. Must satisfy 4k+1 (the VAE compresses time by 4×). Use num_frames=1 for text-to-image.
  • cfg_scale: Classifier-free guidance scale, default 3.0.
  • num_inference_steps: Number of inference steps, default 40.
  • sigma_shift: Flow-matching timestep shift, default 3.0.
  • t_thresh: Refinement start sigma, default None (plain generation). When set, the schedule is truncated so that sampling starts at sigma=t_thresh and input_video is noised to exactly that level. Only meaningful together with input_video; TI2V additionally re-pins the clean first-frame latent after every step. The official refiner setting is 0.85.
  • sigma_tail_steps: Number of extra low-noise steps appended to the tail of the refinement schedule, default 2. Only effective when t_thresh is set.
  • seed: Random seed. Default is None, meaning completely random.
  • rand_device: Device for generating the initial noise, default "cpu".
  • progress_bar_cmd: Progress bar, default tqdm. Can be disabled by setting to lambda x: x.

If VRAM is insufficient, please enable VRAM Management. We provide recommended low-VRAM configurations for each task in the example code, see the table in the "Model Overview" section above.

Two-stage refinement

The MoE refiner performs a short second pass at a higher resolution: the official setup generates at 480×832 with 40 steps, then refines at 1088×1920 with 8 steps. Load the pipeline with the refiner/ shards instead of transformer/, feed the base clip back in through input_video at the higher resolution, and set t_thresh:

input_video = VideoData("video_base.mp4", height=1088, width=1920)
video = pipe(
    prompt=caption,
    negative_prompt=pipe.default_negative_prompt,
    input_video=input_video,
    height=1088, width=1920, num_frames=81,
    num_inference_steps=8, cfg_scale=3.0,
    t_thresh=0.85, sigma_tail_steps=2,
    seed=0,
)

The upscaled clip is VAE-encoded and noised back to sigma=t_thresh, so the pass keeps the structure of the base clip and regenerates detail at the target resolution. Pass the same caption as the base pass and keep the same aspect ratio. The refinement resolution dominates the cost — at 1088×1920 the sequence is ~5× longer than at 480×832 — so run this pass with VRAM management enabled.

Prompt rewriting

LingBot-Video is trained on structured-JSON captions, not free-form prose. Feeding a flat sentence is out-of-distribution and visibly degrades quality. The pipeline accepts a caption as a dict (the format used at training time) or a plain string, and normalises the dict internally.

Released structured captions ship in the example dataset (t2v_example_*.json, ti2v_example.json, t2i_example.json under DiffSynth-Studio/diffsynth_example_dataset, downloaded automatically by the inference example scripts). Load one with json.load and pass the resulting dict to the pipeline, or use one as a template.

To turn a brief idea into a structured caption, use the two-stage rewriter shipped under examples/lingbot_video/model_training/scripts/prompt_rewriter.py — stage 1 expands the idea into a natural-language caption, stage 2 maps it into structured JSON. The rewriter is a separate VLM + stage-2 LoRA adapter and is not downloaded automatically:

RoleModel IDSize
Rewriter base VLM (stage 1 + 2)Qwen/Qwen3.6-27B~55 GB
Rewriter stage-2 LoRA adapterRobbyant/lingbot-video-rewriter-lora~0.5 GB
import os
os.environ["REWRITER_BASE_MODEL"] = "./models/Qwen/Qwen3.6-27B"
os.environ["REWRITER_ADAPTER"] = "./models/Robbyant/lingbot-video-rewriter-lora"

# Run from the repo root so the package-style import resolves.
from examples.lingbot_video.model_training.scripts.prompt_rewriter import rewrite_prompt
caption = rewrite_prompt("a puppy running across a meadow", mode="t2v", duration=5)
video = pipe(prompt=caption, height=480, width=832, num_frames=81, cfg_scale=3.0)

Instead of the env vars you can pass base= / adapter= to rewrite_prompt, or drive a hosted / OpenAI-compatible endpoint by passing a custom object exposing generate(text, image, use_lora) as backend=.

Model Training

Models in the LingBot-Video series are trained uniformly via examples/lingbot_video/model_training/train.py. The script parameters include:

  • General Training Parameters
    • Dataset Configuration
      • --dataset_base_path: Root directory of the dataset.
      • --dataset_metadata_path: Path to the dataset metadata file.
      • --dataset_repeat: Number of dataset repeats per epoch.
      • --dataset_num_workers: Number of processes per DataLoader.
      • --data_file_keys: Field names to load from metadata, typically paths to image or video files, separated by ,.
    • Model Loading Configuration
      • --model_paths: Paths to load models from, in JSON format.
      • --model_id_with_origin_paths: Model IDs with original paths, separated by commas.
      • --extra_inputs: Additional input parameters required by the model Pipeline, separated by ,.
      • --fp8_models: Models to load in FP8 format, currently only supported for models whose parameters are not updated by gradients.
      • --quant_options: Dynamically quantize loaded models. Semicolon-separated entries, each <model_string>:<method>[/<exclude_modules>], where <model_string> matches an entry in --model_paths/--model_id_with_origin_paths, method is a registered method (e.g. bitsandbytes_nf4), and exclude_modules optionally lists layers kept in full precision.
    • Basic Training Configuration
      • --learning_rate: Learning rate.
      • --num_epochs: Number of epochs.
      • --trainable_models: Trainable models, e.g., dit, vae, text_encoder.
      • --find_unused_parameters: Whether unused parameters exist in DDP training.
      • --weight_decay: Weight decay magnitude.
      • --task: Training task, defaults to sft.
    • Output Configuration
      • --output_path: Path to save the model.
      • --remove_prefix_in_ckpt: Remove prefix in the model's state dict.
      • --save_steps: Interval in training steps to save the model.
    • LoRA Configuration
      • --lora_base_model: Which model to add LoRA to.
      • --lora_target_modules: Which layers to add LoRA to.
      • --lora_rank: Rank of LoRA.
      • --lora_checkpoint: Path to LoRA checkpoint.
      • --preset_lora_path: Path to preset LoRA checkpoint for LoRA differential training.
      • --preset_lora_model: Which model to integrate preset LoRA into, e.g., dit.
    • Gradient Configuration
      • --use_gradient_checkpointing: Whether to enable gradient checkpointing.
      • --use_gradient_checkpointing_offload: Whether to offload gradient checkpointing to CPU memory.
      • --gradient_accumulation_steps: Number of gradient accumulation steps.
    • Resolution Configuration
      • --height: Height of the video. Must be divisible by 16.
      • --width: Width of the video. Must be divisible by 16.
      • --max_pixels: Maximum pixel area, images larger than this will be scaled down during dynamic resolution.
      • --num_frames: Number of frames in the video. Must satisfy 4k+1.
  • LingBot-Video Specific Parameters
    • --processor_path: Path to the Qwen3-VL processor directory (or model_id:origin_file_pattern). Used to tokenize prompts.
    • --first_frame_as_condition: Enable image-to-video (TI2V) LoRA / full training. Each clip is conditioned on its own first frame: the frame is VAE-encoded to a clean latent pinned into the first temporal slot (and fed to the Qwen3-VL text encoder as vision input), and excluded from the flow-matching loss.
    • --max_timestep_boundary: Max timestep boundary as a fraction of the training schedule, in [0, 1].
    • --min_timestep_boundary: Min timestep boundary as a fraction of the training schedule, in [0, 1].
    • --initialize_model_on_cpu: Whether to initialize models on CPU.

We provide a sample dataset for your testing. You can download it with the following command:

modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --include "lingbot_video/lingbot-video-dense-1.3b_t2v/*" --local_dir ./data/diffsynth_example_dataset

Training captions should be structured-JSON captions (the same in-distribution format used at inference). If your dataset stores raw prose, rewrite it once offline with examples/lingbot_video/model_training/scripts/rewrite_captions.py before training.

We provide recommended training scripts for each task, please refer to the table in "Model Overview" above. For guidance on writing model training scripts, see Model Training; for more advanced training algorithms, see Training Framework Overview.