ComfyUI-SD-CN-Animation

April 2, 2026 · View on GitHub

Temporally coherent video generation and stylization for ComfyUI using optical flow. Port of SD-CN-Animation to ComfyUI custom nodes.

Generate videos from text prompts (Txt2Vid) or stylize existing videos (Vid2Vid) with any Stable Diffusion checkpoint, ControlNet, and IPAdapter — while maintaining smooth frame-to-frame consistency via optical flow prediction.

Example output

How It Works

Workflow screenshot

Both pipelines use a two-pass approach per frame to balance creativity with coherence:

Txt2Vid uses the FloweR model to predict optical flow, occlusion, and the next frame from the previous 4 frames. Each frame is refined through two SD sampling passes — an inpaint pass guided by the occlusion mask, and a refinement pass for cleanup — with histogram matching to prevent color drift.

Vid2Vid uses RAFT optical flow estimation between consecutive input video frames to warp the previous stylized frame forward. Occlusion detection identifies areas where the warp breaks down, and those regions get inpainted by SD. A second refinement pass and histogram matching complete the frame.

Nodes

Node	Description
Load FloweR Model	Loads FloweR optical flow prediction model
SD-CN Animation Txt2Vid	Generate video from text prompt using FloweR
Load RAFT Model	Loads RAFT optical flow estimation model
SD-CN Animation Vid2Vid	Stylize video frames using RAFT optical flow
SD-CN Prompt Schedule	Keyframed prompt scheduling with interpolation
SD-CN Motion Control	Control flow, zoom, pan, and rotation in Txt2Vid
FloweR Predict	Standalone FloweR prediction (debug/visualization)
Histogram Match	Standalone color histogram matching utility

Installation

Install via ComfyUI Manager or clone manually into ComfyUI/custom_nodes/.

Model weights (~8MB FloweR + ~20MB RAFT) are downloaded automatically from Hugging Face on first launch.

Workflows

Txt2Vid (Text to Video)

CheckpointLoader ──→ model ──────────────────→ SD-CN Animation Txt2Vid ──→ VHS_VideoCombine
                 ──→ clip ──→ CLIPTextEncode ──→ positive                   (set fps: 12)
                 ──→ vae  ──────────────────→
LoadFloweRModel ──→ flower_model ───────────→

Key parameters:

num_frames — total frames to generate (30-120+)
processing_strength — how much SD changes each frame (0.5-0.85). Lower = smoother morphing, higher = more creative variation
fix_frame_strength — refinement pass strength (0.1-0.2). Keep low for stability, set to 0 to skip entirely
loop_frames — set to 10-20 to blend the end back into the first frame for a seamless loop. 0 = disabled

For smooth morphing animation: Use processing_strength: 0.55, fix_frame_strength: 0.12, num_frames: 90, and set your video combine node to 12-14 fps.

Vid2Vid (Video Stylization)

VHS_LoadVideo ──→ frames ───────────────────→ SD-CN Animation Vid2Vid ──→ VHS_VideoCombine
CheckpointLoader ──→ model ─────────────────→
                 ──→ clip ──→ CLIPTextEncode → positive
                 ──→ vae  ──────────────────→
LoadRAFTModel ──→ raft_model ───────────────→

Key parameters:

processing_strength — denoising for main stylization (0.75-0.85)
fix_frame_strength — refinement pass (0.1-0.2, or 0 to skip)
blend_alpha — blend between warped styled frame (0.0) and current input (1.0). Higher values keep more of the original video structure
occlusion_mask_blur — smooths the mask that decides what gets regenerated (2-4)
occlusion_mask_trailing — enable to reduce ghosting artifacts

ControlNet

Wire a Load ControlNet Model node into the control_net input:

Vid2Vid: The current input video frame is automatically used as the ControlNet hint each frame. Great with depth, canny, or lineart ControlNets to preserve structure.
Txt2Vid: The FloweR-predicted frame is used as the hint.

Prompt Scheduling (Experimental)

Note: Prompt scheduling is experimental. At low processing_strength values (below 0.5), the text prompt has limited influence per frame and content may drift from the intended subject over long sequences. Higher processing_strength (0.55+) gives the prompt more steering power but reduces temporal smoothness. We are actively working on improving this balance.

Use the SD-CN Prompt Schedule node to change prompts at specific frames. Prompts interpolate smoothly between keyframes.

SD-CN Prompt Schedule ──→ positive ──→ SD-CN Animation Txt2Vid
         ↑
     CLIP from checkpoint

Write a keyframe schedule in the text box:

"0": "a serene forest landscape"
"30": "the forest transforms into an ocean"
"60": "a vast ocean under starlight"

Frame numbers define when each prompt takes effect
Conditioning is linearly interpolated between keyframes for smooth transitions
Frames beyond the last keyframe hold that prompt
Frame count is controlled by the Txt2Vid/Vid2Vid node, not the schedule
Works with both SD1.5 and SDXL models
Without this node, the standard CLIPTextEncode workflow still works as before

Motion Control

Use the SD-CN Motion Control node to influence camera movement and animation dynamics in Txt2Vid. Wire it into the optional motion_ctrl input — without it, the node behaves as normal.

SD-CN Motion Control ──→ motion_ctrl ──→ SD-CN Animation Txt2Vid

Parameters:

Parameter	Default	Description
`occlusion_multiplier`	10.0	How much of each frame gets regenerated by SD. Higher = more creative freedom, lower = more temporal coherence
`flow_scale`	1.0	Multiplier on FloweR's predicted motion. `< 1` = slower, `> 1` = exaggerated
`pan_x`	0.0	Horizontal drift in pixels/frame. Positive = pan right
`pan_y`	0.0	Vertical drift in pixels/frame. Positive = pan down
`zoom`	1.0	Per-frame scale factor. Values compound — use small increments
`rotate`	0.0	Degrees/frame rotation

Suggested starting points:

Slow zoom in: zoom: 1.005
Slow zoom out: zoom: 0.995
Gentle pan right: pan_x: 1.5
Dreamy drift: flow_scale: 1.3, occlusion_multiplier: 15
Tight coherence: flow_scale: 0.8, occlusion_multiplier: 5

All values are per-frame, so even subtle settings produce visible effects over 60+ frames. Zoom and rotate are applied as affine transforms on the FloweR prediction, while flow_scale and pan modify the optical flow field directly.

Multiple ControlNets / IPAdapter

Apply additional ControlNets or IPAdapter to the conditioning before passing it into the animation node:

CLIPTextEncode ──→ IPAdapterApply ──→ ControlNetApply ──→ positive
                                                            ↓
                                                     SDCNVid2Vid ← control_net (per-frame depth/canny)

ControlNets applied to conditioning beforehand use static hints. The node's control_net input is the one that gets updated per-frame.

Tips

Speed vs quality: Reducing steps to 10-12 speeds things up with minimal quality loss. Skipping the refine pass (fix_frame_strength: 0) halves per-frame time.
Temporal coherence: Lower processing_strength = smoother motion but less stylization.
Frame interpolation: Run output through a RIFE node before combining into video for extra smoothness.
Playback speed: The node outputs raw frames — set FPS on your video combine node (12 fps is a good default).

Credits

SD-CN-Animation — original A1111 extension and FloweR model by volotat (MIT License)
RAFT — Recurrent All-Pairs Field Transforms for Optical Flow by Zachary Teed and Jia Deng, Princeton Vision Lab (paper, code, BSD 3-Clause License)

License

MIT. RAFT model code under BSD 3-Clause (see raft/ directory).