README.md

July 30, 2025 · View on GitHub

Multi-Object Sketch Animation by Scene Decomposition and Motion Planning (ICCV 2025)

Jingyu Liu Zijie Xin Yuhan Fu Ruixiang Zhao Bangxiang Lan Xirong Li^†

Renmin University of China

:new: Latest Update

[2025.08.02] 🔥 We released more created multi-object sketches! There are 560 multi-object sketches now!
[2025.07.29] 🔥 We released the project page for MoSketch.
[2025.07.26] 🔥 We released the code for MoSketch.
[2025.06.26] 🎉 MoSketch is accepted by ICCV 2025!
[2025.03.25] 🔥 We released the MoSketch Paper. MoSketch is an iterative optimization based and thus training-data free method, aiming to animate a multi-object sketch w.r.t. a specific textual instruction.

Sketch animation, which brings static sketches to life by generating dynamic video sequences, has found widespread applications in GIF design, cartoon production, and daily entertainment. While current methods for sketch animation perform well in single-object sketch animation, they struggle in multi-object scenarios. By analyzing their failures, we identify two major challenges of transitioning from single-object to multi-object sketch animation: object-aware motion modeling and complex motion optimization. For multi-object sketch animation, we propose MoSketch based on iterative optimization through Score Distillation Sampling (SDS) and thus animating a multi-object sketch in a training-data free manner. To tackle the two challenges in a divide-and-conquer strategy, MoSketch has four novel modules, i.e., LLM-based scene decomposition, LLM-based motion planning, multi-grained motion refinement, and compositional SDS. Extensive qualitative and quantitative experiments demonstrate the superiority of our method over existing sketch animation approaches. MoSketch takes a pioneering step towards multi-object sketch animation, opening new avenues for future research and applications.

🔧 Setup

Download the code of MoSketch.

git clone https://github.com/jyliu-98/MoSketch.git
cd MoSketch

📌 Environment

Create a new anaconda environment and install dependencies.

# create a new anaconda env
conda create -n mosketch python==3.8 -y
conda activate mosketch

# install torch and dependencies
pip install -r requirements.txt

Install diffvg

# install diffvg's dependencies
conda install -y -c anaconda cmake
conda install -y -c conda-forge ffmpeg
# ************************************************
# if 'conda install -y -c conda-forge ffmpeg' gets stuck, try these:
# conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
# conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free
# conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
# conda config --set show_channel_urls yes
# conda install -y ffmpeg
# ************************************************

# install diffvg
git clone https://github.com/BachiLi/diffvg.git
cd diffvg
git submodule update --init --recursive
python setup.py install
cd ..
rm -rf diffvg

⬇️ Checkpoint of T2V Diffusion Model

Download the checkpoint of ModelScopeT2V, and put it in the code of MoSketch (./text-to-video-ms-1.7b).

✍️ Input Sketch

The input sketches should be provided in SVG format, where a sketch is composed of strokes and each stroke is a cubic Bézier curve controlled by four points, as described in the paper. Make sure the input sketches can be processed with diffvg.

Recommended sketch format:

Rendering size: 256x256
Number of strokes: <300
Strokes type: cubic Bezier curves
Stroke width: 1~3

The 60 multi-object sketches used in the paper are provided in ./data/raw/60sketches. They can be processed with diffvg.

For own sketch preparation, if you want to generate sketches automatically, we recommend CLIPasso, an image-to-sketch method that produces sketches in vector format; if you want to create sketches manually, you can use some free online tools, such as js.design. Note that sketches created by CLIPasso can be processed with diffvg, while sketches created by js.design should go through ./preprocess.py:

python preprocess.py --svg_path './data/raw/Yours.svg'

After that, get the PNG format of your own sketch, which will be used in the scene decomposition, the stroke(point) assignment, and the motion planning. Your can save the PNG in js.design, or run the following Python code to turn SVG to PNG:

import cairosvg

svg_path = './data/raw/Yours.svg'
png_path = './data/raw/Yours.png'
cairosvg.svg2png(url=svg_path,
                 write_to=png_path,
                 scale=1, background_color="white")

Our preparation process:

Creating some single-object vector sketches by CLIPasso.
Using js.design to gather the single-object vector sketches in a reasonable scene.
Using js.design to edit the multi-object sketch (e.g., add or delete strokes).
Save the multi-object sketch (both SVG and PNG), and run ./preprocess.py to make sure the SVG can be processed with diffvg.

We release 500 more created sketches! There are 560 vector multi-object sketches now! (./data/raw/560sketches.zip)

🎥 Generate A Video!

🚀 Quick Start

The scene decomposition (_decomp.txt), the stroke(point) assignment (_semantic.txt) and the motion plan (_traj.txt) of the 60 multi-object sketches are provided in ./data/processed. The text instruction of each sketch can be found in ./data/raw/60sketches/caption.txt. Run this command to get the animation of one sketch (e.g., 'basketball5'):

CUDA_VISIBLE_DEVICES=0 python animate_mosketch.py \
        --sketch 'basketball5' \
        --caption 'The player soars through the air with a basketball, arm extended for an electrifying slam dunk to a hoop.' \
        --num_iter 500 \
        --seed 130 \
        --num_frames 20

The output video will be saved in ./output/basketball5.

The scene decomposition, stroke(point) assignment and motion plan of the 500 more created sketches are provided in ./data/processed/560sketches.zip. Use the above command to animate them!

👩‍🎨 Animate Your Own Sketch

The scene decomposition, the stroke(point) assignment and the motion plan of your own multi-object sketch should be provided before animation, and the format should follow the 60 created sketches. Make a new folder ./data/processed/Yours, and put the vector sketch (Yours.svg) in it.

🧩 Scene Decomposition

We use LLM to get the scene decomposition of the multi-object sketch. The LLM is not limited. We recommend GPT-4, especially ChatGPT-4, to get the result and check it in real time. We should provide the sketch (Yours.png) and the text caption (Yours_Text_Instruction). The GPT-4 instruction and examples are provided in ./data/examples-for-scene-decomposition. Save the result in ./data/processed/Yours/Yours_decomp.txt, and the format should be the same as the 60 created sketches (e.g., ./data/processed/aircrafter3/aircrafter3_decomp.txt).

🧮 Stroke(point) Assignment

The stroke(point) assignment aims to assign the strokes, as well as the control points, to their belonging objects. The stroke(point) assignment is actually the object segmentation task in vector sketch. We employ GoundingDino to conduct object grounding on the multi-object sketch, and then assign strokes to objects based on the bounding boxes.

First, Your should install GoundingDino. Then copy the code MoSketch/stroke_assignment.py to the GoundingDino project. Make a new folder sketch in GoundingDino project, and copy the SVG Yours.svg and PNG Yours.png of the sketch in it. Run Grounded-Segment-Anything/stroke_assignment.py (do not forget adding object names in the parameter--text_prompt):

export CUDA_VISIBLE_DEVICES=0
python stroke_assignment.py \
  --config GroundingDINO/groundingdino/config/GroundingDINO_SwinT_OGC.py \
  --grounded_checkpoint groundingdino_swint_ogc.pth \
  --sam_checkpoint sam_vit_h_4b8939.pth \
  --sketch_dir "sketch" \
  --sketch_img Yours.png \
  --box_threshold 0.2 \
  --text_threshold 0.2 \
  --iou_w 1.0 \
  --text_prompt Your_Object_Names \
  --device "cuda"

The stroke(point) assignment are saved in Yours_semantic.txt, which lists objects and their strokes (IDs in SVG). Objects' bounding boxes are written in Yours_bbox.txt. Yours_color.svg is the visualization of stroke(point) assignment, and you can check it with color-object pairs printed in the output. Copy these files to the processed folder (MoSketch/data/processed/Yours).

Note that:

--box_threshold and --text_threshold are the semantic thresholds in GoundingDino. --iou_w is a parameter about the object overlap (more --iou_w means less tolerance for overlap). Adjust these three parameters flexibly during the stroke(point) assignment.
Sometimes object names should be replaced to get the correct object grounding in GoundingDino, e.g., 'basketball` → 'ball', 'player' → 'man'.
If there are more than one objects in sketch sharing the same name, repeat the name in --text_prompt. For example, 'hurdle9' in the 60 created sketches has three athletes:

export CUDA_VISIBLE_DEVICES=0
python stroke_assignment.py \
  --config GroundingDINO/groundingdino/config/GroundingDINO_SwinT_OGC.py \
  --grounded_checkpoint groundingdino_swint_ogc.pth \
  --sam_checkpoint sam_vit_h_4b8939.pth \
  --sketch_dir "sketch" \
  --sketch_img huldres6.png \
  --box_threshold 0.2 \
  --text_threshold 0.2 \
  --iou_w 1.0 \
  --text_prompt "hurdle,athlete,athlete,athlete" \
  --device "cuda"

🚗 Motion Planning

We use LLM to get the motion plan of the multi-object sketch. We also recommend ChatGPT-4 to get the result and check it in real time. We should provide the sketch (Yours.png) and the text caption (Yours_Text_Instruction) and the object bounding boxes (already saved in Yours_bbox.txt during stroke(point) assignment). The GPT-4 instruction and examples are provided in ./data/examples-for-motion-planning. Save the result in ./data/processed/Yours/Yours_traj.txt, and the format should be the same as the 60 created sketches (e.g., ./data/processed/aircrafter3/aircrafter3_traj.txt).

We highly recommend to check the motion plan in time. Run ./view_plan.py to visualize the motion plan. The motion plan should be aligned with Yours_Text_Instruction. If you are not satisfy with the result, instruct the LLM for modification in time. The incorrect motion planning will lead to the failed animation.

python view_plan.py \
  --sketch_dir './data/processed/Yours' \
  --sketch_name 'Yours' \
  --frame_num 20

The motion plan will be saved as ./data/processed/Yours/Yours_color.gif

After getting the scene decomposition (Yours_decomp.txt), the stroke(point) assignment (Yours_semantic.txt) and the motion plan (Yours_traj.txt) of your own multi-object sketch in the folder ./data/processed/Yours, run ./animate_mosketch.py to get the final animation of your own sketch:

CUDA_VISIBLE_DEVICES=0 python animate_mosketch.py \
        --sketch 'Yours' \
        --caption Yours_Text_Instruction \
        --num_iter 500 \
        --seed 130 \
        --num_frames 20

The output video will be saved in ./output/Yours.

@inproceedings{liu2025multi,
  title={Multi-Object Sketch Animation by Scene Decomposition and Motion Planning},
  author={Liu, Jingyu and Xin, Zijie and Fu, Yuhan and Zhao, Ruixiang and Lan, Bangxiang and Li, Xirong},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  year={2025}
}