Perception-as-Control

Official implementation of "Perception-as-Control: Fine-grained Controllable Image Animation with 3D-aware Motion Representation" (ICCV 2025)

Perception-as-Control: Fine-grained Controllable Image Animation with 3D-aware Motion Representation
Yingjie Chen, Yifang Men, Yuan Yao, Miaomiao Cui, Liefeng Bo

💡 Abstract

Motion-controllable image animation is a fundamental task with a wide range of potential applications. Recent works have made progress in controlling camera or object motion via the same 2D motion representations or different control signals, while they still struggle in supporting collaborative camera and object motion control with adaptive control granularity. To this end, we introduce 3D-aware motion representation and propose an image animation framework, called Perception-as-Control, to achieve fine-grained collaborative motion control. Specifically, we construct 3D-aware motion representation from a reference image, manipulate it based on interpreted user intentions, and perceive it from different viewpoints. In this way, camera and object motions are transformed into intuitive, consistent visual changes. Then, the proposed framework leverages the perception results as motion control signals, enabling it to support various motion-related video synthesis tasks in a unified and flexible way. Experiments demonstrate the superiority of the proposed method.

🔥 Updates

• (2025-08-04) A gradio demo is released.
• (2025-06-27) Our work has been accepted by ICCV 2025 🎉🎉🎉.
• (2025-03-31) We release the inference code and model weights of Perception-as-Control.
• (2025-03-10) We update a new version of paper with more details.
• (2025-01-09) The project page, demo video and technical report are released. The full paper version with more details is in process.

📑 TODO List

• Release inference code and model weights
• Provide a Gradio demo
• Release a DiT version
• Release training code

Usage

Environment

$ pip install -r requirements.txt

Pretrained Weights

1. Download pretrained weights and put them in $INSTALL_DIR/pretrained_weights.

2. Download pretrained weight of based models and put them in $INSTALL_DIR/pretrained_weights:

• sd-vae-ft-mse
• image_encoder

The pretrained weights are organized as follows:

./pretrained_weights/
|-- denoising_unet.pth
|-- reference_unet.pth
|-- cam_encoder.pth
|-- obj_encoder.pth
|-- sd-vae-ft-mse
|   |-- ...
|-- sd-image-variations-diffusers
|   |-- ...

Inference

$ python inference.py

The results will be saved in $INSTALL_DIR/outputs.

Run Gradio Demo

$ python run_gradio.py

🎥 Demo

Fine-grained collaborative motion control

Camera Motion Control

Object Motion Control

Collaborative Motion Control

Potential applications

Motion Generation	Motion Clone
Motion Transfer	Motion Editing

For more details, please refer to our project page.

🔗 Citation

If you find this code useful for your research, please use the following BibTeX entry.

@inproceedings{chen2025perception,
  title={Perception-as-Control: Fine-grained Controllable Image Animation with 3D-aware Motion Representation},
  author={Chen, Yingjie and Men, Yifang and Yao, Yuan and Cui, Miaomiao and Bo, Liefeng},
  journal={arXiv preprint arXiv:2501.05020},
  website={https://chen-yingjie.github.io/projects/Perception-as-Control/index.html},
  year={2025}}

Acknowledgements

We would like to thank the contributors to Moore-AnimateAnyone, Depth-Anything-V2, SpaTracker, Tartanvo, diffusers for their open research and exploration.