ComfyUI-IG-Motion-I2V

September 24, 2024 · View on GitHub

ComfyUI implementation of Motion-I2V This is currently a diffusers wrapper with code adapted from https://github.com/G-U-N/Motion-I2V

Updates

[2024/9/24] 🔥 First Release
[2024/9/23] 🔥 Interactive Motion Painter UI for ComfyUI
[2024/9/20] 🔥 Added basic IP Adapter integration
[2024/9/16] 🔥 Uodated model code to be compatible with Comfy's diffusers version

MI2V Flow Predictor takes as input a first frame and option motion prompt, mask and vectors. Outputs a predicted optical flow for a 16 frame animation with the input image as the first frame. You can view a preview of the motion where the colors correspond to movement in 2 Dimensions
MI2V Flow Animator takes the predicted flow and a starting image and generates a 16 frame animation based on these
MI2V Motion Painter allows you to draw motion vectors onto an image to be used by MI2V Flow Predictor
MI2V Pause allows you to pause the execution of the workflow. Useful for loading a resized image into MI2V Flow Predictor or checking you like the predicted motion before committing to further animation

Here are some example workflows. They can be found along with their input images here:

Motion Painter Here we use motion painting to instruct the binoculars to be lowered. This causes the man's face to be shown, however the model doesn't know what this face should look like as it wasn't in the initial image. To help, we give it a second image of the man's face via a simple IP Adapter (Once the nodes are ComfyUI native this will work even better if we use Tiled IP Adapter)

https://github.com/user-attachments/assets/1e35fb84-f246-4725-bfa9-6bfd955a2ee0
Using motion as an input into another animation Here we use an image of a forest and a prompt of "zoom out" to get a simple video of zooming out motion. This is then input into a second Animate Diff animation to give us complex and controlled motion that would have been difficult to acheive otherwise

https://github.com/user-attachments/assets/27d51c9c-ecfd-4387-9769-fd7d12ff3e07

Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling by Xiaoyu Shi^1*, Zhaoyang Huang^1*, Fu-Yun Wang^1*, Weikang Bian^1*, Dasong Li ¹, Yi Zhang¹, Manyuan Zhang¹, Ka Chun Cheung², Simon See², Hongwei Qin³, Jifeng Dai⁴, Hongsheng Li¹ ¹CUHK-MMLab ²NVIDIA ³SenseTime ⁴ Tsinghua University