Appearance-Based Refinement for Object-Centric Motion Segmentation

Junyu Xie, Weidi Xie, Andrew Zisserman

Visual Geometry Group, Department of Engineering Science, University of Oxford

ECCV, 2024

The YouTubeVOS2018-motion Dataset

• YouTubeVOS2018-motion (short for YTVOS18-m, where "m" is for motion) is a subset selected from training split of YTVOS2018.
• These selected sequences are used for evaluation, with predominantly moving objects involved (i.e., objects can be discovered based on their motion).
• The list of selected sequences can be found in resources/ytvos18m_seq.json.
• The raw video frames can be downloaded from YouTubeVOS.
• The GT annotations are available here.

Pre-computed results and checkpoints

• The masks after refinements can be found here.
• The flow-predicted masks from self-supervised adapted OCLR models are provided as the inputs for the refinement, which can be found here. The corresponding checkpoint for self-supervised adapted OCLR models can be found here.
• The checkpoints for the mask selector and self-supervised adapted mask correctors can be found here.

Scripts

Requirements

pytorch>=2.0, Pillow, opencv, einops

Inference

python main.py --save_pred --dataset DAVIS17m --ckpt_selector={} --ckpt_corrector={}  \ 
                --img_dir={} --gt_dir={} --mask_dir={} --save_dir={} 

where --save_pred saves the refined masks
           --ckpt_selector and --ckpt_corrector indicate the checkpoints for the mask selector and corrector
           --img_dir and --gt_dir denote the directories for dataset images and corresponding gt annotations
           --mask_dir denote the directory of input flow-predicted masks to be refined
           --save_dir specifies the directory to save predicted masks

Citation

If you find our paper/repository helpful, please consider citing our works:

@InProceedings{xie24appearrefine,
    title     = {Appearance-Based Refinement for Object-Centric Motion Segmentation},  
    author    = {Junyu Xie and Weidi Xie and Andrew Zisserman},  
    booktitle = {ECCV},  
    year      = {2024}
}

@inproceedings{xie2022segmenting,
    title     = {Segmenting Moving Objects via an Object-Centric Layered Representation}, 
    author    = {Junyu Xie and Weidi Xie and Andrew Zisserman},
    booktitle = {NeurIPS},
    year      = {2022}
}