README.md

Optimizing 4D Gaussians for Dynamic Scene Video
from Single Landscape Images

In-Hwan Jin · Haesoo Choo · Seong-Hun Jeong · Park Heemoon · Junghwan Kim · Oh-joon Kwon · Kyeongbo Kong
ICLR 2025

Project Page ICLR Paper Arxiv Paper Source Code Poster

Generated Dynamic Scene Video from 3D-MOM.

Setup

Environment Setup

Clone the source code of this repo.

git clone https://github.com/InHwanJin/3D-MOM.git
cd 3D-MOM
git submodule update --init --recursive

Installation through pip is recommended. First, set up your Python environment:

conda create -n 3D-MOM python=3.7 
conda activate 3D-MOM

Make sure to install CUDA and PyTorch versions that match your CUDA environment. We've tested on NVIDIA GeForce RTX 3090 with PyTorch version 1.13.1. Please refer https://pytorch.org/ for further information.

pip install torch

The remaining packages can be installed with:

pip install -r requirements.txt
pip install -e submodules/depth-diff-gaussian-rasterization
pip install -e submodules/simple-knn

Download Checkpoints

We use the pre-trained Flow Estimation Model and Video Generator Model. You can download them at 3d-cinemagraphy for the Flow Estimation Model and StyleCineGAN for the Video Generator Model.

After downloading, place the models in the ckpts folder inside the respective directories under thirdparty.

Preprocess Data

Firstly, use labelme to specify the target regions (masks) and desired movement directions (hints):

conda activate 3D-MOM
cd demo/scene_0/
labelme image.png

A screenshot here:

It is recommended to specify short hints rather than long hints to avoid artifacts. Please follow labelme for detailed instructions if needed.

After that, we can obtain an image.json file. Our next step is to convert the annotations stored in JSON format into datasets that can be used by our method:

# this will generate a folder image_json.
labelme_json_to_dataset image.json
cd ../../
python scripts/generate_mask.py --inputdir demo/0/image_json

💫Training

For generate multi-view images and optimize 3D motion, run

# First, generate multi-view image and flow from single image.
python train_motion.py --input_dir demo/scene_0

• input_dir: input folder that contains src images.

For optimize 4D Gaussians and rendering dynamic scene video, run

# Second, reconstruct 4D scene from generated images and motions. 
python train_4DGS.py --input_dir demo/scene_0 --flow_scale 2
# Finally, render. 
python render_4DGS.py --input_dir demo/scene_0

• flow_scale: scale difference 3D motion(Point Cloud) and flow on animated video.

Results will be saved to the input_dir/video.

Acknowledgements

This code is based on several existing projects: LucidDreamer, 4DGaussians, StyleCineGAN and 3d-cinemagraphy. In particular, we used LucidDreamer as the foundational framework for our study. We extend our gratitude to the authors of these works for their contributions. 😊

📖 Citation

If you find this code useful for your research, please consider to cite our paper:)

@inproceedings{jinoptimizing,
  title={Optimizing 4D Gaussians for Dynamic Scene Video from Single Landscape Images},
  author={Jin, In-Hwan and Choo, Haesoo and Jeong, Seong-Hun and Heemoon, Park and Kim, Junghwan and Kwon, Oh-joon and Kong, Kyeongbo},
  booktitle={The Thirteenth International Conference on Learning Representations},
  year={2025}
}