[ECCV 2026] MoGe4D: Geometry-Aware Single-Image 4D Synthesis via Dense Trajectory Generation

Yanran Zhang^*,1, Ziyi Wang^*,1, Wenzhao Zheng^†,1, Zheng Zhu², Jie Zhou¹, Jiwen Lu¹

¹Department of Automation, Tsinghua University     ²GigaAI

^*Equal Contribution    ^†Corresponding Author

📝 Abstract

Generating interactive and dynamic 4D scenes from a single static image remains a core challenge. Most existing generate-then-reconstruct and reconstruct-then-generate methods decouple geometry from motion, causing spatiotemporal inconsistencies and poor generalization.

To address this, we present MoGe4D (Motion and Geometry-aware image-to-4D synthesis), a geometry-conditioned framework that models a scene as dense 4D point trajectories. Instead of treating geometry and dynamics as two disconnected stages, MoGe4D starts from an initial geometric prior inferred from the input image and predicts future time-varying trajectories in a diffusion process, improving spatiotemporal coherence while preserving structural stability.

We contribute:

• 🗄️ TrajScene-60K — a large-scale dataset of 60,000 video samples with dense 4D point trajectories (3M+ frames, ~12B 3D point annotations).
• 🎯 4D-STraG (4D Scene Trajectory Generator) — a diffusion model that predicts geometry-consistent, motion-plausible trajectory fields, with depth-guided motion normalization and a Motion Perception Module (MPM).
• 🎬 4D-ViSM (4D View Synthesis Module) — renders the generated 4D representation into videos under arbitrary camera trajectories.

🔥 News

• [2026-07-06] model checkpoints, and the TrajScene-60K dataset are now available on HuggingFace and ModelScope.
• [2026-06-20] 🎉 MoGe4D is accepted to ECCV 2026!
• [2025-12-06] Code is released.
• [2025-12-05] Paper submitted to arXiv.

🔧 Getting Started

Installation

1. Clone the repository:

   git clone https://github.com/Zhangyr2022/MoGe4D.git
   cd MoGe4D
   

2. Create a conda environment with Python 3.10:

   conda create -n MoGe4D python=3.10
   conda activate MoGe4D
   

3. Install PyTorch (CUDA 12.4 recommended) and dependencies:

   conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia
   pip install -r requirements.txt
   

4. Install third-party dependencies:
   • UniDepthV2 — follow UniDepth
   • Gaussian Splatting — follow diff-gaussian-rasterization

Download Pretrained Checkpoints

We release the MoGe4D weights on both ModelScope and HuggingFace. Download them into the ./models folder.

Option A — ModelScope (recommended for users in mainland China):

pip install modelscope
modelscope download --model YanranZhang/MoGe4D --local_dir ./models

Option B — HuggingFace:

pip install huggingface_hub
huggingface-cli download Yanran21/MoGe4D --local-dir ./models --resume-download

After download, ./models should contain:

models/
├── 4D-STraG/diffusion_pytorch_model.safetensors   # 4D trajectory generator
├── 4D-ViSM/lora_diffusion_pytorch_model.safetensors  # view-synthesis LoRA
└── VAE/                                            # motion-sensitive VAE (+ training states)

In addition, training also requires the base backbones Wan2.1-Fun-V1.1-14B-Control / Wan2.1-Fun-V1.1-14B-InP, OmniMAE, and UniDepth, placed under ./models.

📊 TrajScene-60K Dataset

TrajScene-60K is a large-scale 4D scene dataset curated from WebVid-10M via VLM-based filtering (CogVLM2 & DeepSeek-V3). It provides the multi-modal supervision needed to learn consistent 4D scene representations and trajectories.

Download:

pip install modelscope
modelscope download --dataset YanranZhang/TrajScene-60K --local_dir ./data/TrajScene-60K

Directory structure — each sample is grouped into a range archive <range>.tar.gz (50 samples per range), covering the full 60K samples:

<range>.tar.gz                 # e.g. 000001_000050.tar.gz  (scenes 000001–000050)
└── <range>/
    ├── <id>.mp4                     # source video
    ├── <id>.txt                     # caption (scene content + dynamics)
    ├── <id>_dt3d_pred.pkl           # dense 4D point trajectories
    ├── <id>_dt3d_render.mp4         # rendered multi-view video
    ├── <id>_mask_render.mp4         # occlusion mask video
    └── <id>_mask_render_binary.npy  # binary occlusion mask

🚀 Usage

Training

bash scripts/4D_STraG_training/train_vae.sh

bash scripts/4D_STraG_training/train_wan.sh

bash scripts/4D_ViSM_training/train.sh

Inference

bash scripts/inference/infer.sh        # whole pipeline (image → 4D scene → multi-view videos)
bash scripts/inference/infer_vae.sh    # VAE only

🎨 Results Showcase

Input	4D Point Tracking (4D-STraG)	Multi-View Videos (4D-ViSM)
A brown bear walks across rocky terrain.
A camel walks along a path in a sunny zoo enclosure.

💡 Methodology

MoGe4D couples geometric modeling with motion generation through two core components:

• 4D-STraG — a diffusion-based trajectory generator that predicts geometry-consistent and motion-plausible 4D point trajectories conditioned on an initial geometric prior, using depth-guided motion normalization and a Motion Perception Module.
• 4D-ViSM — a view-synthesis module that renders the generated 4D point-cloud representation into videos along arbitrary camera trajectories.

🙏 Acknowledgments

We thank the open-source community, especially Wan2.1, Omnivore, and VideoX-Fun.

📖 Citation

If you find our work useful, please consider citing:

@inproceedings{zhang2026moge4d,
  title={Geometry-Aware Single-Image 4D Synthesis via Dense Trajectory Generation},
  author={Zhang, Yanran and Wang, Ziyi and Zheng, Wenzhao and Zhu, Zheng and Zhou, Jie and Lu, Jiwen},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2026}
}

📧 Contact

For questions and discussions, please open an issue or contact:

• Yanran Zhang: GitHub · zhangyr21@mails.tsinghua.edu.cn
• Ziyi Wang: Homepage