:sunglasses: Awesome 3D and 4D World Models

July 29, 2026 · View on GitHub

Awesome Logo arXiv Visitors PR's Welcome

:sunglasses: Awesome 3D and 4D World Models

This survey reviews state-of-the-art 3D and 4D world models - systems that learn, predict, and simulate the geometry and dynamics of real environments from multi-modal signals.

We unify terminology, scope, and evaluations, and organize the space into three complementary paradigms by representation:

Learn generative or predictive models from sequential video streams with geometric and temporal constraints. VideoGen focuses on long-horizon consistency, controllability, and scene-level generation, enabling agents to imagine or forecast plausible video rollouts.
Model 3D/4D occupancy grids that encode geometry and semantics in voxel space. OccGen provides a physics-consistent scaffold for robust perception, forecasting, and simulation, bridging low-level sensor data and high-level reasoning.
Leverage point cloud sequences from LiDAR sensors to generate or predict geometry-grounded scenes. LiDARGen emphasizes high-fidelity 3D structure, robustness to environment changes, and applications in safety-critical domains such as autonomous driving.

For more details, kindly refer to our paper and project page. :rocket:

:books: Citation

If you find this work helpful for your research, please kindly consider citing our papers:

@article{survey_3d_4d_world_models,
    title   = {{3D} and {4D} World Modeling: A Survey},
    author  = {Lingdong Kong and Wesley Yang and Jianbiao Mei and Youquan Liu and Ao Liang and Dekai Zhu and Dongyue Lu and Wei Yin and Xiaotao Hu and Mingkai Jia and Junyuan Deng and Kaiwen Zhang and Yang Wu and Tianyi Yan and Shenyuan Gao and Song Wang and Linfeng Li and Liang Pan and Yong Liu and Jianke Zhu and Wei Tsang Ooi and Steven C. H. Hoi and Ziwei Liu},
    journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)},
    year    = {2026}
}

@inproceedings{worldlens,
    title     = {{WorldLens}: Full-Spectrum Evaluations of Driving World Models in Real World},
    author    = {Ao Liang and Lingdong Kong and Tianyi Yan and Hongsi Liu and Wesley Yang and Ziqi Huang and Wei Yin and Jialong Zuo and Yixuan Hu and Dekai Zhu and Dongyue Lu and Youquan Liu and Guangfeng Jiang and Linfeng Li and Xiangtai Li and Long Zhuo and Lai Xing Ng and Benoit R. Cottereau and Changxin Gao and Liang Pan and Wei Tsang Ooi and Ziwei Liu},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    pages     = {36385-36399},
    year      = {2026}
}

Table of Contents

Background

World modeling has become a cornerstone of modern AI, enabling agents to understand, represent, and predict dynamic environments. While prior research has focused primarily on 2D images and videos, the rapid emergence of native 3D and 4D representations (e.g., RGB-D, occupancy grids, LiDAR point clouds) calls for a dedicated study.

What Are Native 3D Representations?

Unlike 2D projections, native 3D/4D signals directly encode metric geometry, visibility, and motion in the physical coordinates where agents act. Examples include:

  • RGB-D imagery (2D images with depth channels)
  • Occupancy grids (voxelized maps of free vs. occupied space)
  • LiDAR point clouds (3D coordinates from active sensing)
  • Neural fields (e.g., NeRF, Gaussian Splatting)

What Are World Models in 3D and 4D?

A 3D/4D world model is an internal representation that allows an agent to imagine, forecast, and interact with its environment in the 3D space.

Generative World Models:
synthesize plausible 3D/4D worlds under conditions (e.g., text prompts, trajectories).
Predictive World Models:
anticipate the future evolution of 3D/4D scenes given past observations and actions.

Together, these models provide the foundation for simulation, planning, and embodied intelligence in complex environments.

1. Benchmarks & Datasets

Benchmarks

WorldLensVBenchWorldScore

Workshops

ThemeVenueDateLocationRecording
Workshop on 4D World Models: Bridging Generation and ReconstructionCVPR 2026TBDDenver-
The 2nd Workshop on World ModelsICLR 2026April 23, 2026Rio de Janeiro-
Workshop on World Modeling-February 4-6, 2026Montréal-
Workshop on Embodied World Models for Decision MakingNeurIPS 2025December 6, 2025San Diego-
Workshop on Reliable and Interactable World Models: Geometry, Physics, Interactivity and Real-World GeneralizationICCV 2025October 19, 2025Hawai'i-
Workshop on Building Physically Plausible World ModelsICML 2025July 19, 2025Vancouver-
Workshop on Assessing World ModelsICML 2025July 18, 2025Vancouver-
Workshop on Benchmarking World ModelsCVPR 2025June 12, 2025Nashville-
Workshop on World Models: Understanding, Modelling and ScalingICLR 2025April 28, 2025Singapore-
Workshop on Foundation Models for Autonomous SystemsCVPR 2024June 17, 2025Seattle[YouTube]

Datasets

:timer_clock: In chronological order, from the earliest to the latest.

ModelPaperVenueWebsite
KITTIAre We Ready for Autonomous Driving? The KITTI Vision Benchmark SuiteCVPR 2012Website
NYUv2Indoor Segmentation and Support Inference from RGBD ImagesECCV 2012Website
CARLAarXiv
CARLA: An Open Urban Driving Simulator		CoRL 2017Website
SemanticKITTIarXiv
SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences		ICCV 2019Website
nuScenesarXiv
nuScenes: A Multimodal Dataset for Autonomous Driving		CVPR 2020Website
Waymo OpenarXiv
Scalability in Perception for Autonomous Driving: Waymo Open Dataset		CVPR 2020Website
STFarXiv
Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather		CVPR 2020Website
Virtual KITTI 2arXiv
Virtual KITTI 2		arXiv 2020Website
Argoverse 2arXiv
Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting		NeurIPS 2021Website
Lyft-Level5arXiv
One Thousand and One Hours: Self-Driving Motion Prediction Dataset		CoRL 2021Website
nuPlanarXiv
nuPlan: A Closed-Loop ML-Based Planning Benchmark for Autonomous Vehicles		CVPRW 2021Website
PandaSetarXiv
PandaSet: Advanced Sensor Suite Dataset for Autonomous Driving		ITSC 2022Website
OpenCOODarXiv
OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication		ICRA 2022Website
KITTI-360arXiv
KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D		TPAMI 2022Website
CarlaSCarXiv
MotionSC: Data Set and Network for Real-Time Semantic Mapping in Dynamic Environments		RA-L 2022Website
Robo3DarXiv
Robo3D: Towards Robust and Reliable 3D Perception against Corruptions		ICCV 2023Website
OpenOccupancyarXiv
OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception		ICCV 2023Website
Occ3D-nuScenesarXiv
Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving		NeurIPS 2023Website
OpenDV-YouTubearXiv
GenAD: Generalized Predictive Model for Autonomous Driving		CVPR 2024Website
SSCBencharXiv
SSCBench: A Large-Scale 3D Semantic Scene Completion Benchmark for Autonomous Driving		IROS 2024Website
NAVSIMarXiv
NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking		NeurIPS 2024Website
DrivingDojoarXiv
DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Model		NeurIPS 2024Website
EUVSarXiv
Extrapolated Urban View Synthesis Benchmark		ICCV 2025Website
Pi3DETarXiv
Perspective-Invariant 3D Object Detection		ICCV 2025Website
DrivingGenarXiv
DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving		arXiv 2026-
World-in-WorldarXiv
World-in-World: World Models in a Closed-Loop World		arXiv 2025-
4DWorldBencharXiv
4DWorldBench: A Comprehensive Evaluation Framework for 3D/4D World Generation Models		arXiv 2025-
DriveE2EarXiv
DriveE2E: Closed-Loop Benchmark for End-to-End Autonomous Driving through Real-to-Simulation		arXiv 2025-
nuPlan-RarXiv
nuPlan-R: A Closed-Loop Planning Benchmark for Autonomous Driving via Reactive Multi-Agent Simulation		arXiv 2025-
ReactSim-BencharXiv
ReactSim-Bench: Benchmarking Reactive Behavior World Model Simulation in Autonomous Driving		arXiv 2026-
Nuplan-OccarXiv
Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method		arXiv 2025-
4DLidarOpenarXiv
4DLidarOpen: An Open 4D FMCW Lidar Dataset for Motion-Aware Autonomous Driving		arXiv 2026-
OccInteract-85karXiv
OccDirector: Language-Guided Behavior and Interaction Generation in 4D Occupancy Space		arXiv 2026-
TerraZeroarXiv
TerraZero: Procedural Driving Simulation for Zero-Demonstration Self-Play at Scale		arXiv 2026-
AdmissibilityarXiv
Validate the Dream Before You Trust Its Verdict: Admissibility for World-Model Simulators		arXiv 2026-
Seriality GaparXiv
The Seriality Gap in Video Diffusion Models		arXiv 2026-
VISAarXiv
VISA: VLM-Guided Instance Semantic Auditing for 3D Occupancy World Models		arXiv 2026-

2. World Modeling from Video Generation

:one: Data Engines

:timer_clock: In chronological order, from the earliest to the latest.

ModelPaperVenueWebsiteGitHub
BEVControlarXiv
BEVControl: Accurately Controlling Street-View Elements with Multi-Perspective Consistency via BEV Sketch Layout		arXiv 2023--
BEVGenarXiv
Street-View Image Generation from a Bird's-Eye View Layout		RA-L 2024WebsiteGitHub
MagicDrivearXiv
MagicDrive: Street View Generation with Diverse 3D Geometry Control		ICLR 2024WebsiteGitHub
PanaceaarXiv
Panacea: Panoramic and Controllable Video Generation for Autonomous Driving		CVPR 2024WebsiteGitHub
DrivingDiffusionarXiv
DrivingDiffusion: Layout-Guided Multi-View Driving Scene Video Generation with Latent Diffusion Model		ECCV 2024WebsiteGitHub
WoVoGenarXiv
WoVoGen: World Volume-Aware Diffusion for Controllable Multi-Camera Driving Scene Generation		ECCV 2024-GitHub
DelphiarXiv
Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generation		arXiv 2024WebsiteGitHub
SimGenarXiv
SimGen: Simulator-Conditioned Driving Scene Generation		NeurIPS 2024WebsiteGitHub
BEVWorldarXiv
BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latents		arXiv 2024--
Panacea+arXiv
Panacea+: Panoramic and Controllable Video Generation for Autonomous Driving		arXiv 2024Website-
DiVEarXiv
DiVE: DiT-Based Video Generation with Enhanced Control		arXiv 2024WebsiteGitHub
MyGoarXiv
MyGo: Consistent and Controllable Multi-View Driving Video Generation with Camera Control		arXiv 2024--
SyntheOccarXiv
SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D Semantic MPIs		arXiv 2024WebsiteGitHub
HoloDrivearXiv
HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Driving		arXiv 2024--
CogDrivingarXiv
Seeing Beyond Views: Multi-View Driving Scene Video Generation with Holistic Attention		arXiv 2024Website-
UniMLVGarXiv
UniMLVG: Unified Framework for Multi-View Long Video Generation with Comprehensive Control Capabilities for Autonomous Driving		arXiv 2024-GitHub
DrivePhysicaarXiv
Physical Informed Driving World Model		arXiv 2024Website-
DriveDreamer-2arXiv
DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation		AAAI 2025WebsiteGitHub
SubjectDrivearXiv
SubjectDrive: Scaling Generative Data in Autonomous Driving via Subject Control		AAAI 2025Website-
GladarXiv
Glad: A Streaming Scene Generator for Autonomous Driving		ICLR 2025-GitHub
DualDiffarXiv
DualDiff: Dual-Branch Diffusion Model for Autonomous Driving with Semantic Fusion		ICRA 2025-GitHub
UniScenearXiv
UniScene: Unified Occupancy-Centric Driving Scene Generation		CVPR 2025WebsiteGitHub
DriveScapearXiv
DriveScape: Towards High-Resolution Controllable Multi-View Driving Video Generation		CVPR 2025Website-
PerLDiffarXiv
PerLDiff: Controllable Street View Synthesis Using Perspective-Layout Diffusion Models		ICCV 2025WebsiteGitHub
MagicDrive-V2arXiv
MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control		ICCV 2025Website-
DINO-ForesightarXiv
DINO-Foresight: Looking into the Future with DINO		NeurIPS 2025WebsiteGitHub
Cosmos-Transfer1arXiv
Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control		arXiv 2025WebsiteGitHub
DualDiff+arXiv
DualDiff+: Dual-Branch Diffusion for High-Fidelity Video Generation with Reward Guidance		arXiv 2025-GitHub
CoGenarXiv
CoGen: 3D Consistent Video Generation via Adaptive Conditioning for Autonomous Driving		arXiv 2025Website-
NoiseControllerarXiv
NoiseController: Towards Consistent Multi-View Video Generation via Noise Decomposition and Collaboration		arXiv 2025--
STAGEarXiv
STAGE: A Stream-Centric Generative World Model for Long-Horizon Driving-Scene Simulation		arXiv 2025--
WMRewardarXiv
Inference-time Physics Alignment of Video Generative Models with Latent World Models		arXiv 2026--
AutoScapearXiv
AutoScape: Geometry-Consistent Long-Horizon Scene Generation		arXiv 2025--
Rethinking-DWMarXiv
Rethinking Driving World Model as Synthetic Data Generator for Perception Tasks		arXiv 2025--
OmniDrivearXiv
OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generation		arXiv 2026--
OpenLongTailarXiv
OpenLongTail: Generative Scaling of Long-Tail Driving Data		arXiv 2026--
OmniSCSarXiv
OmniSCS: Omni Safety-Critical Scenario Synthesis for Autonomous Driving via a Fully Editable Driving World		arXiv 2026--
DriveWeaverarXiv
DriveWeaver: Point-Conditioned Video Inpainting for Controllable Vehicle Insertion in Autonomous Driving Simulation		arXiv 2026--

:two: Action Interpreters

:timer_clock: In chronological order, from the earliest to the latest.

ModelPaperVenueWebsiteGitHub
GAIA-1arXiv
GAIA-1: A Generative World Model for Autonomous Driving		arXiv 2023Website-
ADriver-IarXiv
ADriver-I: A General World Model for Autonomous Driving		arXiv 2023--
Drive-WMarXiv
Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving		CVPR 2024WebsiteGitHub
DriveDreamerarXiv
DriveDreamer: Towards Real-World-Driven World Models for Autonomous Driving		ECCV 2024WebsiteGitHub
GenADarXiv
GenAD: Generalized Predictive Model for Autonomous Driving		CVPR 2024-GitHub
GenAD (Gen-E2E)arXiv
GenAD: Generative End-to-End Autonomous Driving		ECCV 2024--
VistaarXiv
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability		NeurIPS 2024WebsiteGitHub
InfinityDrivearXiv
InfinityDrive: Breaking Time Limits in Driving World Models		arXiv 2024Website-
DrivingGPTarXiv
DrivingGPT: Unifying Driving World Modeling and Planning with Multi-Modal Autoregressive Transformers		arXiv 2024Website-
DrivingWorldarXiv
DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT		arXiv 2024WebsiteGitHub
GEMarXiv
GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control		CVPR 2025WebsiteGitHub
MaskGWMarXiv
MaskGWM: A Generalizable Driving World Model with Video Mask Reconstruction		CVPR 2025-GitHub
EponaarXiv
Epona: Autoregressive Diffusion World Model for Autonomous Driving		ICCV 2025WebsiteGitHub
VaViM & VaVAMarXiv
VaViM and VaVAM: Autonomous Driving through Video Generative Modeling		arXiv 2025WebsiteGitHub
MiLAarXiv
MiLA: Multi-View Intensive-Fidelity Long-Term Video Generation World Model for Autonomous Driving		arXiv 2025-GitHub
GAIA-2arXiv
GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving		arXiv 2025Website-
Ego-Other-WMarXiv
Other Vehicle Trajectories Are Also Needed: A Driving World Model Unifies Ego-Other Vehicle Trajectories in Video Latent Space		arXiv 2025--
DriVersearXiv
DriVerse: Navigation World Model for Driving Simulation via Multimodal Trajectory Prompting and Motion Alignment		arXiv 2025--
PosePilotarXiv
PosePilot: Steering Camera Pose for Generative World Models with Self-Supervised Depth		arXiv 2025--
ProphetDWMarXiv
ProphetDWM: A Driving World Model for Rolling Out Future Actions and Videos		arXiv 2025--
LongDWMarXiv
LongDWM: Cross-Granularity Distillation for Building A Long-Term Driving World Model		arXiv 2025WebsiteGitHub
UniDrive-WMarXiv
UniDrive-WM: Unified Understanding, Planning and Generation World Model For Autonomous Driving		arXiv 2026Website-
DriveVAarXiv
DriveVA: Video Action Models are Zero-Shot Drivers		arXiv 2026--
DeepSightarXiv
DeepSight: Long-Horizon World Modeling via Latent States Prediction for End-to-End Autonomous Driving		arXiv 2026--

:three: Neural Simulators

:timer_clock: In chronological order, from the earliest to the latest.

ModelPaperVenueWebsiteGitHub
MagicDrive3DarXiv
MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenes		arXiv 2024WebsiteGitHub
DreamForgearXiv
DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenes		arXiv 2024WebsiteGitHub
Doe-1arXiv
Doe-1: Closed-Loop Autonomous Driving with Large World Model		arXiv 2024WebsiteGitHub
DrivingSpherearXiv
DrivingSphere: Building A High-Fidelity 4D World for Closed-Loop Simulation		CVPR 2025WebsiteGitHub
UMGenarXiv
Generating Multimodal Driving Scenes via Next-Scene Prediction		CVPR 2025WebsiteGitHub
DriveArenaarXiv
DriveArena: A Closed-Loop Generative Simulation Platform for Autonomous Driving		ICCV 2025WebsiteGitHub
InfiniCubearXiv
InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Models		ICCV 2025WebsiteGitHub
DiST-4DarXiv
DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generation		ICCV 2025WebsiteGitHub
UniFuturearXiv
Seeing the Future, Perceiving the Future: A Unified Driving World Model for Future Generation and Perception		arXiv 2025WebsiteGitHub
NexusarXiv
Decoupled Diffusion Sparks Adaptive Scene Generation		arXiv 2025WebsiteGitHub
ChallengerarXiv
Challenger: Affordable Adversarial Driving Video Generation		arXiv 2025WebsiteGitHub
Cosmos-DrivearXiv
Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models		arXiv 2025WebsiteGitHub
ReSimarXiv
ReSim: Reliable World Simulation for Autonomous Driving		arXiv 2025--
OmniDreamsarXiv
NVIDIA OmniDreams: Real-Time Generative World Model for Closed-Loop Autonomous Vehicle Simulation		arXiv 2026--
Xiaomi Auto WMarXiv
Xiaomi Auto World Model: A Joint World Model Integrating Reconstruction and Generation for Autonomous Driving		arXiv 2026--
HERMES++arXiv
HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generation		arXiv 2026--
CausalDrivearXiv
CausalDrive: Real-time Causal World Models for Autonomous Driving		arXiv 2026--
Point-as-SkeletonarXiv
Point as Skeleton: Accumulated Point Cloud Enhanced Autoregressive Generation for Closed-Loop Autonomous Driving Simulation		arXiv 2026--
Cam2SimarXiv
Cam2Sim: Neural Scenario Reconstruction for Closed-Loop Autonomous Driving Simulation		arXiv 2026--
CARLA-GSarXiv
CARLA-GS: Decoupling Representation, Reasoning, and Physics Simulation for Autonomous Driving Corner-Case Synthesis		arXiv 2026--

:four: Scene Reconstructors

:timer_clock: In chronological order, from the earliest to the latest.

ModelPaperVenueWebsiteGitHub
3DGSarXiv
3D Gaussian Splatting for Real-Time Radiance Field Rendering		TOG 2023WebsiteGitHub
StreetGaussianarXiv
Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting		ECCV 2024WebsiteGitHub
4DGFarXiv
Dynamic 3D Gaussian Fields for Urban Areas		NeurIPS 2024WebsiteGitHub
SCubearXiv
SCube: Instant Large-Scale Scene Reconstruction using VoxSplats		NeurIPS 2024WebsiteGitHub
HUGSarXiv
HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting		CVPR 2024WebsiteGitHub
MagicDrive3DarXiv
MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenes		arXiv 2024WebsiteGitHub
S3GaussianarXiv
S3Gaussian: Self-Supervised Street Gaussians for Autonomous Driving		arXiv 2024WebsiteGitHub
VDGarXiv
VDG: Vision-Only Dynamic Gaussian for Driving Simulation		arXiv 2024WebsiteGitHub
UniGaussianarXiv
UniGaussian: Driving Scene Reconstruction from Multiple Camera Models via Unified Gaussian Representations		arXiv 2024--
Stag-1arXiv
Stag-1: Towards Realistic 4D Driving Simulation with Video Generation Model		arXiv 2024WebsiteGitHub
DrivingReconarXiv
DrivingRecon: Large 4D Gaussian Reconstruction Model For Autonomous Driving		arXiv 2024-GitHub
OccScenearXiv
OccScene: Semantic Occupancy-Based Cross-Task Mutual Learning for 3D Scene Generation		arXiv 2024--
SGDarXiv
SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior		WACV 2025--
OmniRearXiv
OmniRe: Omni Urban Scene Reconstruction		ICLR 2025WebsiteGitHub
DriveDreamer4DarXiv
DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation		CVPR 2025WebsiteGitHub
DeSiRe-GSarXiv
DeSiRe-GS: 4D Street Gaussians for Static-Dynamic Decomposition and Surface Reconstruction for Urban Driving Scenes		CVPR 2025-GitHub
SplatADarXiv
SplatAD: Real-Time Lidar and Camera Rendering with 3D Gaussian Splatting for Autonomous Driving		CVPR 2025WebsiteGitHub
ReconDreamerarXiv
ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restoration		CVPR 2025WebsiteGitHub
FreeSimarXiv
FreeSim: Toward Free-Viewpoint Camera Simulation in Driving Scenes		CVPR 2025Website-
StreetCrafterarXiv
StreetCrafter: Street View Synthesis with Controllable Video Diffusion Models		CVPR 2025WebsiteGitHub
FlexDrivearXiv
FlexDrive: Toward Trajectory Flexibility in Driving Scene Reconstruction and Rendering		CVPR 2025--
S-NeRF++arXiv
S-NeRF++: Autonomous Driving Simulation via Neural Reconstruction and Generation		TPAMI 2025--
InfiniCubearXiv
InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Models		ICCV 2025WebsiteGitHub
DiST-4DarXiv
Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generation		ICCV 2025WebsiteGitHub
DreamDrivearXiv
DreamDrive: Generative 4D Scene Modeling from Street View Images		arXiv 2025Website-
Uni-GaussiansarXiv
Uni-Gaussians: Unifying Camera and Lidar Simulation with Gaussians for Dynamic Driving Scenarios		arXiv 2025Website-
MuDGarXiv
MuDG: Taming Multi-Modal Diffusion with Gaussian Splatting for Urban Scene Reconstruction		arXiv 2025WebsiteGitHub
UniFuturearXiv
Seeing the Future, Perceiving the Future: A Unified Driving World Model for Future Generation and Perception		arXiv 2025WebsiteGitHub
SceneCrafterarXiv
Unraveling the Effects of Synthetic Data on End-to-End Autonomous Driving Humanoid Robots		arXiv 2025-GitHub
ReconDreamer++arXiv
ReconDreamer++: Harmonizing Generative and Reconstructive Models for Driving Scene Representation		arXiv 2025WebsiteGitHub
RealEnginearXiv
RealEngine: Simulating Autonomous Driving in Realistic Context		arXiv 2025-GitHub
GeoDrivearXiv
GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Control		arXiv 2025-GitHub
PseudoSimulationarXiv
Pseudo-Simulation for Autonomous Driving		arXiv 2025-GitHub
DreamlandarXiv
Dreamland: Controllable World Creation with Simulator and Generative Models		arXiv 2025Website-
Diff4SplatarXiv
Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Models		arXiv 2025WebsiteGitHub
SpaceTimePilotarXiv
SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time		arXiv 2025WebsiteGitHub
FLAG-4DarXiv
FLAG-4D: Flow-Guided Local-Global Dual-Deformation Model for 4D Reconstruction		arXiv 2026-GitHub
MotionCrafterarXiv
MotionCrafter: Dense Geometry and Motion Reconstruction with a 4D VAE		arXiv 2026WebsiteGitHub
WorldSplatarXiv
WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Driving		arXiv 2025--
GaussianDWMarXiv
GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generation		arXiv 2025--
RealityBridgearXiv
RealityBridge: Bridging Editable 3D Gaussian Splatting Driving Simulations and Real-World Videos		arXiv 2026--
Adaptive Gaussian GrapharXiv
Beyond Perfect Priors: Adaptive Gaussian Graph for 4D Driving Reconstruction in the Wild		arXiv 2026--
Glob3RarXiv
Glob3R: Global Structure-from-Motion with 3D Foundation Models		arXiv 2026--
NoDrift3RarXiv
NoDrift3R: Raymap-Guided Coupling for Drift-Robust Unposed Feed-Forward 3D Reconstruction		arXiv 2026--
OmniXarXiv
OmniX: Any-view and Any-time 4D Reconstruction via Feed-forward Trajectory Fields		arXiv 2026--

3. World Modeling from Occupancy Generation

:one: Scene Representors

:timer_clock: In chronological order, from the earliest to the latest.

ModelPaperVenueWebsiteGitHub
SSDarXiv
Diffusion Probabilistic Models for Scene-Scale 3D Categorical Data		arXiv 2023-GitHub
SemCityarXiv
SemCity: Semantic Scene Generation with Triplane Diffusion		CVPR 2024WebsiteGitHub
WoVoGenarXiv
WoVoGen: World Volume-Aware Diffusion for Controllable Multi-Camera Driving Scene Generation		ECCV 2024-GitHub
UrbanDiffarXiv
Urban Scene Diffusion through Semantic Occupancy Map		arXiv 2024Website-
OccGenarXiv
OccGen: Generative Multi-modal 3D Occupancy Prediction for Autonomous Driving		arXiv 2024--
DrivingSpherearXiv
DrivingSphere: Building A High-Fidelity 4D World for Closed-Loop Simulation		CVPR 2025WebsiteGitHub
UniScenearXiv
UniScene: Unified Occupancy-Centric Driving Scene Generation		CVPR 2025WebsiteGitHub
OccScenearXiv
OccScene: Semantic Occupancy-Based Cross-Task Mutual Learning for 3D Scene Generation		arXiv 2024--
InfiniCubearXiv
InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Models		ICCV 2025WebsiteGitHub
Control-3D-ScenearXiv
Controllable 3D Outdoor Scene Generation via Scene Graphs		ICCV 2025WebsiteGitHub
X-ScenearXiv
X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability		arXiv 2025WebsiteGitHub
GenieDrivearXiv
GenieDrive: Towards Physics-Aware Driving World Model with 4D Occupancy Guided Video Generation		CVPR 2026WebsiteGitHub
EditSSCarXiv
EditSSC: Toward Editable Semantic Occupancy Scenes with Unconditional Diffusion Models		arXiv 2026--

:two: Occupancy Forecasters

:timer_clock: In chronological order, from the earliest to the latest.

ModelPaperVenueWebsiteGitHub
Emergent-OccarXiv
Differentiable Raycasting for Self-supervised Occupancy Forecasting		ECCV 2022-GitHub
FF4DarXiv
Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting		CVPR 2023WebsiteGitHub
UniWorldarXiv
UniWorld: Autonomous Driving Pre-Training via World Models		arXiv 2023--
UniScenearXiv
UniScene: Multi-Camera Unified Pre-Training via 3D Scene Reconstruction for Autonomous Driving		arXiv 2023-GitHub
OccWorldarXiv
OccWorld: Learning A 3D Occupancy World Model for Autonomous Driving		ECCV 2024WebsiteGitHub
Cam4DOccarXiv
Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications		CVPR 2024-GitHub
DriveWorldarXiv
DriveWorld: 4D Pre-Trained Scene Understanding via World Models for Autonomous Driving		CVPR 2024--
OccSoraarXiv
OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Driving		arXiv 2024WebsiteGitHub
UnOarXiv
UnO: Unsupervised Occupancy Fields for Perception and Forecasting		CVPR 2024Website-
LOPRarXiv
Self-Supervised Multi-Future Occupancy Forecasting for Autonomous Driving		arXiv 2024--
FSF-NetarXiv
FSF-Net: Enhance 4D Occupancy Forecasting with Coarse BEV Scene Flow for Autonomous Driving		arXiv 2024--
OccLLaMAarXiv
OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving		arXiv 2024--
DOMEarXiv
DOME: Taming Diffusion Model into High-Fidelity Controllable Occupancy World Model		arXiv 2024WebsiteGitHub
GaussianADarXiv
GaussianAD: Gaussian-Centric End-to-End Autonomous Driving		arXiv 2024WebsiteGitHub
GaussianWorldarXiv
GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Prediction		CVPR 2025-GitHub
DFIT-OccWorldarXiv
An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-Assisted Training		arXiv 2024--
Drive-OccWorldarXiv
Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving		AAAI 2025WebsiteGitHub
PreWorldarXiv
Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving		ICLR 2025-GitHub
OccProphetarXiv
OccProphet: Pushing Efficiency Frontier of Camera-Only 4D Occupancy Forecasting with Observer-Forecaster-Refiner Framework		ICLR 2025-GitHub
RenderWorldarXiv
RenderWorld: World Model with Self-Supervised 3D Label		ICRA 2025--
Occ-LLMarXiv
Occ-LLM: Enhancing Autonomous Driving with Occupancy-Based Large Language Models		ICRA 2025--
EfficientOCFarXiv
Spatiotemporal Decoupling for Efficient Vision-Based Occupancy Forecasting		CVPR 2025--
DIOarXiv
DIO: Decomposable Implicit 4D Occupancy-Flow World Model		CVPR 2025--
T³FormerarXiv
Temporal Triplane Transformers as Occupancy World Models		arXiv 2025--
UniOccarXiv
UniOcc: A Unified Benchmark for Occupancy Forecasting and Prediction in Autonomous Driving		ICCV 2025WebsiteGitHub
COMEarXiv
COME: Adding Scene-Centric Forecasting Control to Occupancy World Model		arXiv 2025-GitHub
I²WorldarXiv
I²-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecasting		ICCV 2025-GitHub
OmniNWMarXiv
OmniNWM: Omniscient Driving Navigation World Models		arXiv 2025-GitHub
SparseOccVLAarXiv
SparseOccVLA: Bridging Occupancy and Vision-Language Models via Sparse Queries for Unified 4D Scene Understanding and Planning		arXiv 2026-GitHub
GenieDrivearXiv
GenieDrive: Towards Physics-Aware Driving World Model with 4D Occupancy Guided Video Generation		CVPR 2026WebsiteGitHub
OccTENSarXiv
OccTENS: 3D Occupancy World Model via Temporal Next-Scale Prediction		arXiv 2025--
SparseWorldarXiv
SparseWorld: A Flexible, Adaptive, and Efficient 4D Occupancy World Model Powered by Sparse and Dynamic Queries		arXiv 2025--
IR-WMarXiv
Vision-Centric 4D Occupancy Forecasting and Planning via Implicit Residual World Models		arXiv 2025--
ForecastOccarXiv
ForecastOcc: Vision-based Semantic Occupancy Forecasting		arXiv 2026--
GEM (Occ)arXiv
GEM: Gaussian Evolution Model for Occupancy Forecasting and Motion Planning		arXiv 2026--
OWMDrivearXiv
OWMDrive: Causality-Aware End-to-End Autonomous Driving via 4D Occupancy World Model		arXiv 2026--
CascadeOccarXiv
CascadeOcc: Rethinking 3D Occupancy World Models with Cascaded VQ Representations		arXiv 2026--

:three: Autoregressive Simulators

:timer_clock: In chronological order, from the earliest to the latest.

ModelPaperVenueWebsiteGitHub
SemCityarXiv
SemCity: Semantic Scene Generation with Triplane Diffusion		CVPR 2024WebsiteGitHub
XCubearXiv
XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies		CVPR 2024WebsiteGitHub
PDDarXiv
Pyramid Diffusion for Fine 3D Large Scene Generation		ECCV 2024WebsiteGitHub
OccSoraarXiv
OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Driving		arXiv 2024WebsiteGitHub
DynamicCityarXiv
DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes		ICLR 2025WebsiteGitHub
DrivingSpherearXiv
DrivingSphere: Building A High-Fidelity 4D World for Closed-Loop Simulation		CVPR 2025WebsiteGitHub
InfiniCubearXiv
InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Models		ICCV 2025WebsiteGitHub
X-ScenearXiv
X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability		arXiv 2025WebsiteGitHub
PrITTIarXiv
PrITTI: Primitive-Based Generation of Controllable and Editable 3D Semantic Scenes		arXiv 2025WebsiteGitHub
OccSimarXiv
OccSim: Multi-kilometer Simulation with Long-horizon Occupancy World Models		arXiv 2026--
AutoWorldarXiv
AutoWorld: Scaling Multi-Agent Traffic Simulation with Self-Supervised World Models		arXiv 2026--
OccDirectorarXiv
OccDirector: Language-Guided Behavior and Interaction Generation in 4D Occupancy Space		arXiv 2026--

4. World Modeling from LiDAR Generation

:one: Data Engines

:timer_clock: In chronological order, from the earliest to the latest.

ModelPaperVenueWebsiteGitHub
DUStyarXiv
Learning to Drop Points for LiDAR Scan Synthesis		IROS 2021WebsiteGitHub
LiDARGenarXiv
Learning to Generate Realistic LiDAR Point Clouds		ECCV 2022-GitHub
DUSty v2arXiv
Generative Range Imaging for Learning Scene Priors of 3D LiDAR Data		WACV 2023WebsiteGitHub
UltraLiDARarXiv
UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generation		CVPR 2023Website-
Copilot4DarXiv
Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion		ICLR 2024Website-
R2DMarXiv
LiDAR Data Synthesis with Denoising Diffusion Probabilistic Models		ICRA 2024WebsiteGitHub
ViDARarXiv
Visual Point Cloud Forecasting enables Scalable Autonomous Driving		CVPR 2024-GitHub
LiDiffarXiv
Scaling Diffusion Models to Real-World 3D LiDAR Scene Completion		CVPR 2024-GitHub
LiDMarXiv
Towards Realistic Scene Generation with LiDAR Diffusion Models		CVPR 2024-GitHub
RangeLDMarXiv
RangeLDM: Fast Realistic LiDAR Point Cloud Generation		ECCV 2024-GitHub
Text2LiDARarXiv
Text2LiDAR: Text-Guided LiDAR Point Cloud Generation via Equirectangular Transformer		ECCV 2024-GitHub
LiDARGRITarXiv
Taming Transformers for Realistic LiDAR Point Cloud Generation		arXiv 2024-GitHub
BEVWorldarXiv
BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latents		arXiv 2024-GitHub
SDSarXiv
Simultaneous Diffusion Sampling for Conditional LiDAR Generation		arXiv 2024--
DiffSSCarXiv
DiffSSC: Semantic LiDAR Scan Completion using Denoising Diffusion Probabilistic Models		IROS 2025--
HoloDrivearXiv
HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Driving		arXiv 2024--
LOGenarXiv
LOGen: Toward LiDAR Object Generation by Point Diffusion		arXiv 2024WebsiteGitHub
OLiDMarXiv
OLiDM: Object-Aware LiDAR Diffusion Models for Autonomous Driving		AAAI 2025WebsiteGitHub
X-DrivearXiv
X-Drive: Cross-Modality Consistent Multi-Sensor Data Synthesis for Driving Scenarios		ICLR 2025-GitHub
LidarDMarXiv
LidarDM: Generative LiDAR Simulation in a Generated World		ICRA 2025WebsiteGitHub
LiDAR-EDITarXiv
LiDAR-EDIT: LiDAR Data Generation by Editing the Object Layouts in Real-World Scenes		ICRA 2025WebsiteGitHub
R2FlowarXiv
Fast LiDAR Data Generation with Rectified Flows		ICRA 2025WebsiteGitHub
WeatherGenarXiv
WeatherGen: A Unified Diverse Weather Generator for LiDAR Point Clouds via Spider Mamba Diffusion		CVPR 2025-GitHub
LiDPMarXiv
LiDPM: Rethinking Point Diffusion for Lidar Scene Completion		IV 2025WebsiteGitHub
HERMESarXiv
HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation		ICCV 2025WebsiteGitHub
SuperPCarXiv
SuperPC: A Single Diffusion Model for Point Cloud Completion, Upsampling, Denoising, and Colorization		CVPR 2025Website-
SPIRALarXiv
SPIRAL: Semantic-Aware Progressive LiDAR Scene Generation and Understanding		NeurIPS 2025WebsiteGitHub
SG-LDMarXiv
SG-LDM: Semantic-Guided LiDAR Generation via Latent-Aligned Diffusion		arXiv 2025--
TopoLiDMarXiv
TopoLiDM: Topology-Aware LiDAR Diffusion Models for Interpretable and Realistic LiDAR Point Cloud Generation		IROS 2025--
3DiSSarXiv
Towards Generating Realistic 3D Semantic Training Data for Autonomous Driving		arXiv 2025-GitHub
Distill-DPOarXiv
Diffusion Distillation With Direct Preference Optimization For Efficient 3D LiDAR Scene Completion		arXiv 2025-GitHub
DriveXarXiv
DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Driving		arXiv 2025--
OpenDWMarXiv
OpenDWM: Open Driving World Models		arXiv 2025-GitHub
RadarGenarXiv
RadarGen: Automotive Radar Point Cloud Generation from Cameras		arXiv 2025WebsiteGitHub
La La LiDARarXiv
La La LiDAR: Large-Scale Layout Generation from LiDAR Data		AAAI 2026--
LiDARCrafterarXiv
LiDARCrafter: Dynamic 4D World Modeling from LiDAR Sequences		AAAI 2026WebsiteGitHub
VeilaarXiv
Veila: Panoramic LiDAR Generation from a Monocular RGB Image		ICRA 2026--
R3DPAarXiv
R3DPA: Leveraging 3D Representation Alignment and RGB Pretrained Priors for LiDAR Scene Generation		arXiv 2026--
L3DRarXiv
L3DR: 3D-aware LiDAR Diffusion and Rectification		arXiv 2026--
OmniLiDARarXiv
OmniLiDAR: A Unified Diffusion Framework for Multi-Domain 3D LiDAR Generation		arXiv 2026--
LiDARDraftarXiv
LiDARDraft: Generating LiDAR Point Cloud from Versatile Inputs		arXiv 2025--
T2LDM++arXiv
T2LDM++: A Self-Conditioned Representation Guided Diffusion Model for Realistic Text-to-LiDAR Scene Generation		arXiv 2026--
Adv. LiDAR DiffusionarXiv
Adversarially Guided Diffusion for LiDAR Range Image Synthesis		arXiv 2026--
PointDiffusionarXiv
PointDiffusion: Diffusion-Based Scene Completion in the Point Cloud Domain		arXiv 2026--
PatchScenearXiv
PatchScene: Patch-based Voxel Diffusion for Large-Scale Scene Completion		arXiv 2026--

:two: Action Forecasters

:timer_clock: In chronological order, from the earliest to the latest.

ModelPaperVenueWebsiteGitHub
Copilot4DarXiv
Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion		ICLR 2024Website-
ViDARarXiv
Visual Point Cloud Forecasting enables Scalable Autonomous Driving		CVPR 2024-GitHub
BEVWorldarXiv
BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latents		arXiv 2024-GitHub
HERMESarXiv
HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation		ICCV 2025WebsiteGitHub
DriveXarXiv
DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Driving		arXiv 2025--

:three: Autoregressive Simulators

:timer_clock: In chronological order, from the earliest to the latest.

ModelPaperVenueWebsiteGitHub
HoloDrivearXiv
HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Driving		arXiv 2024--
LidarDMarXiv
LidarDM: Generative LiDAR Simulation in a Generated World		ICRA 2025WebsiteGitHub
OpenDWMarXiv
OpenDWM: Open Driving World Models		arXiv 2025-GitHub
LiDARCrafterarXiv
LiDARCrafter: Dynamic 4D World Modeling from LiDAR Sequences		arXiv 2025WebsiteGitHub
Gen-4D-LiDARarXiv
Learning to Generate 4D LiDAR Sequences		arXiv 2025--
LiSTARarXiv
LiSTAR: Ray-Centric World Models for 4D LiDAR Sequences in Autonomous Driving		arXiv 2025--
DriveLiDAR4DarXiv
DriveLiDAR4D: Sequential and Controllable LiDAR Scene Generation for Autonomous Driving		arXiv 2025--
LaGenarXiv
LaGen: Towards Autoregressive LiDAR Scene Generation		arXiv 2025--
U4DarXiv
U4D: Uncertainty-Aware 4D World Modeling from LiDAR Sequences		CVPR 2026--
GEM (LiDAR)arXiv
GEM: Generating LiDAR World Model via Deformable Mamba		arXiv 2026--

5. Applications

:one: Autonomous Driving

ModelPaperVenueWebsiteGitHub
OccSoraarXiv
OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Driving		arXiv 2024-GitHub
DFIT-OccWorldarXiv
An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-Assisted Training		arXiv 2024--
LiDARCrafterarXiv
LiDARCrafter: Dynamic 4D World Modeling from LiDAR Sequences		arXiv 2025WebsiteGitHub
UniSimarXiv
UniSim: A Neural Closed-Loop Sensor Simulator		CVPR 2023Website-
PanaceaarXiv
Panacea: Panoramic and Controllable Video Generation for Autonomous Driving		CVPR 2024WebsiteGitHub
DelphiarXiv
Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generation		arXiv 2024WebsiteGitHub
DriveDreamer-2arXiv
DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation		AAAI 2025-GitHub
Panacea+arXiv
Panacea+: Panoramic and Controllable Video Generation for Autonomous Driving		arXiv 2024Website-
MiLAarXiv
MiLA: Multi-View Intensive-Fidelity Long-Term Video Generation World Model for Autonomous Driving		arXiv 2025-GitHub
GAIA-2arXiv
GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving		arXiv 2025Website-
GenieDrivearXiv
GenieDrive: Towards Physics-Aware Driving World Model with 4D Occupancy Guided Video Generation		CVPR 2026WebsiteGitHub
AD-R1arXiv
AD-R1: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving with Impartial World Models		CVPR 2026--
Xiaomi OneVLarXiv
Xiaomi OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation		arXiv 2026--

:two: Robotics

ModelPaperVenueWebsiteGitHub
Habitat 2.0arXiv
Habitat 2.0: Training Home Assistants to Rearrange Their Habitat		arXiv 2021--
VLMPSarXiv
Visual Language Maps for Robot Navigation		ICRA 2023WebsiteGitHub
-arXiv
Foundation Models in Robotics: Applications, Challenges, and the Future		IJRR 2024-GitHub
RoboDreamerarXiv
RoboDreamer: Learning Compositional World Models for Robot Imagination		arXiv 2024WebsiteGitHub
BEHAVIORarXiv
BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activities		CoRL 2025WebsiteGitHub
BridgeV2WarXiv
BridgeV2W: Bridging Video Generation Models to Embodied World Models via Embodiment Masks		arXiv 2026Website-
PAIWorldarXiv
PAIWorld: A 3D-Consistent World Foundation Model for Robotic Manipulation		arXiv 2026--
Qwen-RobotWorldarXiv
Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation		arXiv 2026--
FLUXarXiv
FLUX: Accelerating Cross-Embodiment Generative Navigation Policies via Rectified Flow and Static-to-Dynamic Learning		arXiv 2026--
NavThinkerarXiv
NavThinker: Action-Conditioned World Models for Coupled Prediction and Planning in Social Navigation		arXiv 2026--
-arXiv
Language-Conditioned World Modeling for Visual Navigation		arXiv 2026--
-arXiv
Towards Unified World Models for Visual Navigation via Memory-Augmented Planning and Foresight		arXiv 2026--

:three: Video Games & XR

ModelPaperVenueWebsiteGitHub
ILVEarXiv
Interactive Latent Variable Evolution for the Generation of Minecraft Structures		ICFDG 2021--
ProcTHORarXiv
ProcTHOR: Large-Scale Embodied AI Using Procedural Generation		NeurIPS 2022WebsiteGitHub
WorldGPTarXiv
WorldGPT: Empowering LLM as Multimodal World Model		ACM MM 2024-GitHub
WorldExplorerarXiv
WorldExplorer: Towards Generating Fully Navigable 3D Scenes		SIGGRAPH Asia 2025-GitHub
Text2WorldarXiv
Text2World: Benchmarking Large Language Models for Symbolic World Model Generation		arXiv 2025WebsiteGitHub
FlexWorldarXiv
FlexWorld: Progressively Expanding 3D Scenes for Flexiable-View Synthesis		arXiv 2025WebsiteGitHub
Hunyuan-GameCraftarXiv
Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Condition		arXiv 2025WebsiteGitHub
HunyuanWorld 1.0arXiv
HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixels		arXiv 2025WebsiteGitHub
MGVQarXiv
MGVQ: Could VQ-VAE Beat VAE? A Generalizable Tokenizer with Multi-Group Quantization		arXiv 2025-GitHub
EvoWorldarXiv
EvoWorld: Evolving Panoramic World Generation with Explicit 3D Memory		arXiv 2025-GitHub
3D4DarXiv
3D4D: An Interactive, Editable, 4D World Model via 3D Video Generation		arXiv 2025--
ViewRopearXiv
Geometry-Aware Rotary Position Embedding for Consistent Video World Model		arXiv 2026--
MINDarXiv
MIND: Benchmarking Memory Consistency and Action Control in World Models		arXiv 2026WebsiteGitHub
Matrix-Game 3.0arXiv
Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory		arXiv 2026--
Hunyuan-GameCraft-2arXiv
Hunyuan-GameCraft-2: Instruction-following Interactive Game World Model		arXiv 2025--
SolarisarXiv
Solaris: Building a Multiplayer Video World Model in Minecraft		arXiv 2026--

:four: Digital Twins

ModelPaperVenueWebsiteGitHub
DynamicCityarXiv
DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes		ICLR 2025WebsiteGitHub
UrbanScene3DarXiv
Capturing, Reconstructing, and Simulating: the UrbanScene3D Datase		ECCV 2022WebsiteGitHub
UrbanWorldarXiv
UrbanWorld: An Urban World Model for 3D City Generation		arXiv 2024-GitHub
GaussianCityarXiv
GaussianCity: Generative Gaussian Splatting for Unbounded 3D City Generation		CVPR 2025WebsiteGitHub
SceneDiffuser++arXiv
SceneDiffuser++: City-Scale Traffic Simulation via a Generative World Model		CVPR 2025--
CityGenAgentarXiv
Imagine a City: CityGenAgent for Procedural 3D City Generation		arXiv 2026Website-
MajutsuCityarXiv
MajutsuCity: Language-driven Aesthetic-adaptive City Generation with Controllable 3D Assets and Layouts		arXiv 2025--

:five: Other Topics

ModelPaperVenueWebsiteGitHub
-arXiv
Interpreting Physics in Video World Models		arXiv 2026--
Medical World ModelsarXiv
Medical world models: representing medical states, modelling clinical dynamics and guiding intervention policies		arXiv 2026--
Cosmos 3arXiv
Cosmos 3: Omnimodal World Models for Physical AI		arXiv 2026--
PANarXiv
PAN: A World Model for General, Interactable, and Long-Horizon World Simulation		arXiv 2025--
HY-World 2.0arXiv
HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds		arXiv 2026--
Lyra 2.0arXiv
Lyra 2.0: Explorable Generative 3D Worlds		arXiv 2026--
-arXiv
Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond		arXiv 2026--
-arXiv
Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future		arXiv 2026--
PerpetualWonderarXiv
PerpetualWonder: Long-Horizon Action-Conditioned 4D Scene Generation		arXiv 2026--
INSPATIO-WORLDarXiv
INSPATIO-WORLD: A Real-Time 4D World Simulator via Spatiotemporal Autoregressive Modeling		arXiv 2026--
-arXiv
World Models: A Comprehensive Survey of Architectures, Methodologies, Reasoning Paradigms, and Applications		arXiv 2026--
-arXiv
Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends		arXiv 2026--
CP4DarXiv
CP4D: Compositional Physics-aware 4D Scene Generation		arXiv 2026--

6. Other Resources

Codebases & Toolkits

  • OpenWorldLib — A unified codebase for world models, providing a standardized pipeline interface over existing open-source models (Matrix-Game-2, Hunyuan-GameCraft, FlashWorld, Cosmos-Predict-2.5, and others) across video generation, 3D scene generation, and reasoning. Apache-2.0. GitHub

Knowledge Hubs

  • world-models.io — A structured knowledge hub for AI world models, with model profiles, research syntheses, comparisons, benchmarks, and a practical taxonomy spanning video, 3D/4D, occupancy, LiDAR, robotics, and autonomous driving. Leaderboard

Tutorials

Talks & Seminars

7. Acknowledgements