LiveScene: Language Embedding Interactive Radiance Fields for Physical Scene Rendering and Control

This is the official implementation for LiveScene.

LiveScene: Language Embedding Interactive Radiance Fields for Physical Scene Rendering and Control
Delin Qu*, Qizhi Chen*, Pingrui Zhang Xianqiang Gao, Binzhao, Zhigang Wang, Dong Wang†, Xuelong Li†

Paper | Demo | Video

Update

• Project Pages for LiveScene: Language Embedding Interactive Radiance Fields for Physical Scene Rendering and Control [2024-5-18]
• DATASET for LiveScene [2024-10-21]
• Video and slides [2024-11-04]
• Code for LiveScene [2025-01-09]

⚙️ Installation

1. Install Nerfstudio

Follow the official instructions to install the latest version of nerfstudio.

2. Install Livescene

Clone the repository and install the package:

git clone https://github.com/Tavish9/livescene.git
pip install -e .

📥 Download Dataset

The InterReal and OmniSim datasets are available on Huggingface.

To download the entire dataset using huggingface_cli:

huggingface-cli download --local-dir livescene_dataset --repo-type dataset IPEC-COMMUNITY/LiveScene

🚀 Running LiveScene

1. Check running options

To view available options for training:

ns-train livescene --help

use livescene-real-data for InterReal dataset and livescene-sim-data for OmniSim dataset.

2. Launch Training

You can use it like any other third-party nerfstudio project.

ns-train livescene --data /path/to/your/livescene_dataset/scene_name livescene-real/sim-data

3. Interact with viewer

Slider Interact

Use slider to control specific objects within scene.

Language Interact

Use natural language to interact with the 3D scenes.

Relevancy Map

Visualize relevancy map in the scene.

Acknowledgement

We adapt codes from some awesome repositories, including Nerfstudio, Omnigibson, Kplanes_nerfstudio, Lerf, and Conerf. Thanks for making the code available! 🤗

Citation

If you use this work or find it helpful, please consider citing: (bibtex)

@misc{livescene2024,
  title={LiveScene: Language Embedding Interactive Radiance Fields for Physical Scene Rendering and Control}, 
  author={Delin Qu, Qizhi Chen, Pingrui Zhang, Xianqiang Gao, Bin Zhao, Zhigang Wang, Dong Wang†, Xuelong Li†},
  year={2024},
  eprint={2406.16038},
  archivePrefix={arXiv},
}