Welcome to the official code repository for IndEgo, a NeurIPS 2025 Datasets & Benchmarks Track accepted dataset and open-source framework for industrial egocentric vision, designed to support training, real-time guidance, process improvement, and collaboration.

🔍 Key Features

• 3000+ egocentric videos, 1000+ exocentric videos
• Task steps, audio narration, SLAM, gaze, motion data
• Reasoning-based video QA benchmark
• Annotated collaborative sequences with tools and workspace layout

🛠️ Environment Setup

Create and install from requirements.txt

# Create a new virtual environment
python3 -m venv $HOME/indego_env
source $HOME/indego_env/bin/activate

# Install dependencies
pip install -r requirements.txt

📦 Dataset Structure

Each Category includes:

• Egocentric + Exocentric videos
• Gaze, motion, hand-pose logs
• Narrations (where applicable)
• Keysteps and mistakes (if any)
• SLAM data (missing for some sequences)

⚙️ Data Processing

We utilise the following state-of-the-art tools and pipelines:

• Eye-Gaze: Generated using the open-weight model from Meta Reality Labs.
  🔗 Project Aria Eye Tracking

• SLAM & Motion: 6DoF trajectories and semi-dense point clouds were computed using Machine Perception Services (MPS).
  🔗 Project Aria MPS Documentation

• Transcripts & Narration: Time-stamped audio transcriptions were generated using WhisperX for accurate alignment.
  🔗 WhisperX Repository

🤝 Related Works

This repository builds upon and integrates components from several open-source projects and pretrained models. We gratefully acknowledge the contributions of the following repositories and their authors:

• facebookresearch/projectaria_tools – for egocentric device support and video/sensor tooling
• DAMO-NLP-SG/VideoLLaMA3 for baseline evaluations.
• QwenLM/Qwen3 for baseline evaluations.
• OpenGVLab/InternVL for baseline evaluations.

This project also leverages the open-source AI ecosystem, including 🤗 Hugging Face Transformers, FlashAttention, Decord, and other publicly released models and frameworks.

We thank these communities for making research reproducible and accessible.

🧩 Citation

If you use the IndEgo dataset or code in your research, please cite our paper:

@inproceedings{Chavan2024IndEgo,
  author    = {Vivek Chavan and Yasmina Imgrund and Tung Dao and Sanwantri Bai and Bosong Wang and Ze Lu and Oliver Heimann and J{\"o}rg Kr{\"u}ger},
  title     = {IndEgo: A Dataset of Industrial Scenarios and Collaborative Work for Egocentric Assistants},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track},
  year      = {2024},
  url       = {https://neurips.cc/virtual/2025/poster/121501}
}

🏆 Acknowledgments & Funding

This work is funded by the German Federal Ministry of Research, Technology and Space (BMFTR) and the German Aerospace Center (DLR) under the KIKERP project (Grant No. 16IS23055C) in the KI4KMU program. We thank the Meta AI team and Reality Labs for the Project Aria initiative, including the research kit, the open-source tools and related services. The data collection for this study was carried out at the IWF research labs and the test field at TU Berlin. Lastly, we sincerely thank the student volunteers and workers who participated in the data collection process.

             ˆ