Overview

EgoTraj is a large-scale, multimodal egocentric trajectory dataset designed to advance research in first-person trajectory forecasting and assistive AR navigation. Collected using Meta Quest Pro (MQPro) headsets in real-world urban environments, EgoTraj provides synchronized RGB video, 6DoF head pose, per-frame 3D eye gaze vectors, and structured scene annotations from 75 participants navigating self-chosen routes across sidewalks, crosswalks, and busy streets.

Why EgoTraj?

Existing human trajectory prediction research relies heavily on bird's-eye view (BEV) or static-camera datasets that capture where people move but not how they perceive, plan, and initiate their motion from a first-person perspective. The few egocentric trajectory datasets that exist are limited in scale, often collected from a single participant, restricted to indoor environments, or lack synchronized gaze data. EgoTraj bridges this gap by being the first large-scale egocentric trajectory dataset to jointly provide:

• Synchronized 6DoF head pose at 30 Hz
• Per-frame 3D eye gaze vectors with pixel-level calibration
• Egocentric RGB video (1024 x 1024, 30 fps)
• VLM-generated scene annotations for navigation-relevant context
• Data from 75 diverse participants (14 nationalities, ages 18--38, gender-balanced)

---

Pipeline Overview

The EgoTraj pipeline encompasses the full workflow from data collection through egocentric trajectory prediction:

1. Data Collection → Participants wear MQPro headsets and navigate urban environments while the system records synchronized multimodal streams via a custom Unity application paired with a Python recording script.

2. Data Processing → Raw sensor data (50 Hz) and RGB video (30 fps) are temporally aligned, synchronized to a common 30 Hz timeline, privacy-filtered using EgoBlur, and packaged into per-session HDF5 files.

3. Analysis & Annotation → Scene annotations are generated using Qwen2.5-VL-7B, gaze is calibrated to pixel coordinates, and the EgoViz Dashboard enables frame-level quality control.

4. Trajectory Prediction → State-of-the-art baselines are benchmarked on egocentric trajectory forecasting using multimodal inputs (ego-motion, gaze, scene, social context).

---

Dataset Details

Recording Setup

Each participant wears a Meta Quest Pro headset operating in full-color passthrough mode. The headset integrates:

• A passthrough RGB camera (1024 x 1024 @ 30 fps)
• Two infrared eye-tracking cameras
• Four inside-out tracking cameras for visual-inertial SLAM
• A 6-axis IMU

A custom Unity application interfaces with the built-in SLAM system and records time-synchronized data at 30 Hz, including 6DoF head pose, 3D gaze origin and direction vectors, and egocentric RGB video. Participants use the MQPro controller to start/stop recording.

Recording Protocol

• Participants navigate between 7 predefined outdoor waypoints across urban areas
• Routes are self-chosen (not scripted), enabling naturalistic behavior
• Sessions are capped at 15 minutes (~8 min average)
• A researcher follows at a safe distance for safety monitoring
• Participants obey traffic rules and navigate real crosswalks, sidewalks, and busy streets

Participant Diversity

---

Dataset Comparison

EgoTraj compared against existing egocentric trajectory datasets:

Dataset	Year	Setting	Hours	Frames	Subjects	Gaze	6DoF	Scene Ann.
KrishnaCam	2016	Outdoor	70.0	7.6M	1	-	-	-
EgoMotion	2016	In+Out	9.1	65.5K	N/P	-	-	-
FPL	2018	Outdoor	4.5	162K	N/P	-	-	-
Nymeria	2024	In+Out	300	32.4M	264	Y	Y	Y
EgoNav	2024	In+Out	3.3	237.6K	N/P	-	Y	-
LookOut	2025	In+Out	4.0	288K	N/P	Y	Y	-
EgoCogNav	2025	In+Out	6.0	432K	17	Y	Y	-
EgoTraj (Ours)	2026	Outdoor	10.7	1.15M	75	Y	Y	Y

---

Multimodal Streams

EgoTraj provides rich, synchronized multimodal data per frame:

Each row shows a frame at consecutive timesteps. From left to right: egocentric RGB with gaze fixation (red dot), relative depth (Depth Anything V2), semantic segmentation (OneFormer), detected poses (YOLOv8-Pose) ranked by depth, and ground-truth vs. predicted trajectory.

Gaze Calibration

A per-session quadratic calibration model maps 3D gaze yaw-pitch angles to pixel coordinates (u, v) in the video frame, enabling direct projection of gaze fixation points into image space.

Scene Annotation

Structured egocentric scene descriptions are generated using Qwen2.5-VL-7B-Instruct at 1 fps, targeting navigation-relevant elements: surrounding context, traffic activity, gaze fixation targets, and inferred movement intent.

• 96% structural compliance (with chain-of-thought retries)
• 93% inter-annotator agreement (verified by two human annotators)

---

EgoViz Dashboard

We developed EgoViz, an interactive visualization and inspection tool for the EgoTraj dataset, synchronizing four complementary views: trajectory plot, BEV path, egocentric RGB frame, and VLM scene annotation.

EgoViz ships with this repo as ego_viz.py. See Dataset Download & Format below to get the data and run it.

---

Benchmarking Results

Baselines

We evaluate five methods on egocentric trajectory prediction (1.5s observation → 3.5s prediction):

Model	ADE (m) ↓	FDE (m) ↓	L1_head ↓
Const_Vel	0.24	0.35	0.82
Lin_Ext	0.26	0.39	1.39
M_Transformer	0.20	0.32	0.74
CXA-Transformer	0.19	0.29	0.69
EgoCast	0.16	0.28	0.78

Generalization Across Splits

To examine how well the multimodal models generalize beyond the random-participant split, we evaluated CXA-Transformer on two additional, stricter splits. The waypoint-pair held-out split reserves 3 of the 21 origin–destination pairs entirely for testing (n = 10 sessions), and the unfamiliar split reserves 8 of the 31 participants who reported being unfamiliar with the recording environment (n = 8 sessions). By construction, each session is contributed by a unique participant, so all three splits are subject-disjoint.

Generalization across three test splits using CXA-Transformer. Values are ADE/FDE in meters with 95% bootstrap confidence intervals from 1000 resamples. Best per split in bold.

Key findings: The full multimodal configuration (Y + P + S + G) remains the strongest across all three splits, with a modest generalization gap (random-participant ADE 0.12 → waypoint-held-out ADE 0.14 → unfamiliar ADE 0.14). This indicates that the multimodal cues transfer to held-out landmark pairs and to participants unfamiliar with the area, rather than overfitting to specific route templates.

Qualitative Results

Trajectory predictions from multiple baselines on three test scenarios. Left: gentle segment. Center: moderate turn. Right: sharp ~90 degree intersection turn where all baselines underestimate the turning magnitude.

---

Egocentric Trajectory Prediction Demo

Egocentric pedestrian trajectory prediction with projected gaze (red dot), detected human poses, depth estimation, and predicted future path overlaid on the egocentric RGB stream.

---

Dataset Download & Format

Download: EgoTraj dataset (UT Box)

EgoTraj is distributed as two lightweight files plus the RGB video (hosted separately).

egotraj_dataset.h5

One HDF5 group per session (named by capture timestamp). Each session contains:

• Head pose — pose/position (N×3, meters), pose/rotation (N×4 quaternion, qw,qx,qy,qz), pose/velocity, pose/angular_velocity, pose/timestamp (epoch seconds).
• Gaze — gaze/direction (N×3 unit vectors) and gaze/origin (N×3).
• Video pointers — video/segment, video/frame, video/has_video, mapping each pose sample to a frame in the blurred video.
• Route waypoints (group attributes) — waypoint_start, waypoint_end, from, to.

Coordinate note: Y is the up-axis (height). The ground plane is (x, z). Positions are session-relative — each session has its own local origin — and are not absolute or geographic coordinates.

egotraj_annotations.json

A list of per-frame scene annotations sampled at 1 fps. Each record has session, pose_idx (the join key back into the H5), timestamp, second, gaze_dot (the detected gaze-marker location, or null if off-frame), and annotation (a short scene description covering context, the gaze target, and inferred intent).

To join an annotation to its trajectory and gaze, index the session's H5 datasets at pose_idx.

Video (egotraj_videos_part1.zip … egotraj_videos_part5.zip)

Privacy-blurred egocentric video, organized into per-session folders and referenced by the H5 video/segment / video/frame pointers. The red gaze dot is rendered into the video at capture time.

⚠️ Download all five parts together. The video is split across five archives (egotraj_videos_part1.zip through egotraj_videos_part5.zip); the set is only complete with all five. Download every part and extract them into a single shared videos/ folder, then point EgoViz at it with --videos-root videos/.

Using the EgoViz Dashboard

ego_viz.py is an interactive dashboard that synchronizes four views for a session: the local trajectory plot, the full-path minimap, the egocentric video frame, and the scene annotation.

Install dependencies:

pip install -r requirements.txt

tkinter ships with standard CPython; on Debian/Ubuntu install it with apt-get install python3-tk.

Run on a session:

python ego_viz.py \
  --h5 egotraj_dataset.h5 \
  --session 20251020_163423 \
  --videos-root videos/ \
  --annotations-json egotraj_annotations.json

Flag	Description
--h5	Path to egotraj_dataset.h5.
--session	Session key = HDF5 group name (e.g. 20251020_163423).
--videos-root	Parent folder of per-session video subfolders (each holding video_*_part*.mp4); enables the session dropdown. Use --videos <folder> to point at a single session's videos instead.
--annotations-json	Path to egotraj_annotations.json; the matching annotation is shown for the current second.

Trajectory-only mode works without video or annotations — just pass --h5 and --session.

Controls: ←/→ step by one point, ↑/↓ by 10, PgUp/PgDn by 100, Home/End jump to start/end; mouse wheel zooms, right-click + drag pans; press H or ? for the full shortcut list (including QC/editing keys), Q/Esc to quit.

---

Citation

If you find this work useful, please cite our paper:

@inproceedings{yehia2026egotraj,
  title     = {EgoTraj: Real-World Egocentric Human Trajectory Dataset for Multimodal Prediction},
  author    = {Yehia, Ahmad and Mohamed, Abduallah and Wang, Tianyi and Byeon, Jiseop and Qian, Kun and Jiao, Junfeng and Claudel, Christian},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026},
  eprint    = {2605.19004},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV}
}

---

License

This project is licensed under the MIT License. See LICENSE for details.

---

Acknowledgements

This work was supported by Honda Development & Manufacturing of America, LLC. We thank Jorge Monsivais, Haithi Donahue, and Steven Feit of the Emerging Technology Department for their contributions. We are also grateful to Kristen Grauman at UT Austin for her guidance throughout the data collection.