EgoEMG

A multimodal egocentric dataset with bilateral surface EMG and vision for hand pose estimation
EMG-to-pose · vision-to-pose · EMG+vision fusion

[ Web · Setup · Training · Visualization · Results · License · Citation ]

✨ Highlights

• 📦 Multi-modal dataset & benchmark — 53 participants, 18+ h of synchronized recordings: bilateral wristband EMG, IMU, egocentric RGB, RGB-D, and mocap joint angles.
• 🧠 EMG-to-pose — EMGFormer: 13.8° Avg MAE on EgoEMG (M); 12.3° User+Stage MAE on the EMG2Pose benchmark (S) — ahead of all prior methods.
• 👁️ Vision-to-pose — 5.85° with ResNet-18 on the egocentric webcam stream.
• 🔀 EMG+vision fusion — 5.41° with ResNet-18 fusion: 7.5% better than vision-only at only +3M parameters.

🎯 Headline results

MAE in degrees. * = released-checkpoint values (reproduce with the commands in Results and evaluation); unstarred values are paper-reported. Avg = average over the Gesture/User/Both test splits (per-sample-weighted for EMG-to-pose, unweighted split mean for vision and fusion); the EMG2Pose row follows the benchmark's User+Stage protocol.

Full per-method and per-split breakdowns are in the Results and evaluation section below.

⚙️ Setup

conda env create -f environment.yml && conda activate egoemg
pip install -e '.[viz]'

Download data and checkpoints with Asset Setup, then set export EGOEMG_ROOT=/path/to/egoemg_assets.

🚀 Training

# EMG-to-pose (EMGFormer-M)
python -m egoemg.train \
  experiment=emgformer/regression_egoemg \
  'trainer.devices=[0,1,2,3,4,5]' '+trainer.strategy=ddp' \
  batch_size=500

# Vision-only (ResNet-18)
python -m egoemg.train \
  experiment=fusion/vision_resnet18 \
  train=true eval=true 'trainer.devices=[0]'

# EMG+vision fusion
python -m egoemg.train \
  experiment=fusion/fusion_rn18_s_center_16ch_wl7790 \
  train=true eval=true 'trainer.devices=[0,1,2,3,4]'

# EMGFormer multi-task pretraining
python -m egoemg.train_pretrain \
  experiment=emgformer/pretrain_multitask \
  train=true eval=false 'trainer.devices=[0]'

Hardware: the EMG example uses six GPUs, fusion five. For a single-GPU smoke run, set trainer.devices=[0] and reduce batch_size.

Experiment configs live in config/experiment/{emgformer,fusion,emg2pose}/. For evaluation use egoemg.test_analysis (fusion configs default train=true). The EMG+vision fusion model (MidFusionPoseFormer) is detailed in Fusion Architecture.

🎥 Visualization

python scripts/viz/visualize_dataset.py vision \
  --memmap-dir ${EGOEMG_ROOT}/data/EgoEMG_full_memmap \
  --allintra-root ${EGOEMG_ROOT}/data/EgoEMG_videos \
  --crops-dir ${EGOEMG_ROOT}/data/EgoEMG_crops \
  --data-root ${EGOEMG_ROOT}/data \
  --output-dir /tmp/egoemg_vision_viz \
  --episode-id episode_000000 --stride 10 --max-frames 300

Sample predictions on the preview shard's three episodes — a vision + MANO-mesh overlay, embedded for each as a native video player (uploaded via Attach files).

episode_000020 · wmh · train / user / gesture / both

https://github.com/user-attachments/assets/86ab0035-fccc-4117-a66d-f82a5a12e9e6

episode_000008 · zbk · user / both

https://github.com/user-attachments/assets/80e4646c-51d2-43c4-9d1f-5d63b06e82f7

episode_000028 · wsj · train / gesture

https://github.com/user-attachments/assets/66da0936-609e-4e83-9077-a626f9b1fe9b

Preview / small dataset

Every workflow above also runs end-to-end on a small preview shard instead of the full EgoEMG_full_memmap. It is a 3-episode, v3-schema memmap in the same flat layout as the published dataset_egoemg_unified, so it drops in as a mini dataset root: point the same commands at its data/memmap_data.

# EMG-to-pose eval (EMGFormer-M, 8-ch) on the shard
python -m egoemg.test_analysis \
  experiment=emgformer/egoemg_emgformer_middle \
  'checkpoint=checkpoints/egoemg_emgformer_middle.ckpt' \
  egoemg_unified_memmap_dir=$EGOEMG_ROOT/data/dataset_egoemg_preview/data/memmap_data \
  'trainer.devices=[0]' \
  datamodule.per_dataset_norm_stats_path=assets/per_dataset_norm_stats_unified.json

# Vision overlay (episode_000028 is in the shard; needs MANO model files per
# docs/ASSET_SETUP.md §3 — pass --mano-model-path or place them under
# $EGOEMG_ROOT/data/mano_data/models)
python scripts/viz/visualize_dataset.py vision \
  --memmap-dir $EGOEMG_ROOT/data/dataset_egoemg_preview/data/memmap_data \
  --allintra-root $EGOEMG_ROOT/data/dataset_egoemg_preview/data/webcam_videos \
  --crops-dir $EGOEMG_ROOT/data/dataset_egoemg_preview/data/pre-crop_webcam_videoframes \
  --data-root $EGOEMG_ROOT/data/dataset_egoemg_preview/data \
  --episode-id episode_000028 --stride 10 --max-frames 300 \
  --mano-model-path $EGOEMG_ROOT/data/mano_data/models

# Smoke training (1 epoch, 2 batches; small batch_size for a single GPU)
python -m egoemg.train experiment=emgformer/egoemg_emgformer_small \
  egoemg_unified_memmap_dir=$EGOEMG_ROOT/data/dataset_egoemg_preview/data/memmap_data \
  'trainer.devices=[0]' 'trainer.max_epochs=1' batch_size=8 \
  '+trainer.limit_train_batches=2' '+trainer.limit_val_batches=0' \
  datamodule.per_dataset_norm_stats_path=assets/per_dataset_norm_stats_unified.json

📊 Results and evaluation

Evaluation

S/M/L in this README map to the small/middle/large config and checkpoint variants (e.g. EMGFormer-M → egoemg_emgformer_middle.ckpt).

export EGOEMG_ROOT=/absolute/path/to/dataset_root

# EgoEMG EMGFormer-M (per-group stats + overall)
python -m egoemg.test_analysis \
  experiment=emgformer/egoemg_emgformer_middle \
  'checkpoint=checkpoints/egoemg_emgformer_middle.ckpt'

# EMG2Pose benchmark (EMGFormer-S)
python -m egoemg.test_analysis \
  experiment=emgformer/emg2pose_emgformer_small \
  'checkpoint=checkpoints/emg2pose_emgformer_small.ckpt' \
  data_location=${EGOEMG_ROOT}/data/emg2pose_memmap

# Vision-only ResNet-18
python -m egoemg.test_analysis \
  experiment=fusion/vision_resnet18 \
  'checkpoint=checkpoints/vision_resnet18.ckpt'

# Vision-only ViT-S/14
python -m egoemg.test_analysis \
  experiment=fusion/vision_vit_small \
  'checkpoint=checkpoints/vision_vit_small.ckpt'

# Fusion (ResNet-18 + 16ch EMG, WL 7790)
python -m egoemg.test_analysis \
  experiment=fusion/fusion_rn18_s_center_16ch_wl7790 \
  'checkpoint=checkpoints/fusion_resnet18_emgfusion_center.ckpt'

# Fusion (ViT-S/14 + 16ch EMG, WL 7790)
python -m egoemg.test_analysis \
  experiment=fusion/fusion_vits_s_center_eval_released \
  'checkpoint=checkpoints/fusion_vit_emgfusion_center.ckpt'

🗂️ Repository Layout

egoemg/        models, datasets, training/eval, vendored UmeTrack FK
config/        Hydra experiments ({emgformer,fusion,emg2pose}/ + lineage/)
scripts/       data conversion, downloads, visualization, paper figures
experiments/   shell launchers for the paper's experiments
docs/          asset setup, support scope, config & fusion architecture
assets/        EMG layout figures and normalization statistics

📜 License

Code: MIT. Third-party material (incl. UmeTrack, CC-BY-NC-4.0) under their own terms — see THIRD_PARTY_NOTICES.md.

📖 Citing EgoEMG

The paper is under review. Until a public version is available, cite this repository and the exact commit — GitHub renders CITATION.cff.

@misc{egoemg2026,
  title        = {EgoEMG: A multimodal egocentric dataset with bilateral surface
                  EMG and vision for hand pose estimation},
  author       = {Zhenqi Shi and others},
  howpublished = {GitHub repository},
  note         = {https://github.com/zhenqis123/EgoEMG},
  year         = {2026},
}