HOI-M³ Dataset Toolbox

HOI-M³: Capture Multiple Humans and Objects Interaction within Contextual Environment
CVPR 2024 (Highlight)

[Paper]  •  [Video]  •  [Project Page]  •  [Videos — HuggingFace]  •  [Annotations — Google Drive]

HOI-M³ is a large-scale dataset for modeling the interactions of multiple humans and multiple objects within realistic, contextual environments. This toolbox provides everything to download, preprocess, and visualize the dataset: multi-view image extraction, instance-mask reading/validation, MHR→SMPL-X conversion, and rendering of the human/object annotations.

This is the HOI-M³ toolbox. For the single-person, dome-captured HODome / NeuralDome dataset (CVPR 2023) see the separate NeuralDome Toolbox.

Dataset at a glance

🚩 Updates

• Jul 1, 2024 — Due to the large mask size, the annotated masks are initially released for view 3 only; the full masks + this toolbox's regeneration pipeline recover all 42 views.
• Jun 30, 2024 — Note: object rotations in the initial release were saved as the transpose of the rotation matrix — transpose them back before use.
• Jun 12, 2024 — HOI-M³ dataset upload to Google Drive.

📖 Setup

conda create -n hoim3 python=3.10 -y
conda activate hoim3
conda install pytorch==2.4.0 torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia

# PyTorch3D (for the visualization toolkit)
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable"

pip install -r requirements.txt

📦 Download & data structure

The dataset is split across two hosts:

• Videos (~5.85 TB, 204 seqs × 42 views) → HuggingFace: JuzeZhang/HOI-M3
• Everything else (masks, calibration, mocap/object poses, scanned objects, metadata) → Google Drive: HOI-M3 annotations

# videos (HuggingFace)
huggingface-cli download JuzeZhang/HOI-M3 --repo-type dataset --local-dir HOI-M3
# annotations (Google Drive) — then extract any tars
for f in *.tar; do tar -xf "$f"; done

HOI-M3/
├─ videos/                {seq}/videos/{view}.mp4        # 42 views, 4K HEVC
├─ images/                {seq}/{view}/{frame:06d}.jpg   # extracted (see below)
├─ mask/                  released masks (all 42 views)
├─ calib_ground_refined/  {date}/calibration.json        # refined PINHOLE calib (42 views)
├─ calib_with_distortion/ {date}/calibration.json        # distortion-aware calib (42 views; more accurate)
├─ mocap/                 per-frame object 6-DoF poses (ground frame)
├─ scanned_object/        object meshes
├─ dataset_information.json     # seq → capture date, per-scene metadata
└─ startframe.json

Sequences map to a capture date via dataset_information.json; the matching camera calibration lives in {calib}/{date}/calibration.json. Each view is { K: 3×3, RT: 3×4 world→camera, distCoeff: 5-param OpenCV [k1,k2,p1,p2,k3], imgSize } at the 4K capture resolution — rescale K by image_height / imgSize_height if you work at a lower resolution (the images shipped/extracted are 720p). distCoeff is dimensionless (normalized coords) — do NOT rescale it; use it raw with the rescaled K.

Two calibrations (pick one — never mix)

Hard pairing. The two calibrations live in different world frames (4–8 cm apart). Always use a calibration with its own fit set — e.g. calib_ground_refined + smplx/, or calib_with_distortion

• smplx_with_distortion/. Never cross them.

🛠️ Pipeline

1. Extract images from videos

python scripts/video2image.py --root_path /path/to/HOI-M3

2. Instance masks

The dataset ships instance masks for all 42 views (person + each object). The authoritative format is the merged mask_shards/{seq}/ (LZ4 + bit-packed, 1080p), read by the SequenceShardReaders — see Reading the masks below. If you have raw per-frame NPZ masks, pack them into the shard format:

python scripts/convert_masks_npz_to_lz4.py --src_root /path/to/HOI-M3 --seq <seq>

3. Reading the masks

from scripts.utils.mask_io import SequenceShardReaders, load_frame_masks_shard_full, load_frame_masks_shard

readers = SequenceShardReaders("/path/to/HOI-M3/mask_shards/bedroom_data01")
print(readers.objects, readers.views, readers.height, readers.width)
# e.g. ['person0','person1','bed',...]  42  1080  1920

# all 42 views of one frame, per object  ->  {obj: uint8 (V, H, W)}, values 0/1
masks = load_frame_masks_shard_full(readers, frame_id=0)
person0 = masks["person0"]        # (42, 1080, 1920)

# or only specific views (cheaper)  ->  {obj: {view_idx: (H, W)}}
masks_v = load_frame_masks_shard(readers, frame_id=0, view_indices=[0, 7, 14])
readers.close()

4. Mask validity check

# single sequence
python scripts/multi_view_mask_check.py --root_path /path/to/HOI-M3 --seq_name bedroom_data01 \
  --output_path /path/to/HOI-M3/mask_validity --mask_format shard \
  --mask_root /path/to/HOI-M3/mask_shards --all_views --batch_size 128
# all sequences (skips already-processed; --no_skip_existing to force)
python scripts/multi_view_mask_check_all.py --root_path /path/to/HOI-M3 \
  --output_path /path/to/HOI-M3/mask_validity --mask_format shard \
  --mask_root /path/to/HOI-M3/mask_shards --all_views --batch_size 128
# visualize the validity (single seq / all seqs)
python scripts/visualize_mask_validity.py --root_path /path/to/HOI-M3 --seq_name bedroom_data01 \
  --validity_path /path/to/HOI-M3/mask_validity --output_path /path/to/HOI-M3/mask_validity_vis --combined --step 20
python scripts/visualize_mask_validity_all.py --root_path /path/to/HOI-M3 \
  --validity_path /path/to/HOI-M3/mask_validity --output_path /path/to/HOI-M3/mask_validity_vis --combined --step 20

5. Visualization toolkit

Render the multi-view human + object annotations onto the images (PyTorch3D):

python scripts/hoim3_visualization.py --root_path /path/to/HOI-M3 \
  --seq_name bedroom_data01 --resolution 720 --output_path /path/to/out --vis_view 0

viz/ also provides a PyTorch3D wrapper (pyt3d_wrapper.py) and a human–object contact visualizer (contact_viz.py).

6. Splits & utilities

python scripts/generate_train_test_split.py       # train/test sequence split
python scripts/extract_sequence_contents.py        # per-sequence content inventory

SMPL-X body fits & visualization

The dataset ships mono MHR per-view body estimates and multi-view fitted human parameters (MHR) plus per-frame object 6-DoF poses (ground frame; remember the transpose fix). For interoperability we convert the multi-view MHR fits to standard SMPL-X.

Converted SMPL-X human meshes overlaid on the multi-view video — bedroom / diningroom / fitnessroom.

MHR → SMPL-X conversion

scripts/mhr_to_smplx.py forwards the multi-view MHR fit and runs MHR's official convert_mhr2smpl (per person), writing one packed NPZ per person:

python scripts/mhr_to_smplx.py --seqs bedroom_data01 \
  --output-dir /path/to/HOI-M3/smplx --max-frames-per-call 8192

Output {seq}_person{id}.npz — T-stacked SMPL-X axis-angle arrays: frame_ids (T,) · transl (T,3) · global_orient (T,3) · body_pose (T,63) · left_hand_pose (T,45) · right_hand_pose (T,45) · jaw_pose/leye_pose/reye_pose (T,3) · betas (T,10) · expression (T,10) · fitting_errors (T,). In the ground/world frame; validated at ~1 cm mean fitting error. (--max-frames-per-call chunks long sequences to bound GPU memory.)

Using the SMPL-X parameters

Build the model with the exact flags the converter used, then forward per frame:

import numpy as np, torch, smplx
m = smplx.create('/path/to/smplx_models', model_type='smplx', gender='neutral',
                 use_pca=False, flat_hand_mean=True, num_betas=10, num_expression_coeffs=10)
d = np.load('smplx/bedroom_data01_person0.npz')
t = 0
out = m(betas=torch.tensor(d['betas'][t:t+1]), global_orient=torch.tensor(d['global_orient'][t:t+1]),
        transl=torch.tensor(d['transl'][t:t+1]), body_pose=torch.tensor(d['body_pose'][t:t+1]),
        left_hand_pose=torch.tensor(d['left_hand_pose'][t:t+1]),
        right_hand_pose=torch.tensor(d['right_hand_pose'][t:t+1]),
        jaw_pose=torch.tensor(d['jaw_pose'][t:t+1]), expression=torch.tensor(d['expression'][t:t+1]))
verts = out.vertices[0].detach().numpy()   # world/ground frame; project with calib_ground_refined K,RT

Visualizing (humans + objects)

Overlay the meshes on the real camera images via pyrender (world→camera handled internally; K rescaled to the image resolution). Runs in an env with smplx + pyrender.

All four visualizers take --calib {ground_refined, with_distortion} (default ground_refined):

• ground_refined (default) — refined pinhole calib; images used as-is.
• with_distortion — the more accurate calib; the frame is undistorted with cv2.undistort (rescaled K, raw distCoeff) before the mesh is overlaid, and the mesh is projected with the same K. Pair it with the *_with_distortion fit sets (a warning prints if the loaded fits are the non-distortion set — they live in a different world frame).

# humans + objects together, animated single-view overlay -> mp4/gif
PYOPENGL_PLATFORM=egl python scripts/visualize_human_object.py \
  --seq bedroom_data01 --view 5 --obj_source objpose_v3 \
  --start_frame 0 --end_frame 600 --step 10 --width 480 --fps 10 --out out.gif
# humans only (animated) -> the gallery GIFs above
PYOPENGL_PLATFORM=egl python scripts/visualize_smplx_pyrender.py \
  --seq bedroom_data01 --view 0 --start_frame 0 --end_frame 360 --step 6 --width 480 --fps 10 --out out.gif
# objects only (still or --anim); --source {objpose_v3, mocap_ground}
PYOPENGL_PLATFORM=egl python scripts/visualize_objpose.py \
  --seq bedroom_data01 --frame 0 --view 7 --source objpose_v3 --out objects.png
# static multi-view SMPL-X grid (sanity-check alignment across views)
PYOPENGL_PLATFORM=egl python scripts/visualize_smplx_grid.py \
  --seq bedroom_data01 --frame 0 --views 0 7 14 21 28 35 --out grid.png
# same grid using the distortion-aware calibration (image gets undistorted first)
PYOPENGL_PLATFORM=egl python scripts/visualize_smplx_grid.py \
  --seq bedroom_data01 --frame 0 --views 0 7 14 21 28 35 --calib with_distortion --out grid_ud.png

Object poses come from object/{seq}_object.npz (ground frame; object_source=mocap_ground) or the refined objpose_v3/ fit; the human meshes from smplx/. With --calib with_distortion, use the matching smplx_with_distortion/ + mhr_withdist/ fits.

The SMPL-X parameters are in the 3D ground/world frame (resolution-independent). See the project page and paper for the multi-view fitting method.

📖 Citation

@inproceedings{zhang2024hoi,
  title={HOI-M3: Capture Multiple Humans and Objects Interaction within Contextual Environment},
  author={Zhang, Juze and Zhang, Jingyan and Song, Zining and Shi, Zhanhe and Zhao, Chengfeng and Shi, Ye and Yu, Jingyi and Xu, Lan and Wang, Jingya},
  booktitle={CVPR},
  year={2024}
}

If you also use the single-person dome dataset, please cite NeuralDome:

@inproceedings{zhang2023neuraldome,
  title={NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions},
  author={Juze Zhang and Haimin Luo and Hongdi Yang and Xinru Xu and Qianyang Wu and Ye Shi and Jingyi Yu and Lan Xu and Jingya Wang},
  booktitle={CVPR},
  year={2023}
}

License

See LICENSE.