extra_human_video.md

Process video of human interaction data

We provide the videos of human interaction data for further studies. These raw videos are not used as direct input of planner and motion skills.

Download

Download additional dataset

mv RHINO-Human-Intention/human_data deploy_control/

Segmentation the video by label

# Split mp4 file into slides 
python scripts/split_human_data.py  # generates `rgb_imgs.hdf5 in sub directories

Aligning human video to processed motion data

in RHINO-Human-Intention/data_id.csv, each line describes the corresponding src data path of each data index in react_data/motions_processed

For example, the first line in RHINO-Human-Intention/data_id.csv is:

1,human_data/train_1026/thumbup/20241026-151856/splited_data/idle/0

indicating that the first data in react_data (namely all 1.npys and 1.hdf5s) is processed from human_data/train_1026/thumbup/20241026-151856/splited_data/idle/0/rgb_imgs.hdf5

In each rgb_imgs.hdf5 file, we store the rgb images from zed camera in zed_imgs; rgb images from human camera in human_imgs; and rgb images from humanoid camera in humanoid_imgs.

Extract more info from svo (Optional)

This is the example script of extracting rgb video from .svo files, you can modify the script and extract more information.

# convert .svo to .mp4 files
cd deploy_control
python scripts/process_demo_video_zed_rgb.py --root human_data/train_1026/
python scripts/process_demo_video_zed_rgb.py --root human_data/train_1030/
python scripts/process_demo_video_zed_rgb.py --root human_data/train_1129/
python scripts/process_demo_video_zed_rgb.py --root human_data/train_1211/
python scripts/process_demo_video_zed_rgb.py --root human_data/train_1224/
python scripts/process_demo_video_zed_rgb.py --root human_data/eval_1/
python scripts/process_demo_video_zed_rgb.py --root human_data/eval_2/