Tracking Dexterous Hand–Object Manipulation from Human Demonstration

October 10, 2025 · View on GitHub

Official repository for the Dexterous HO Tracker 2025 challenge, including baseline implementations, data preparation scripts, and evaluation utilities.

🛠️ Installation

Steps:
  1. Clone the repository and initialize submodules: 
    git clone https://github.com/ManipTrans/ManipTrans.git
git submodule init && git submodule update
  2. Create a virtual environment named maniptrans with Python 3.8. Note that IsaacGym only supports Python versions up to 3.8. 
    conda create -y -n maniptrans python=3.8
conda activate maniptrans
pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117
  3. Download IsaacGym Preview 4 from the official website and follow the installation instructions in the documentation. Test the installation by running an example script, such as joint_monkey.py, located in the python/examples directory.
  4. Install additional dependencies. 
    pip install git+https://github.com/ZhengyiLuo/smplx.git
pip install git+https://github.com/KailinLi/bps_torch.git
pip install fvcore~=0.1.5
pip install --no-index --no-cache-dir pytorch3d==0.7.3 -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py38_cu117_pyt1131/download.html
pip install -r requirements.txt
pip install -e . # include the current directory in the Python path. Or use: `export PYTHONPATH=.:$PYTHONPATH`
pip install numpy==1.23.5 # downgrade numpy to 1.23.5 to avoid compatibility issues

📋 Prerequisites

Download data for HO-Tracker dataset

Download the HO-Tracker dataset from Hugging Face and extract it to data/HO-Tracker.

MISC

Refer to the ManipTrans README for additional prerequisites and environment notes.

▶️ Training

Design your own training tasks based on the provided examples. The commands below will help you get started:

# To train the ManipTrans baseline:
python main/dataset/mano2dexhand.py --data_idx 0f900@0 --side left --dexhand inspire --headless --iter 7000

python main/rl/train.py task=ResDexHand dexhand=inspire side=LH headless=true num_envs=4096 learning_rate=2e-4 test=false randomStateInit=true rh_base_model_checkpoint=assets/imitator_rh_inspire.pth lh_base_model_checkpoint=assets/imitator_lh_inspire.pth dataIndices=[0f900@0] actionsMovingAverage=0.4 experiment=baseline

Train your model(s) on the data/HO-Tracker/data/test_sample set.

Note: Evaluation compliance. Save checkpoints under runs/ following the naming pattern: runs/{your exp tag}_{seq id (do not modify)}_{dexhand (i.e. inspire)}_{hand side (e.g. rh, lh, or bih)}__{timestamp}/nn/last_{your exp tag}_ep_{#epoch}_xxxx.pth.

▶️ Evaluation

After training, evaluate your model with:

# To eval the ManipTrans baseline:
python main/rl/eval_rollout.py --tag baseline --dexhand inspire --extra "rh_base_model_checkpoint=assets/imitator_rh_inspire.pth lh_base_model_checkpoint=assets/imitator_lh_inspire.pth"
# You can modify the arguments / rollout code according to your needs.

For scoring saved rollouts:

# To eval the scores of the saved rollouts:
python main/rl/eval_score.py

You will obtain summary metrics similar to:

================ Overall Results ================
Number of successful sequences: X
Average success rate: Single hand:  X, Bi-hand:  X
Average et (cm):  X
Average er (degree):  X
Average ej (cm):  X
Average eft (cm):  X