Workflow: ManipTrans (IsaacGym)

February 18, 2026 · View on GitHub

SPIDER MPC for dexterous hand-object manipulation using IsaacGym and HO-Tracker-Baseline (ManipTrans).

Quick Start (Reproducible Setup)

1. Download Data

All required data is hosted on HuggingFace. Download it into the spider directory:

sudo apt install git-lfs
git lfs install
cd /path/to/spider
git clone https://huggingface.co/datasets/retarget/retarget_example example_datasets

This downloads example_datasets/ containing:

raw/maniptrans/HO-Tracker/ --- 16 trajectory demos (8 left-hand, 8 right-hand)
raw/maniptrans/assets/ --- pre-trained base imitator checkpoints
raw/maniptrans/rl_runs/ --- 16 trained RL baseline checkpoints
raw/maniptrans/retargeting/ --- optional retargeting initialization
processed/maniptrans/ --- pre-computed rollouts, videos, and metrics (160 runs)

2. Create Data Symlink

ManipTrans uses relative paths (data/HO-Tracker/...). Create a symlink:

cd /path/to/spider
ln -sf example_datasets/raw/maniptrans data

3. Install Dependencies

Requires Python 3.8 + PyTorch 1.13.1 + IsaacGym (older than SPIDER's default Python 3.12+):

# Create conda environment
conda create -n maniptrans python=3.8
conda activate maniptrans

# Install PyTorch 1.13.1 with CUDA 11.7
pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 --extra-index-url https://download.pytorch.org/whl/cu117

# Install IsaacGym (requires NVIDIA GPU)
cd /path/to/isaacgym/python && pip install -e .

# Install pytorch3d (prebuilt wheel for py38 + cu117 + pytorch 1.13.1)
pip install pytorch3d==0.7.3 -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py38_cu117_pyt1131/download.html

# Install additional dependencies
pip install git+https://github.com/otaheri/chamfer_distance.git
pip install git+https://github.com/otaheri/bps_torch.git
pip install git+https://github.com/lixiny/manotorch.git
pip install numpy==1.23.5 gym==0.23.1 opencv-python-headless scikit-learn smplx trimesh

# Clone and install HO-Tracker-Baseline
cd /path/to
git clone https://github.com/KailinLi/HO-Tracker-Baseline.git
cd HO-Tracker-Baseline
touch requirements.txt  # Create empty file (prevents pip install error)
pip install -e .

# Install SPIDER (minimal, no-deps to avoid conflicts)
cd /path/to/spider
pip install --ignore-requires-python --no-deps -e .
pip install loguru mujoco hydra-core omegaconf imageio[ffmpeg]

4. Set Up HO-Tracker-Baseline Data

HO-Tracker-Baseline also needs access to the trajectory data and imitator checkpoints. Point its data paths to spider's example_datasets/:

# Link HO-Tracker data
cd /path/to/HO-Tracker-Baseline
rm -f data/HO-Tracker
ln -sf /path/to/spider/example_datasets/raw/maniptrans/HO-Tracker data/HO-Tracker

# Link retargeting data
rm -rf data/retargeting/HO-Tracker
ln -sf /path/to/spider/example_datasets/raw/maniptrans/retargeting/HO-Tracker data/retargeting/HO-Tracker

# Copy imitator checkpoints
cp /path/to/spider/example_datasets/raw/maniptrans/assets/imitator_*.pth assets/

5. Patch HO-Tracker-Baseline for Compatibility

The dexhandimitator module may fail to import in some environments. Patch maniptrans_envs/lib/envs/tasks/__init__.py to make it optional:

# Replace:
from .dexhandimitator import DexHandImitatorRHEnv, DexHandImitatorLHEnv

# With:
try:
    from .dexhandimitator import DexHandImitatorRHEnv, DexHandImitatorLHEnv
except ImportError:
    DexHandImitatorRHEnv = None
    DexHandImitatorLHEnv = None

6. Run SPIDER

conda activate maniptrans
cd /path/to/spider
export LD_LIBRARY_PATH="/path/to/miniconda3/envs/maniptrans/lib:${LD_LIBRARY_PATH:-}"

# Default: 0f900@0, inspire hand, left hand
python examples/run_maniptrans.py

# Different trajectory:
python examples/run_maniptrans.py task=81a95@5

# Right hand:
python examples/run_maniptrans.py task=07bb1@1 embodiment_type=right

# Fewer iterations (faster):
python examples/run_maniptrans.py max_num_iterations=16

Output is saved to example_datasets/processed/maniptrans/inspire/{side}/{task}/{data_id}/:

rollout_isaac.npz --- trajectory data
rollout_isaac.mp4 --- video (when save_video=true)
metrics_isaac.json --- evaluation metrics

Available Trajectories

16 trajectories from HO-Tracker dataset (8 per hand):

Side	Task IDs
Left	`0f900@0`, `81a95@5`, `0f4a7@1`, `70ca2@0`, `39e5e@5`, `380c7@0`, `2ddc8@2`, `0b3a0@6`
Right	`07bb1@1`, `235ac@0`, `07301@0`, `49793@0`, `01857@10`, `031b7@0`, `03ac9@0`, `155cb@0`

Configuration

Key parameters in examples/config/maniptrans.yaml:

Parameter	Default	Description
`task`	`0f900@0`	HO-Tracker data sequence
`embodiment_type`	`left`	`left` or `right`
`num_samples`	`1024`	Parallel CEM samples (IsaacGym envs)
`max_num_iterations`	`32`	CEM iterations per MPC step
`horizon`	`0.6`	Planning horizon (seconds)
`ctrl_dt`	`0.2`	Control timestep
`seed`	`0`	Random seed
`data_id`	`0`	Output subdirectory index

RL Baseline Evaluation

Trained RL checkpoints are included for all 16 trajectories. To evaluate:

cd /path/to/HO-Tracker-Baseline

# Example: evaluate left-hand trajectory
python main/rl/train.py \
    task=ResDexHand dexhand=inspire side=lh \
    headless=true num_envs=256 test=true \
    +forceFullLength=true seed=0 \
    rolloutStateInit=false randomStateInit=false \
    "dataIndices=[0f900@0]" \
    checkpoint=/path/to/spider/example_datasets/raw/maniptrans/rl_runs/baseline_0f900@0_inspire_lh/nn/baseline_0f900@0.pth \
    actionsMovingAverage=0.4 \
    rh_base_model_checkpoint=assets/imitator_rh_inspire.pth \
    lh_base_model_checkpoint=assets/imitator_lh_inspire.pth

Side tags: lh for left-hand trajectories, rh for right-hand trajectories.

Evaluation Metrics

Four metrics computed against ground-truth demo data:

Metric	Description
`e_t`	Object translation error (cm)
`e_r`	Object rotation error (degrees, geodesic)
`e_j`	Joint position error (cm)
`e_ft`	Fingertip position error (cm)

Run evaluation across all trajectories:

# Single-seed evaluation (data_id=0 only)
python spider/postprocess/evaluate_maniptrans.py

# Multi-seed evaluation (all data_ids, mean +/- std)
python spider/postprocess/evaluate_maniptrans.py --multiseed

Common Issues

DexHandFactory registration failure (KeyError: 'inspire_rh'): This is a circular import issue. SPIDER includes a fix in spider/simulators/maniptrans.py that lazily triggers hand auto-registration. Ensure you are using the latest SPIDER code.
dexhandimitator import error: Apply the patch in step 5 above to make this import optional.
pytorch3d installation: Use the prebuilt wheel for your PyTorch/CUDA combination. Building from source requires matching CUDA toolkit version (not just driver) and compatible GCC. The command in step 3 uses prebuilt wheels for py38 + cu117 + pytorch 1.13.1.
np.float deprecation (AttributeError: module 'numpy' has no attribute 'float'): Install numpy==1.23.5 which still supports the deprecated aliases.
Missing requirements.txt in HO-Tracker-Baseline: Run touch requirements.txt in the HO-Tracker-Baseline root before pip install -e ..
imageio[ffmpeg] required for video: Install with pip install imageio[ffmpeg] for video recording support.
LD_LIBRARY_PATH must include conda env lib: Without this, IsaacGym may fail to find shared libraries. Set export LD_LIBRARY_PATH="/path/to/miniconda3/envs/maniptrans/lib:${LD_LIBRARY_PATH:-}".

Important Notes

Python 3.8 Compatibility

ManipTrans requires Python 3.8 + PyTorch 1.13.1 + IsaacGym. SPIDER handles this via:

from __future__ import annotations in spider/config.py (for tuple[float, float] syntax)
hasattr(torch, "compile") guards in spider/optimizers/sampling.py

SPIDER optimizes 18-dim residual actions (6 wrist force/torque + 12 DOF positions). The base imitator provides nominal tracking actions. These combine in the env's pre_physics_step to produce 36-dim actions for the full base+residual architecture.

ManipTrans Config Fields

ManipTrans-specific fields (e.g., maniptrans_episode_length) are not part of the SPIDER Config dataclass. In examples/run_maniptrans.py, they are extracted from the Hydra config and attached as attributes after Config construction.