CLAP is a cross-embodiment, action-conditioned video generation framework that unifies disparate human and robot action spaces via end-effector poses, language, and latent actions.

• Two-Stage Curriculum: Learns unsupervised physical priors from unlabeled video, then grounds them in end-effector space for zero-shot deployment.

• Sample-Efficient Adaptation: Facilitates few-shot transfer to target robots, matching or beating single-embodiment baselines like DROID (without post-training) and Bridge (with post-training).

• Zero-Shot Real-World Generalization: Generalizes out-of-the-box to power real-world inference planning and RL finetuning with robot policies like π₀.₅ and MolmoAct-2.

This package covers the full lifecycle: cross-embodiment video modeling, post-training, novel-embodiment adaptation, offline evaluation, and real-robot policy-in-the-loop deployment. New to the terminology? See the Glossary for EE, LAM, OXE, DROID, and other terms used throughout.

Contents

• 📢 Updates
• ✨ Features
• 📋 Requirements
• 📦 Installation
  • 1. Core package
  • 2. Policy backends (optional)
  • 3. Environment
• 🏁 Getting started
  • 📥 Download a checkpoint (optional)
  • 🔁 Replay demonstration trajectories
  • 🎮 Live Teleop (DROID, Bridge, Bimanual YAM, G1 Humanoid)
  • 🚀 Deploy (policy-in-the-loop)
• 🧩 Sample-efficient adaptation to new embodiments
• 🗂️ Checkpoints
  • 🌐 Cross-embodiment models
  • 🎯 Post-trained on a single robot platform
  • 🆕 Novel-embodiment adaptation
• 📁 Project layout
• 📚 Reference (configuration, entrypoints, dataset registry, testing — docs/REFERENCE.md)
• 🛠️ Tips and Troubleshooting
• 🙏 Acknowledgment
• 📄 BibTeX
• 📖 Glossary
• ⚖️ License

📢 Updates

• [2026/08/12] 🚀 CLAP is live.

✨ Features

CLAP delivers the most comprehensive suite of action-conditioned video world models to date — spanning diverse action-conditioning spaces (end-effector, language, and latent) and robot morphologies, across DROID, Bridge, bimanual YAM robots, and G1 humanoids.

📋 Requirements

• Python ≥ 3.9 (use 3.11 for in-process openpi policies).

• A CUDA GPU: A single ≥12GB GPU (e.g., RTX 3060) comfortably runs eval/replay/teleop/deploy; in-process openpi deployment needs ≥32GB (e.g., RTX 5090) — server mode instead runs openpi on its own machine/GPU, so clap-rollout-deploy itself only needs CLAP's own ~10GB. Inference VRAM usage and timing statistics per nominal prediction (11 total frames, 25 denoising steps, mean ± std with n = 20 trials):

  	A100-40GB	A100-80GB	H200
  CLAP alone, idle	9.6GB	9.6GB	9.7GB
  + openpi in-process, idle	17.6GB	17.6GB	17.7GB
  + openpi in-process, inference peak	24.2GB	24.2GB	25.6GB
  predict_chunk (steady-state)	3.24s ± 0.02s	2.88s ± 0.00s	1.49s ± 0.00s

  Steady-state excludes each process's first predict_chunk call, which pays a one-time cuDNN kernel-autotuning cost. How much that first call costs depends on whether anything else already touched the GPU first: for clap-rollout-deploy --in-process (the table above), openpi loads before CLAP's first prediction and absorbs part of that cost, so the first call only takes ~3s; for clap-rollout-replay/clap-teleop/clap-eval, which never load openpi, the very first call is genuinely cold and can take ~15s instead. Either way, it's a one-time cost — every call after the first lands at the steady-state numbers above.

• Full cross-embodiment modeling runs (examples/slurm/*.slurm) are sized for 8-GPU nodes (80GB-class GPUs, ~180-450GB system RAM) — post-training/adaptation runs on a single dataset need much less.

• CUDA 12.8-compatible driver — torch/torchvision are pinned to the cu128 wheel index (see Installation below).

• Real training/eval against the full OXE mix needs the underlying video datasets on disk (not included — see Dataset registry); the Getting started walkthrough below needs none of that.

📦 Installation

1. Core package

This package is developed and tested with uv — recommended:

git clone <this-repo> clap && cd clap
VENV_HOME=${UV_ENV_DIR:-".venv"}
uv venv ${VENV_HOME}/clap --python 3.11   # or point --python at any interpreter >= 3.9
source ${VENV_HOME}/clap/bin/activate
uv pip install -e .

Or, in one step with uv sync (creates the venv and installs the exact locked versions from uv.lock together — more reproducible, since uv pip install -e . above re-resolves versions each time instead of pinning to the committed lockfile):

git clone <this-repo> clap && cd clap
VENV_HOME=${UV_ENV_DIR:-".venv"}
UV_PROJECT_ENVIRONMENT=${VENV_HOME}/clap uv sync --python 3.11
source ${VENV_HOME}/clap/bin/activate

2. Policy backends (optional)

Real-robot/policy-in-the-loop deployment (clap-rollout-deploy) additionally needs a trained policy — openpi or MolmoAct2, matching deploy_config.policy_type. Both are only imported inside clap.rollout.policies.{openpi_policy,molmoact_policy} when you actually construct one; skip this step for training/eval/replay/teleop.

🤖 openpi (pi0 / pi05 / pi0fast)

git clone --recurse-submodules git@github.com:Physical-Intelligence/openpi.git optional_dependencies/openpi
cd optional_dependencies/openpi
# Or, if you already cloned without --recurse-submodules:
#   git submodule update --init --recursive

# For separate server-based policy calls
#   for in-process (comment the next line)
GIT_LFS_SKIP_SMUDGE=1 uv sync
GIT_LFS_SKIP_SMUDGE=1 uv pip install -e .

Full details (other install methods): Physical-Intelligence/openpi#installation. Checkpoints, and in-process vs. server tradeoffs, are covered under Tips and Troubleshooting.

🤖 MolmoAct2

git clone https://github.com/allenai/molmoact2.git optional_dependencies/molmoact2
cd optional_dependencies/molmoact2
# CLAP supports only server-based policy calls for now
uv sync

Full details (other install methods): allenai/molmoact2#4-start-a-server. Checkpoints and bimanual_yam setup are covered under Tips and Troubleshooting.

Server warmup crashing with predict_action() got an unexpected keyword argument 'action_mode'? See Tips and Troubleshooting.

3. Environment

All cluster/site-specific paths are read from CLAP_* environment variables (never hardcoded) — see setup/env.example.sh for the full list with explanations. At minimum:

cp setupe/env.example.sh setup/env.sh
# edit env.sh: set CLAP_OXE_BASE_PATH, and any per-dataset overrides your data needs
source setup/env.sh

CLAP_OXE_BASE_PATH is the only strictly required variable (clap.config.paths.PathConfig raises immediately if it's unset). Everything else has a generic default — dataset_meta_info/ (per-dataset stat.json normalization bounds) ships pre-populated in this repo for every registered dataset, so most setups don't need CLAP_META_INFO_ROOT at all.

🏁 Getting started

No dataset or account setup needed for this section — everything below runs against sample_data/oxe/ (~232MB, real full-length val episodes for droid/ bridge/taco_play and the two novel-embodiment adaptation targets bimanual_yam/g1_humanoid, shipped in this repo) and downloads one checkpoint from HF on first use. To get started, activate your virtual environment:

source ${UV_ENV_DIR:-.venv}/clap/bin/activate   # activate the venv created in Installation ▸ 1. Core package

📥 Download a checkpoint (optional)

Checkpoints are automatically downloaded on demand to model_ckpt/<CKPT_NAME>/ — set CKPT_NAME to pick the default; see the checkpoints table. To pre-fetch one manually instead:

hf download omni-CLAP/CLAP --include "cross_embodiment_oxe_ee/checkpoint-100000.pt" --local-dir model_ckpt

🔁 Replay demonstration trajectories

Roll a checkpoint out against a sample episode and write a side-by-side GT/prediction video + PSNR/SSIM/LPIPS:

bash examples/getting_started/replay.sh                                 # droid, clap-curr
DATASET=bridge bash examples/getting_started/replay.sh                  # or: droid | bridge | taco_play
CKPT_NAME=clap-lam DATASET=bridge bash examples/getting_started/replay.sh  # or other models: clap-lam, clap-lang, clap-ee-droid, clap-curr-bridge, ...
CKPT_NAME=clap-ee DATASET=droid MAX_EPISODES=3 bash examples/getting_started/replay.sh  # multiple episodes

# Novel-embodiment adaptation targets (14-/26-dim joint-space actions, not 7-dim EE
# cartesian) -- pair with their matching adapt_* checkpoint; MAX_CHUNKS caps their
# much longer sample episodes (1000+ frames) for a quick demo
CKPT_NAME=adapt-yam DATASET=bimanual_yam MAX_EPISODES=3 MAX_CHUNKS=20 TRIM_STATIC=1 bash examples/getting_started/replay.sh
CKPT_NAME=adapt-g1 DATASET=g1_humanoid MAX_EPISODES=3 MAX_CHUNKS=20 bash examples/getting_started/replay.sh

See Tips and Troubleshooting for what --num-inference-steps (25 here, vs. the real default of 50), MAX_EPISODES, and MAX_CHUNKS (all used above) do.

Output: eval_outputs/getting_started_replay/<CKPT_NAME>_<timestamp>/{video,info}/.

🎮 Live Teleop for DROID, BRIDGE, TACO-PLAY, BIMANUAL YAM, G1 Humanoid

Interactively drive the world model's imagined next frame with your keyboard, seeded from a sample episode (Ctrl-C to quit). droid/bridge/taco_play use a single-arm cartesian scheme (w/a/s/d/z/x + roll/pitch/yaw + gripper); bimanual_yam/g1_humanoid use a shared per-joint scheme targeting whichever arm/hand is active (Tab cycles it, Space mirrors keypresses onto its left/right counterpart too). The live-preview browser page (below) shows the actual key map and highlights each key as you press it, which is easier to follow live than reading it here — see clap.rollout.teleop_controls's module docstring if you want the exact static reference instead:

bash examples/getting_started/teleop.sh                              # default DATASET=droid, CKPT_NAME=clap-curr
DATASET=bridge bash examples/getting_started/teleop.sh                # or: bridge | droid | taco_play
DATASET=bimanual_yam bash examples/getting_started/teleop.sh          # CKPT_NAME auto-set to adapt_bimanual_yam
DATASET=g1_humanoid bash examples/getting_started/teleop.sh           # CKPT_NAME auto-set to adapt_g1_humanoid
EPISODE=1002 DATASET=bridge bash examples/getting_started/teleop.sh   # other episodes: bridge -> 10|1002|1003, droid -> 2799|7099|9199, taco_play -> 1002|1003|1010
CKPT_NAME=clap-ee DATASET=droid bash examples/getting_started/teleop.sh  # or other checkpoints (clap-lang, clap-ee-droid, post-trained, ...); explicit CKPT_NAME always wins over the auto-default above
LIVE_VIEW_WS_PORT=7765 LIVE_VIEW_HTTP_PORT=7766 bash examples/getting_started/teleop.sh  # modify the websocket/html viewer ports

Output: eval_outputs/getting_started_teleop/<CKPT_NAME>_<timestamp>/. See Tips and Troubleshooting for the live-preview viewer, scripted (KEYS=) mode, --num-inference-steps, workspace bounds, g1_humanoid control tips, and camera stacking order.

🚀 Deploy (policy-in-the-loop) on DROID and Bimanual YAM with Live Preview

Simulated closed-loop deployment: a real policy outputs actions and the world model predicts the future each round with live preview. Needs MolmoAct2 or openpi installed (see Installation) and a policy server/checkpoint filled into examples/getting_started/deploy_config.yaml first — see Tips and Troubleshooting for other deploy-specific notes:

bash examples/getting_started/deploy.sh                                       # uses deploy_config.yaml's own policy_type/policy_server/policy_ckpt setting
POLICY_SERVER_OVERRIDE=127.0.0.1:8001 bash examples/getting_started/deploy.sh  # or a different policy server host/port
POLICY_TYPE=pi05 bash examples/getting_started/deploy.sh                      # or openpi/pi05 instead of the MolmoAct2 default
POLICY_TYPE=pi05 IN_PROCESS=1 bash examples/getting_started/deploy.sh         # or an in-process policy instead of a server
CKPT_NAME=clap-ee POLICY_SERVER_OVERRIDE=127.0.0.1:8001 bash examples/getting_started/deploy.sh  # or other checkpoints
LIVE_VIEW_WS_PORT=6765 LIVE_VIEW_HTTP_PORT=6766 bash examples/getting_started/deploy.sh  # modify the websocket/html viewer ports

bash examples/getting_started/deploy_yam.sh  # bimanual_yam instead of DROID -- joint-space, MolmoAct2 server mode only (see the bimanual_yam paragraph under Installation's MolmoAct2 section)

Output: eval_outputs/getting_started_deploy/<CKPT_NAME>_<timestamp>/ (or eval_outputs/getting_started_deploy_yam/<CKPT_NAME>_<timestamp>/ for deploy_yam.sh).

🧩 Sample-efficient adaptation to new embodiments

CLAP establishes a paradigm for training high-fidelity single-embodiment video world models via sample-efficient, few-shot adaptation of cross-embodiment models to target robot platforms. This section provides a worked example using bimanual_yam (and g1_humanoid — see the notes inline). This repo ships one train-split sample episode for both, alongside their existing val episodes (sample_data/oxe/{bimanual_yam,g1_humanoid}/{videos,annotation}/train/), just enough to exercise this whole pipeline end to end.

1. Format your data as <oxe_base_path>/<dataset_name>/{videos,annotation}/<split>/... — sample_data/oxe/bimanual_yam is a concrete (tiny) reference for the expected layout: one videos/<split>/<episode_id>/<cam>.mp4 per camera, one annotation/<split>/<episode_id>.json per episode (state, texts, ... — see Dataset registry). g1_humanoid's annotations instead come from its own clap-preprocess-g1 pipeline, which builds this same layout from unitreerobotics/* HF datasets.

CLAP_OXE_BASE_PATH=$CLAP_OXE_BASE_PATH bash examples/preprocess/compute_g1_meta_info.sh  # only needed for g1_humanoid

2. Compute action-normalization stats (dataset_meta_info/<name>/stat.json, what BoundNormalizer reads at train/eval time — train split only, so val stays unseen by anything derived from training data, including the normalization bounds themselves):

clap-preprocess-oxe-meta --oxe-base-path sample_data/oxe --dataset-name bimanual_yam --full-state
# --full-state: bimanual_yam's 14-dim action is the raw `state` array, not state[:6]+gripper
# like cartesian ee7 datasets. g1_humanoid gets its stat.json from clap-preprocess-g1 (see step 1 instead)

3. Point CLAP at your data — two env vars (see env.example.sh):

export CLAP_OXE_BASE_PATH=sample_data/oxe    # your real dataset root, in practice
export CLAP_META_INFO_ROOT=dataset_meta_info # where step 2 wrote stat.json (this is the default -- only needed if you chose a different --meta-info-path)

4. Launch adaptation training (finetunes from the cross-embodiment curriculum checkpoint; configs/experiment/adapt_bimanual_yam.yaml resets the action encoder since the input dim changes 7 → 14; see Tips and Troubleshooting for how this script picks up your CLAP_* env vars):

bash examples/adapt/adapt_bimanual_yam.sh
# g1_humanoid: bash examples/adapt/adapt_g1_humanoid.sh

5. Build a test set (episode selection cached under clap/eval/test_sets_cache/, so every eval run pins to the same episodes):

clap-build-test-sets --names bimanual_yam_val --oxe-base-path "$CLAP_OXE_BASE_PATH" --no-lam-intersection
# --no-lam-intersection: episode selection normally also requires LAM latent-action data
# (ann_keys & lam_keys) -- skip that for datasets that never compute LAM at all, like
# bimanual_yam/g1_humanoid (joint-space-only), or it'll always select 0 episodes for them.
# g1_humanoid:
# clap-build-test-sets --names g1_humanoid_val --oxe-base-path "$CLAP_OXE_BASE_PATH" --no-lam-intersection

6. Evaluate the adapted checkpoint:

clap-eval --config configs/experiment/adapt_bimanual_yam.yaml --experiment adapt_bimanual_yam \
    --test-set bimanual_yam_val
# g1_humanoid:
# clap-eval --config configs/experiment/adapt_g1_humanoid.yaml --experiment adapt_g1_humanoid \
#     --test-set g1_humanoid_val

Trying this locally without touching your repo's own dataset_meta_info/bimanual_yam/stat.json — point every step at a scratch dir instead of its default output location:

SCRATCH=$(mktemp -d)
clap-preprocess-oxe-meta --oxe-base-path sample_data/oxe --dataset-name bimanual_yam --full-state \
    --meta-info-path "$SCRATCH/dataset_meta_info"
export CLAP_OXE_BASE_PATH=sample_data/oxe
export CLAP_META_INFO_ROOT="$SCRATCH/dataset_meta_info"
clap-build-test-sets --names bimanual_yam_val --oxe-base-path "$CLAP_OXE_BASE_PATH" \
    --no-lam-intersection --out-dir "$SCRATCH/test_sets_cache"
clap-eval --config configs/experiment/adapt_bimanual_yam.yaml --experiment adapt_bimanual_yam \
    --test-set bimanual_yam_val --test-sets-cache-dir "$SCRATCH/test_sets_cache" \
    --max-chunks 1   # fast test -- 1 autoregressive chunk (config.model.num_frames predicted
                     # frames) per episode instead of a full rollout; omit for a full-duration eval

🗂️ Checkpoints

We provide a broad suite of cross-embodiment video model checkpoints at omni-CLAP/CLAP — trained for 100K steps — and finetuned checkpoints derived from them for target embodiments.

🌐 Cross-embodiment models

The cross-embodiment models are trained on a mix of OXE datasets: (fractal/fmb/bc_z/taco_play/furniture_bench/bridge/droid, egodex too for lam). All ee-conditioned models use absolute actions, while the lang-conditioned model uses relative-to-anchor-frame language captions.

🎯 Post-trained on a single robot platform

Each finetunes the matching cross-embodiment checkpoint above onto one dataset alone (still ee-conditioned regardless of the base's own conditioning — see clap.eval.experiments' _POST_TRAIN_BASES comment).

DROID

Bridge

🆕 Novel-embodiment adaptation

Finetune on data for a new robot embodiment, builds on physical priors from the cross-embodiment models (defaults to clap-curr)

📁 Project layout

src/clap/
  config/       dataclass configs (CLAPModelConfig, DataConfig, TrainingConfig, PathConfig, ...)
                + load_config() (Hydra-style YAML composition)
  data/         per-embodiment dataset registry + EE/LAM/Language dataset classes
  models/       CLAPModel (SVD-based UNet + action/text conditioning)
  training/     clap-train and its dataloader/checkpoint/validation helpers
  eval/         clap-eval and the checkpoint-comparison/test-set-building utilities
  rollout/      autoregressive replay, real-robot deployment, keyboard teleop
  preprocess/   per-dataset stat.json / G1-humanoid data prep
  utils/        shared rich-console logging setup

configs/        YAML configs consumed by load_config() (model/, data/, training/, experiment/)
examples/       runnable shell scripts wrapping the CLI entrypoints below (train/eval/rollout/adapt/slurm)
dataset_meta_info/   shipped stat.json per dataset (normalization bounds)
tests/          unit/ (no data needed) + integration/ (needs real OXE data / a GPU, auto-skipped otherwise)

📚 Reference

Full configuration reference, the complete entrypoint/CLI documentation, the dataset registry, and testing instructions live in docs/REFERENCE.md — split out to keep this README focused on getting up and running.

🛠️ Tips and Troubleshooting

# in the molmoact2 repo
sed -i 's/action_mode="continuous"/inference_action_mode="continuous"/' examples/droid/host_server_droid.py

cd optional_dependencies/openpi
OPENPI_POLICY_BASE_DIR=${OPENPI_POLICY_BASE_DIR:-~/.cache/openpi}
# an example with pi05
uv run scripts/serve_policy.py policy:checkpoint --port=8000 \
    --policy.config=pi05_droid --policy.dir=${OPENPI_POLICY_BASE_DIR}/pi05_droid

export HF_HUB_ENABLE_HF_TRANSFER=1
uv run hf download allenai/MolmoAct2-DROID          # for the DROID server
uv run hf download allenai/MolmoAct2-BimanualYAM    # for the YAM server

uv run python examples/droid/host_server_droid.py --host 0.0.0.0 --port 8000 --dtype bfloat16
# or, for bimanual YAM:
uv run python examples/yam/host_server_yam.py --host 0.0.0.0 --port 8202 --dtype bfloat16

🙏 Acknowledgment

CLAP builds on and integrates with several open-source projects:

• Stable Video Diffusion (Stability AI) — the pretrained video diffusion backbone CLAPModel's UNet is built on.
• Ctrl-World — state-of-the-art action-conditioned video world model for the DROID robot platform.
• openpi (Physical Intelligence) — policy backend supported for policy-in-the-loop deployment (π₀/π₀.₅/π₀-FAST).
• MolmoAct2 (Allen Institute for AI) — policy backend supported for policy-in-the-loop deployment.

We thank the authors of these projects for making their code and models available.

(More coming soon)

📄 BibTeX

Coming soon

📖 Glossary

⚖️ License

MIT — see LICENSE.