DM05 Tutorial
September 16, 2026 ยท View on GitHub
DM0.5 is Dexmal's open-world VLA: a 4B VLM plus a 680M Action Expert that emits continuous actions with Flow Matching. It is larger than DM0, and history, embodied reasoning, action supervision, and data quality are stronger, so the model can track task progress, follow open instructions, and stay stable under camera change and human interference.
Main gains (pretrained):
- Zero-shot: follow language in unseen open scenes (8 primitives, 7 condition types).
- Fine-tune: a stronger base gives cheaper, better specialists.
- Long memory: history up to 60 seconds.
- Robust actions: lighting, camera motion, human perturbation.
- Multi-embodiment: multi-robot pretrain, then transfer.
This repo implements Gemma3 VLM + Gemma3 Action Expert.
More narrative and official eval videos: DM0.5 blog.
Pretrained Model
Download the Hugging Face base weights to checkpoints/DM05. Playground training defaults resolve to that path.
| Model | Description | Input Images | Action Dim | Backbone | Link |
|---|---|---|---|---|---|
| DM05 | Base Gemma3 VLM + Flow Matching Action Expert | set by the playground (LIBERO: agentview + wrist) | padded to 32D (LIBERO valid dim 7) | 4B VLM | ๐ค Hugging Face |
huggingface-cli download Dexmal/DM05 \
--local-dir checkpoints/DM05 \
--local-dir-use-symlinks False
Entrypoints default to ./checkpoints/DM05 for training. The official LIBERO eval checkpoint is Dexmal/DM05-libero; see Benchmark Results.
Installation and Environment Setup
The DM05 image is dexmal/dexbotic:dm05. Clone the repo on the host, mount it into the container, then install the mounted source.
git clone https://github.com/Dexmal/dexbotic.git
docker run -it --rm --gpus all --network host \
-v /path/to/dexbotic:/dexbotic \
dexmal/dexbotic:dm05 \
bash
cd /dexbotic
conda activate dexbotic
pip install -e .
Local conda install if you are not using Docker:
git clone https://github.com/Dexmal/dexbotic.git
conda create -n dexbotic python=3.10 -y
conda activate dexbotic
pip install torch torchvision \
--index-url https://download.pytorch.org/whl/cu128
pip install ninja packaging
MAX_JOBS=2 pip install flash-attn --no-build-isolation
cd dexbotic
pip install -e .
pip install transformers==5.3.0
The default train recipes below need 8 GPUs (H20 / A100 / H100 for full SFT; 4090 works for LoRA). One GPU is enough for inference. FSDP2 notes: FSDP2.md.
LIBERO Data
LIBERO is the shipped example. The in-repo recipes train on libero_pi0_all from Dexmal/libero. After download, that split should sit under data/libero. Full SFT vs LoRA differences are in the Training table.
huggingface-cli download Dexmal/libero \
--repo-type dataset \
--local-dir data/.hf_downloads/libero \
--local-dir-use-symlinks False
mkdir -p data/libero
cp -a data/.hf_downloads/libero/libero_pi0_all data/libero/
test -d data/libero/libero_pi0_all/jsonl
test -d data/libero/libero_pi0_all/image
If the download is archives instead of a folder, extract into data/libero so those two paths exist. Dataset catalog: Data.md.
Training
Start from ./checkpoints/DM05 and data/libero/libero_pi0_all above. Full SFT and LoRA are two playground files.
Training a Model with Provided Data
| Item | Full SFT | LoRA |
|---|---|---|
| Entrypoint | libero_dm05.py | libero_dm05_lora.py |
use_lora | False | True |
| Backend | FSDP2 | DDP only (deepspeed=None) |
| GPUs | 8 ร H20 / A100 / H100 | 8 ร 4090 or H20; on multi-GPU 4090 set NCCL_P2P_DISABLE=1 NCCL_IB_DISABLE=1 |
| Extra install | โ | pip install "peft>=0.13.0" |
| LLM attention | flex_attention | eager |
| Vision / action attention | flash_attention_2 / sdpa | sdpa / sdpa |
| VLM / AE gradient checkpointing | off | on |
| LR / warmup | 2.5e-5 / 1000 | 5e-4 / 500 |
| Per-device batch | 8 | 4 |
model_max_length | 768 | 1024 |
| Steps / save | 50000 / 10000 | 50000 / 10000 |
| LoRA | โ | r=32, alpha=16, dropout=0, all-linear (no lm_head) |
| Output | ./user_checkpoints/dexbotic/libero_dm05/libero-sft | ./user_checkpoints/dexbotic/libero_dm05_lora/libero-lora-MMDD |
| Infer default path | ./checkpoints/DM05 unless you pass --model_name_or_path | ./user_checkpoints/dexbotic/libero_dm05_lora/libero-lora-MMDD |
Shared: model_name_or_path=./checkpoints/DM05, dataset_name=libero_pi0_all, 2 views, chunk_size=10, diffusion_steps=10, pad-to-square then 448ร448, bf16, save_hf_sidecar=True.
# full SFT
torchrun --nproc_per_node=8 playground/benchmarks/libero/libero_dm05.py --task train --train-backend fsdp2
# LoRA
pip install "peft>=0.13.0"
export NCCL_P2P_DISABLE=1 # multi-GPU 4090 only
export NCCL_IB_DISABLE=1
torchrun --nproc_per_node=8 playground/benchmarks/libero/libero_dm05_lora.py --task train --train-backend ddp
If norm_stats.json is missing at train start, rank 0 computes it and the other ranks wait. Training raises if that file never appears. Inference does not compute it; the file must already sit next to --model_name_or_path.
python playground/benchmarks/libero/libero_dm05.py --task compute_norm_stats
LoRA writes adapter_model.safetensors + adapter_config.json and checkpoint-{step}-hf. Infer merges the adapter; continuing from an adapter keeps PeftModel. More LoRA recipe notes: LiberoLora.md.
Training a Model with Your Own Data
Write Dexdata JSONL, register it under dexbotic/data/data_source, then copy a LIBERO playground file and edit it for your data. Each frame needs images, state, and prompt. The example below is three-view:
dexdata_frame = {
"images_1": {"type": "image", "url": frame["image_path_1"]},
"images_2": {"type": "image", "url": frame["image_path_2"]},
"images_3": {"type": "image", "url": frame["image_path_3"]},
"state": frame["robot_state"],
"prompt": frame["instruction"],
"is_robot": True,
}
Register under dexbotic/data/data_source/:
from dexbotic.data.data_source.register import register_dataset
import math
MY_CUSTOM_DATASET = {
"my_robot_data": {
"data_path_prefix": "",
"annotations": "/path/to/your/custom_dataset/",
"frequency": 1,
},
}
meta_data = {
"non_delta_mask": [6],
"periodic_mask": [3, 4, 5],
"periodic_range": 2 * math.pi,
}
register_dataset(MY_CUSTOM_DATASET, meta_data=meta_data, prefix="my_custom")
The registered name is my_custom_my_robot_data. Copy libero_dm05.py or libero_dm05_lora.py and change the fields below to match this three-view example (keep two-view defaults if your data is two-view):
# DM05TrainerConfig
output_dir = [checkpoint dir]
num_train_steps = [steps]
# DM05DataConfig
dataset_name = "my_custom_my_robot_data"
num_images = 3
images_keys = ["images_1", "images_2", "images_3"]
valid_action_dim = 7
# DM05ModelConfig / DM05InferenceConfig
model_name_or_path = "./checkpoints/DM05"
action_dim = 7
camera_order = ["agentview", "wrist", "right_wrist"]
# DM05DataCollator (class attr; default is 2 views)
image_prompts = ("Head", "Left wrist", "Right wrist")
num_images must equal len(DM05DataCollator.image_prompts) (default ("Head", "Left wrist")). A mismatch raises.
torchrun --nproc_per_node=8 path/to/your_dm05_exp.py --task train
More Dexdata fields and multi-dataset + mixing: Data.md.
Inference Backends
DM05InferenceConfig.backend selects one of two inference paths:
| Backend | Vision path | Prefix and action path | History frames |
|---|---|---|---|
default | PyTorch | Dynamic KV-cache inference | Supported |
fast | TensorRT | DM05 Triton kernels with fixed prefix buckets | Supported; requests containing history bypass CUDA Graph capture |
Both backends expose the same HTTP service on 0.0.0.0:7891 by default,
including /health, /v1/capabilities, /v1/infer, /v1/reset, and the
legacy /process_frame route. The backend only changes how the model executes;
clients do not need to use a different request format.
Run the commands below from the repository root. Replace
/path/to/dm05-checkpoint with a full Hugging Face checkpoint directory or a
checkpoint-*-hf directory. The checkpoint directory must contain
norm_stats.json.
Start the LIBERO default backend
The default backend uses PyTorch and does not require the optional fast plugin. Start it with the LIBERO entrypoint:
CUDA_VISIBLE_DEVICES=0 python playground/benchmarks/libero/libero_dm05.py \
--task inference \
--backend default \
--port 7891 \
--model-name-or-path /path/to/dm05-checkpoint
--backend default is optional because it is the configured default. The
command blocks while the server is running.
Start the LIBERO fast backend
The fast backend requires the optional DM05 plugin. Install it in the active Dexbotic environment:
bash plugins/dm05-fast/install.sh
Start the same LIBERO entrypoint with --backend fast:
CUDA_VISIBLE_DEVICES=0 python playground/benchmarks/libero/libero_dm05.py \
--task inference \
--backend fast \
--port 7891 \
--model-name-or-path /path/to/dm05-checkpoint \
--vision-trt-engine-path checkpoints/trt_engines/dm05-checkpoint.engine
The CLI leaves build_vision_engine_if_missing=True by default, so the first
startup builds a fixed-shape TensorRT vision engine if a matching engine does
not already exist. Later startups reuse that engine. Use a separate engine path
for each checkpoint because the engine contains checkpoint-specific vision
weights. Alternatively, pass --force-rebuild-vision-engine after changing
the checkpoint. The fast backend supports batch size 1.
Other fast options are available through --prefix-seq-len-buckets,
--fast-overflow-policy, --fast-prefix-qkv-mode, and
--no-build-vision-engine-if-missing. Run the entrypoint with --help for the
complete CLI.
To build the TensorRT engine separately instead, pass the total number of image
slots accepted by the engine. The standard history-disabled setup uses two
current camera views, so --num-images is 2:
CUDA_VISIBLE_DEVICES=0 python -m dexbotic.model.dm05.infer.fast.build_vision_trt \
--checkpoint /path/to/dm05-checkpoint \
--onnx-path checkpoints/trt_engines/dm05_vision.onnx \
--engine-path checkpoints/trt_engines/dm05_vision.engine \
--num-images 2
Verify and call the service
After either backend finishes loading, verify that the service is ready:
curl http://localhost:7891/health
curl http://localhost:7891/v1/capabilities
Send two images in camera_order order (agentview, then wrist) to the
legacy route:
curl -X POST \
-F "text=What action should the robot take to put the bowl on the plate?" \
-F "image=@/path/to/agentview.png" \
-F "image=@/path/to/wrist.png" \
http://localhost:7891/process_frame
See InferenceAPI.md for the /v1/infer JSON schema and
DexClient example.
Start a RoboTwin2 service
RoboTwin2 uses a dedicated entrypoint because its inference contract differs
from LIBERO: three RGB views (Head, Left wrist, Right wrist), a required
14-dimensional Aloha state, 14-dimensional actions, and a 50-step action
chunk. Do not use libero_dm05.py for a RoboTwin2 checkpoint.
Start the default backend:
CUDA_VISIBLE_DEVICES=0 python \
playground/benchmarks/robotwin2/robotwin2_dm05.py \
--backend default \
--port 7891 \
--model-name-or-path /path/to/robotwin2-dm05-checkpoint
Start the fast backend after installing plugins/dm05-fast:
bash plugins/dm05-fast/install.sh
CUDA_VISIBLE_DEVICES=0 python \
playground/benchmarks/robotwin2/robotwin2_dm05.py \
--backend fast \
--port 7891 \
--model-name-or-path /path/to/robotwin2-dm05-checkpoint \
--vision-trt-engine-path \
checkpoints/trt_engines/robotwin2-dm05-checkpoint.engine
The checkpoint directory must contain norm_stats.json. If the statistics
live elsewhere, pass --norm-stats /path/to/norm_stats.json. The fast command
builds a fixed-shape three-image TensorRT engine on first startup and remains
single-request. Use a checkpoint-specific engine path, as in the example.
Both backends accept the same legacy request and return an action array with
shape 50 x 14:
curl -X POST \
-F "text=Pick up the block" \
-F "image=@/path/to/head.png" \
-F "image=@/path/to/left-wrist.png" \
-F "image=@/path/to/right-wrist.png" \
-F 'states=[0,0,0,0,0,0,0,0,0,0,0,0,0,0]' \
http://localhost:7891/process_frame
RoboTwin2 supports the same opt-in, explicit history-image contract as LIBERO.
Start the service with --history-enabled --max-history-images 5, then repeat
the multipart history_images field in oldest-to-newest order on each request.
The service does not retain frames between requests.
Backend configuration reference
Relevant DM05InferenceConfig settings:
| Setting | Default | Meaning |
|---|---|---|
backend | "default" | Select default or fast inference |
model_max_length | None | Optional preprocessing prefix-length limit |
vision_trt_engine_path | checkpoints/trt_engines/dm05_vision.engine | TensorRT engine used by the fast backend |
build_vision_engine_if_missing | True | Ensure that a matching engine exists during startup |
force_rebuild_vision_engine | False | Rebuild the engine even if its input shape already matches |
prefix_seq_len_buckets | [576, 704, 768, 896, 1024] | Fixed prefix lengths available to the fast runtime |
fast_overflow_policy | "fallback" | Use dynamic execution or raise when no bucket fits |
fast_prefix_qkv_mode | "packed" | Select packed or separate prefix QKV projection |
history_enabled | False | Accept explicit history frames |
max_history_images | 5 | Maximum history frames accepted by the policy |
The fast backend supports batch size 1. Without history frames, the first
request assigned to a prefix bucket lazily captures prefix prefill and
fixed-step action denoising in one CUDA Graph. Later requests using that bucket
stage their inputs into stable buffers and replay the graph. No graph profile is
captured during service startup, although CUDA Graph availability is validated
when the fast runtime is initialized. A prefix longer than the largest
configured bucket either uses the dynamic fallback or raises, according to
fast_overflow_policy.
Inference Performance
| GPU | Default latency | Fast latency | Approx. speedup |
|---|---|---|---|
| RTX 4090 D | 523.7 ms | 103.4 ms | 5.1x |
| H20 | 610.0 ms | 129.1 ms | 4.7x |
Measured with a real DM05 checkpoint, BF16 loading, batch size 1, chunk size 10, and 10 diffusion steps. Results use warmed-up CUDA Graph replay with two current images and no history.
These are approximate model-inference speedups, excluding HTTP, preprocessing, and first-request capture. History and overflow fallback performance may differ.
History-frame inference
History is opt-in and explicit: the server does not collect frames between
requests. The caller supplies an ordered history_images list with every
request. Frames must be ordered from oldest to newest.
Enable history on either backend by adding these CLI options to its startup command:
--history-enabled --max-history-images 5
Both backends accept history frames. The default maximum is five images. The fast backend has a fixed maximum of five and reserves five history slots in its TensorRT vision engine. Therefore, a history-enabled engine for the standard two-camera setup needs seven image slots. Give the fast history service a different engine path from the history-disabled service:
CUDA_VISIBLE_DEVICES=0 python playground/benchmarks/libero/libero_dm05.py \
--task inference \
--backend fast \
--model-name-or-path /path/to/dm05-checkpoint \
--vision-trt-engine-path checkpoints/trt_engines/dm05-checkpoint-history.engine \
--history-enabled \
--max-history-images 5
For this LIBERO configuration, automatic engine building calculates the
seven-slot shape. To build the same engine separately, use --num-images 7:
CUDA_VISIBLE_DEVICES=0 python -m dexbotic.model.dm05.infer.fast.build_vision_trt \
--checkpoint /path/to/dm05-checkpoint \
--onnx-path checkpoints/trt_engines/dm05_vision_history.onnx \
--engine-path checkpoints/trt_engines/dm05_vision_history.engine \
--num-images 7
The RoboTwin2 entrypoint accepts the same two flags. Its fast backend uses three current-view slots plus five fixed history slots, so use a separate eight-slot TensorRT engine path when history is enabled. Automatic engine building selects the eight-slot shape.
When automatic engine building is enabled, the fast backend calculates this slot count automatically. A request containing history still uses TensorRT for vision and the optimized DM05 kernels, but it runs prefix prefill and denoising without a request-level CUDA Graph. Requests without history can continue to use the lazily captured graph profiles. Because the TensorRT input shape is fixed, use different engine files for history-enabled and history-disabled configurations.
The v1 API accepts history as an array of base64-encoded images. The legacy
/process_frame route accepts repeated multipart fields named
history_images:
curl -X POST \
-F "text=Put the bowl on the plate" \
-F "image=@/path/to/agentview.png" \
-F "image=@/path/to/wrist.png" \
-F "history_images=@/path/to/history-oldest.png" \
-F "history_images=@/path/to/history-newest.png" \
http://localhost:7891/process_frame
See InferenceAPI.md for the v1 JSON schema and DexClient
example.
Evaluation
Start an inference server and POST to /process_frame, or point dexbotic-benchmark at that server.
Pass --model_name_or_path at a finished run dir or a checkpoint-*-hf. norm_stats.json must sit next to that path. Full SFT without the flag falls back to ./checkpoints/DM05 (the pretrained base, not your SFT run). Server port 7891. Two images: camera_order=["agentview", "wrist"].
CUDA_VISIBLE_DEVICES=0 python playground/benchmarks/libero/libero_dm05.py \
--task inference --model_name_or_path ./user_checkpoints/dexbotic/libero_dm05/libero-sft
# or playground/benchmarks/libero/libero_dm05_lora.py --task inference \
# --model_name_or_path ./user_checkpoints/dexbotic/libero_dm05_lora/libero-lora-MMDD
curl -X POST \
-F "text=What action should the robot take to put the bowl on the plate?" \
-F "image=@/path/to/agentview.png" \
-F "image=@/path/to/wrist.png" \
http://localhost:7891/process_frame
cd dexbotic-benchmark
docker run --gpus all --network host -v $(pwd):/workspace \
dexmal/dexbotic_benchmark \
bash /workspace/scripts/env_sh/libero.sh /workspace/evaluation/configs/libero/example_dm0_libero.yaml
The client must send those two cameras. Docker-free setup: dexbotic-benchmark README. To reproduce the official LIBERO numbers below, serve Dexmal/DM05-libero as in Benchmark Results.
Benchmark Results
Official DM0.5 scores on LIBERO and RoboChallenge Table30 V2.
LIBERO
Four-suite protocol; last column is the average.
| Method | Spatial | Object | Goal | Long | Average |
|---|---|---|---|---|---|
| ฯ0 | 96.8 | 98.8 | 95.8 | 85.2 | 94.2 |
| ฯ0.5 | 98.8 | 98.2 | 98.0 | 92.4 | 96.9 |
| OpenVLA-OFT | 97.6 | 98.4 | 97.9 | 94.5 | 97.1 |
| GR00T N1.7 | 97.7 | 98.5 | 97.5 | 94.4 | 97.0 |
| StarVLA | 99.0 | 99.8 | 98.5 | 94.1 | 97.9 |
| ABot-M0 | 98.8 | 99.8 | 99.0 | 96.6 | 98.6 |
| Being-H0.5 | 99.2 | 99.6 | 99.4 | 97.4 | 98.9 |
| Cosmos Policy | 98.1 | 100.0 | 98.2 | 97.6 | 98.5 |
| DM0.5 | 99.0 | 99.8 | 99.6 | 97.4 | 99.0 |
The DM0.5 row is the official Dexmal/DM05-libero checkpoint. Download it and serve (norm_stats.json is in the repo):
huggingface-cli download Dexmal/DM05-libero \
--local-dir checkpoints/DM05-libero \
--local-dir-use-symlinks False
CUDA_VISIBLE_DEVICES=0 python playground/benchmarks/libero/libero_dm05.py \
--task inference --model_name_or_path ./checkpoints/DM05-libero
RoboChallenge Table30 V2
Real-robot Table30 V2 score and success rate.
| Metric | DM0.5 | ฯ0 | ฯ0.5 | GR00T N1.7 |
|---|---|---|---|---|
| Score | 54.42 | - | 31.48 | - |
| SR | 43.0% | - | 14.3% | - |