GR00T Deployment & Inference Guide

Run inference with PyTorch or TensorRT acceleration for the GR00T N1.7 policy.

Prerequisites

• Model checkpoint: nvidia/GR00T-N1.7-3B
• Dataset in LeRobot format (e.g., demo_data/libero_demo)
• CUDA-enabled GPU
• Setup uv environment following README.md

• dGPU local environment: use the installation commands below, then use the PyTorch or TensorRT commands in this guide
• Thor Docker or bare metal: skip to Jetson Thor Setup
• Spark Docker or bare metal: skip to DGX Spark Setup
• Orin Docker or bare metal: skip to Jetson Orin Setup

dGPU Installation

uv sync

GPU dependencies (flash-attn, onnx, tensorrt) are included in the default install.

Download Model and Dataset

Download the finetuned model to a local directory (HuggingFace does not support nested repo paths directly):

uv run hf download nvidia/GR00T-N1.7-LIBERO \
  --include "libero_10/config.json" "libero_10/embodiment_id.json" \
  "libero_10/model-*.safetensors" "libero_10/model.safetensors.index.json" \
  "libero_10/processor_config.json" "libero_10/statistics.json" \
  --local-dir checkpoints/GR00T-N1.7-LIBERO

For demo dataset setup, see the Data Format section in the main README.

Quick Start: PyTorch Inference

Run inference on demo trajectories using PyTorch (no TRT setup needed):

uv run python scripts/deployment/standalone_inference_script.py \
  --model-path checkpoints/GR00T-N1.7-LIBERO/libero_10 \
  --dataset-path demo_data/libero_demo \
  --embodiment-tag LIBERO_PANDA \
  --traj-ids 0 1 2 3 4 \
  --inference-mode pytorch \
  --execution-horizon 8

TensorRT Acceleration

The trt_full_pipeline mode (passed via --inference-mode trt_full_pipeline in standalone_inference_script.py) accelerates all model components with TRT engines. Speedup varies by platform — see benchmark tables below for measured results on each device. The same pipeline is referred to as n17_full_pipeline inside the engine-loading and build scripts (trt_model_forward.py, build_trt_pipeline.py); the two names describe the same set of engines.

Lightweight ops remain in PyTorch: embed_tokens, masked_scatter, get_rope_index, VLLN.

TRT Export Modes

Build TRT Engines

The unified build_trt_pipeline.py script runs all steps (export ONNX → build engines → verify accuracy → benchmark) in a single command:

uv run python scripts/deployment/build_trt_pipeline.py \
  --model-path checkpoints/GR00T-N1.7-LIBERO/libero_10 \
  --dataset-path demo_data/libero_demo \
  --embodiment-tag LIBERO_PANDA

Finetuned models: Replace --model-path with your checkpoint path. The pipeline is identical for base and finetuned models.

Note: Engine build takes ~2-5 minutes depending on GPU. Engines are GPU-architecture-specific and must be rebuilt for different GPUs.

Batch size: The --batch-size value is baked as a static dimension into the ONNX and TRT models. Engines built with one batch size cannot be used with a different batch size at runtime. If you need a different batch size, re-run the full pipeline (--steps export,build,verify) with the new --batch-size value.

ONNX exporter: The N1.7 TRT export path uses PyTorch's legacy ONNX exporter explicitly (dynamo=False) for all components. Keep this explicit in custom export code as well: the dynamo exporter can specialize dynamic sequence dimensions such as vl_seq_len, which breaks TensorRT runtime flexibility for different token lengths.

You can also run a subset of steps:

# Export + build only (skip verify and benchmark)
uv run python scripts/deployment/build_trt_pipeline.py \
  --model-path checkpoints/GR00T-N1.7-LIBERO/libero_10 \
  --dataset-path demo_data/libero_demo \
  --embodiment-tag LIBERO_PANDA \
  --steps export,build

Performance

Benchmark Results

GR00T N1.7 Inference Timing (4 denoising steps, 1 camera):

Note: Orin uses DiT-only TensorRT (--inference-mode tensorrt) because TRT 10.3 does not support the backbone engine. All other platforms use the full pipeline (--inference-mode trt_full_pipeline).

Hardware: NVIDIA H100 80GB HBM3
Model: checkpoints/GR00T-N1.7-LIBERO/libero_10
1 camera, Denoising Steps: 4

PyTorch Eager:
  E2E:             85.8 ms (11.7 Hz)
  Data Processing: 6.2 ms | Backbone: 31.3 ms | Action Head: 48.2 ms

torch.compile:
  E2E:             48.6 ms (20.6 Hz), 1.77x speedup
  Data Processing: 6.2 ms | Backbone: 30.4 ms | Action Head: 12.0 ms

TensorRT (Full Pipeline):
  E2E:             27.9 ms (35.9 Hz), 3.08x speedup
  Data Processing: 6.2 ms | Backbone: 8.8 ms  | Action Head: 12.3 ms

Standalone Inference with TRT

The standalone inference script serves as both an accuracy validation and a reference for deploying TRT inference in your own code. It runs per-step inference on real trajectories and compares action predictions:

uv run python scripts/deployment/standalone_inference_script.py \
  --model-path checkpoints/GR00T-N1.7-LIBERO/libero_10 \
  --dataset-path demo_data/libero_demo \
  --embodiment-tag LIBERO_PANDA \
  --traj-ids 0 1 2 3 4 \
  --inference-mode trt_full_pipeline \
  --trt-engine-path ./gr00t_trt_deployment/engines \
  --save-plot-path ./output/trt_inference.png

Expected accuracy: MSE/MAE match PyTorch within noise. TRT produces identical action quality. Speedup varies by platform — run build_trt_pipeline.py --steps benchmark on your hardware for exact numbers.

Optional: LIBERO Closed-Loop Sim Evaluation

To validate TRT accuracy in end-to-end robotic tasks, run the LIBERO closed-loop evaluation. This requires a separate environment setup (~10-30 min, MuJoCo simulator + dependencies).

# One-time LIBERO setup (~10 min)
bash gr00t/eval/sim/LIBERO/setup_libero.sh

# Activate LIBERO venv and install additional deps
source gr00t/eval/sim/LIBERO/libero_uv/.venv/bin/activate
uv pip install diffusers transformers accelerate safetensors torchcodec

# TRT full pipeline evaluation
python gr00t/eval/rollout_policy.py \
  --model-path checkpoints/GR00T-N1.7-LIBERO/libero_10 \
  --env-name "libero_sim/KITCHEN_SCENE3_turn_on_the_stove_and_put_the_moka_pot_on_it" \
  --n-episodes 20 --n-envs 1 --max-episode-steps 504 \
  --trt-engine-path ./gr00t_trt_deployment/engines \
  --trt-mode n17_full_pipeline

Run python scripts/deployment/build_trt_pipeline.py --steps benchmark to generate benchmarks for your hardware.

Platform-Specific Setup

Jetson and Spark platforms use different dependency stacks than dGPU. Thor and Spark use CUDA 13 with PyTorch 2.10.0 from the Jetson AI Lab cu130 index. Orin uses CUDA 12.6 with PyTorch 2.10.0 from the Jetson AI Lab cu126 index.

⚠️ aarch64 users (Spark / Thor / Orin): After running install_deps.sh, always activate the venv with source .venv/bin/activate && source scripts/activate_<platform>.sh (activate_spark.sh, activate_thor.sh, or activate_orin.sh) and run the example commands with plain python / torchrun, not uv run python / uv run torchrun — the latter re-syncs against the root pyproject.toml (x86_64 Python 3.12) and destroys the platform-specific environment. This applies to every platform below; see also the aarch64 note in the main README.

Shared Docker Workflow

The Docker steps below are identical across Thor, Spark, and Orin except the image tag (gr00t-thor, gr00t-spark, or gr00t-orin). Each platform section builds its image, then reuses these steps:

1. Download the model once on the host — see Download Model and Dataset.
2. Start an interactive session (replace <profile> with thor, spark, or orin):

docker run -it --rm --runtime nvidia --gpus all \
  --ipc=host \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  --network host \
  -v "$(pwd)":/workspace/repo \
  -v "${HF_HOME:-${HOME}/.cache/huggingface}":/root/.cache/huggingface \
  -w /workspace/repo \
  -e HF_TOKEN="${HF_TOKEN:-}" \
  gr00t-<profile> \
  bash

Then run the platform's TRT pipeline command inside the container (shown per platform below).

Jetson Thor Setup

Thor uses CUDA 13 and Python 3.12, which require a different dependency stack than x86 or Orin. Tested with JetPack 7.1. There are two ways to run on Thor: Docker (recommended) or bare metal.

cd docker && bash build.sh --profile=thor && cd ..

python scripts/deployment/build_trt_pipeline.py \
  --model-path checkpoints/GR00T-N1.7-LIBERO/libero_10 \
  --dataset-path demo_data/libero_demo \
  --embodiment-tag LIBERO_PANDA

# One-time install (temporarily copies the Thor pyproject.toml and uv.lock to repo root,
# installs NVPL libs, uv, Python deps, and builds torchcodec from source against the
# system FFmpeg runtime)
bash scripts/deployment/thor/install_deps.sh

# In each new shell
source .venv/bin/activate
source scripts/activate_thor.sh

DGX Spark Setup

Spark uses CUDA 13 and Python 3.12 like Thor, but requires a dedicated dependency stack and source-built flash-attn for sm121. There are two ways to run on Spark: Docker (recommended) or bare metal.

cd docker && bash build.sh --profile=spark && cd ..

python scripts/deployment/build_trt_pipeline.py \
  --model-path checkpoints/GR00T-N1.7-LIBERO/libero_10 \
  --dataset-path demo_data/libero_demo \
  --embodiment-tag LIBERO_PANDA

# One-time install (temporarily copies the Spark pyproject.toml and uv.lock to repo root,
# installs NVPL libs, uv, Python deps, source-builds flash-attn for sm121, and builds
# torchcodec from source against the system FFmpeg runtime)
bash scripts/deployment/spark/install_deps.sh

# In each new shell
source .venv/bin/activate
source scripts/activate_spark.sh

Jetson Orin Setup

Note: On Orin, only the DiT (action head) TRT export is currently supported. Use --export-mode dit_only instead of full_pipeline. Full pipeline support is in progress.

Orin uses CUDA 12.6 and Python 3.10 (JetPack 6.2), which require a different dependency stack than x86 or Thor. Tested with JetPack 6.2. There are two ways to run on Orin: Docker (recommended) or bare metal.

cd docker && bash build.sh --profile=orin && cd ..

python scripts/deployment/build_trt_pipeline.py \
  --model-path checkpoints/GR00T-N1.7-LIBERO/libero_10 \
  --dataset-path demo_data/libero_demo \
  --embodiment-tag LIBERO_PANDA \
  --export-mode dit_only

# One-time install (temporarily copies the Orin pyproject.toml and uv.lock to repo root,
# installs uv, Python deps, and builds torchcodec from source against JetPack's FFmpeg
# runtime)
bash scripts/deployment/orin/install_deps.sh

# In each new shell
source .venv/bin/activate
source scripts/activate_orin.sh

Orin storage tip: If your eMMC root is low on space, redirect the HuggingFace cache to an NVMe SSD with export HF_HOME=/path/to/ssd/.cache/huggingface before downloading models.

Orin TRT limitations: TRT 10.3 on Orin does not support the backbone (LLM) engine. Use --export-mode dit_only for the unified pipeline and --inference-mode tensorrt (DiT-only TRT, backbone runs in PyTorch) for inference:

python scripts/deployment/build_trt_pipeline.py \
  --model-path checkpoints/GR00T-N1.7-LIBERO/libero_10 \
  --dataset-path demo_data/libero_demo \
  --embodiment-tag LIBERO_PANDA \
  --export-mode dit_only

python scripts/deployment/standalone_inference_script.py \
  --model-path checkpoints/GR00T-N1.7-LIBERO/libero_10 \
  --dataset-path demo_data/libero_demo \
  --embodiment-tag LIBERO_PANDA \
  --traj-ids 0 \
  --inference-mode tensorrt \
  --trt-engine-path ./gr00t_trt_deployment/engines

Command-Line Arguments

build_trt_pipeline.py

standalone_inference_script.py

Files

Troubleshooting

Engine Build Fails

• Ensure you have enough GPU memory (16GB+ recommended for full pipeline)
• Try reducing workspace size: --workspace 4096
• Ensure TensorRT version matches your CUDA version
• LLM engine requires batch_size dimension handling when using custom shape profiles

ONNX Export Issues

• If export fails with COMPLEX128 error: ensure _simple_causal_mask is used (not HuggingFace's create_causal_mask)
• If masked_scatter size assertion fails: ensure visual_pos_masks has the correct number of True values matching deepstack tensor size
• Check that the dataset path is valid and contains at least one trajectory

Accuracy Issues

• If cosine < 0.99: check that LLM export does NOT include the final RMSNorm (backbone returns pre-norm hidden_states[-1])
• If output magnitude is ~12x too small: this is the norm bug — see above
• Run build_trt_pipeline.py --steps verify --export-mode action_head first to isolate backbone vs action head drift