EnsembleVLA: Ensemble Learning for Vision-Language Action Models (ICML 2026)
July 16, 2026 · View on GitHub
EnsembleVLA: Ensemble Learning for Vision-Language Action Models (ICML 2026)
1Harbin Institute of Technology, Shenzhen 2PengCheng Laboratory 3Shenzhen Loop Area Institute
📖 Introduction
Recent Vision-Language-Action (VLA) models have demonstrated strong capabilities in robotic manipulation, yet how to effectively ensemble heterogeneous VLAs remains largely underexplored. Unlike discriminative predictors, generative action policies often represent high-dimensional and multimodal action distributions, which makes conventional ensemble strategies such as voting or direct action averaging insufficient for robust policy composition.
We propose EnsembleVLA, an energy-based framework for principled composition of diverse VLA policies. EnsembleVLA formulates diffusion-based and flow-based VLA models under a unified energy perspective, where additive energy aggregation naturally induces policy composition at the distribution level. This formulation allows multiple pretrained policies to remain frozen while being aggregated into a stronger ensemble policy that can exploit their complementary action modes.
Building on this compositional view, EnsembleVLA introduces learnable composition weights for dynamic policy balancing, together with a confidence-aware gating mechanism that modulates bounded residual corrections. The resulting policy maintains compatibility with the RoboTwin2 rollout interface: each base policy is queried under its own observation and language context, and the composed action is executed in the simulator for stable task completion.
⚙️ Environment Setup
This project should be installed on top of a working RoboTwin2 environment. Please first follow the official RoboTwin2 documentation for installation, asset download, configuration files, and policy evaluation:
- RoboTwin2 documentation: https://robotwin-platform.github.io/doc/index.html
A typical setup is:
git clone https://github.com/MingC715/EnsembleVLA.git
cd EnsembleVLA
conda create -n RoboTwin python=3.10 -y
conda activate RoboTwin
mkdir -p external
git clone https://github.com/RoboTwin-Platform/RoboTwin.git external/RoboTwin
cd external/RoboTwin
bash script/_install.sh
bash script/_download_assets.sh
python script/update_embodiment_config_path.py
cd ../..
After RoboTwin2 is installed, make the assets visible to this repository and check that the task, camera, and embodiment configuration files point to your local installation:
ln -s /path/to/RoboTwin/assets assets
Recommended runtime variables for headless GPU evaluation are:
export PYOPENGL_PLATFORM=egl
export MUJOCO_GL=egl
export SAPIEN_OFFSCREEN_ONLY=1
export NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics
export TORCH_EXTENSIONS_DIR=${TORCH_EXTENSIONS_DIR:-$HOME/.cache/torch_extensions}
The base policy backends should be available under policy/:
policy/DP/ # Diffusion Policy backend
policy/DP3/ # 3D Diffusion Policy backend
policy/pi05/ # pi0.5 / openpi backend
Then install the EnsembleVLA composition/evaluation dependencies and the base policy backends you plan to use:
# EnsembleVLA composition & evaluation dependencies
pip install -r policy/Ensemble-Policy-easy/requirements.txt
# Base policy backends (install the ones you evaluate):
pip install -e policy/DP # Diffusion Policy (DP)
pip install -e policy/DP3/3D-Diffusion-Policy # 3D Diffusion Policy (DP3)
pip install -e policy/pi05 # pi0.5 / openpi
For backend-specific setup, also follow the RoboTwin2 policy pages and the corresponding upstream repositories.
📦 Checkpoints
We release both the lightweight EnsembleVLA heads and the required base policy
checkpoints on Hugging Face:
mingchens/EnsembleVLA.
After downloading the release assets, place or symlink them under
best_checkpoint/ using the layout below.
Download with Git LFS:
git lfs install
git clone https://huggingface.co/mingchens/EnsembleVLA hf_assets
rsync -a hf_assets/best_checkpoint/ best_checkpoint/
Expected best-checkpoint layout:
best_checkpoint/
+-- dp+dp3/<task>/
| +-- ensemble_checkpoint/best.pt
| +-- base_dp/<ckpt>.ckpt
| +-- base_dp3/<ckpt>.ckpt
+-- dp+pi0.5/<task>/
+-- ensemble_checkpoint/best.pt
+-- base_dp/<ckpt>.ckpt
+-- base_pi05_checkpoint_dir/
+-- model.safetensors
+-- metadata.pt
+-- assets/<task>/norm_stats.json
Only inference checkpoints are required for evaluation. Raw optimizer states, training logs, and rollout logs are not required.
DP + DP3 Checkpoints
DP + pi0.5 Checkpoints
Each pi0.5 base also needs metadata.pt and the assets/ folder from the same base_pi05_checkpoint_dir/ (both on Hugging Face).
See docs/checkpoints.md for the full checkpoint manifest.
🧪 Evaluation
DP + DP3 example:
conda activate RoboTwin
bash policy/Ensemble-Policy-easy/eval_wlearn.sh \
beat_block_hammer DP DP3 0 best L515 100 0 100 base base \
best_checkpoint/dp+dp3/beat_block_hammer/ensemble_checkpoint
DP + pi0.5 example:
conda activate RoboTwin
bash policy/Ensemble-Policy-easy/eval_wlearn.sh \
click_alarmclock DP pi05 0 best L515 100 0 100 100 1000 \
best_checkpoint/dp+pi0.5/click_alarmclock/ensemble_checkpoint
Arguments:
task policy1 policy2 gpu ensemble_ckpt camera data_num seed test_num policy1_ckpt policy2_ckpt output_dir [composition_mode] [policy2_use_pytorch]
For DP + pi0.5 evaluation, pass the released composition mode argument used by
eval_wlearn.sh and set policy2_use_pytorch=true.
🗂️ Project Structure
EnsembleVLA-ICML2026/
+-- README.md
+-- docs/
| +-- checkpoints.md
+-- description/
+-- envs/
+-- task_config/
+-- script/
| +-- eval_policy.py
+-- policy/
| +-- DP/
| +-- DP3/
| +-- pi05/
| +-- Ensemble-Policy-easy/
| +-- composition.py
| +-- energy_head.py
| +-- eval.py
| +-- eval_wlearn.py
| +-- eval_wlearn.sh
+-- best_checkpoint/
+-- dp+dp3/
+-- dp+pi0.5/
🙏 Acknowledgement
We thank the open-source projects RoboTwin2 and GPC for making their code and tools publicly available.
📚 Citation
@inproceedings{song2026ensemblevla,
title={EnsembleVLA: Ensemble Learning for Vision-Language Action Models},
author={Song, Mingchen and Deng, Xiang and Wei, Jie and Jiang, Dongmei and Nie, Liqiang and Guan, Weili},
booktitle={International Conference on Machine Learning},
year={2026}
}