README.md

⚙️ Setup

1) Create and activate environment

conda create -n unilact python=3.10 -y
conda activate unilact

2) Clone the repo and install dependencies

git clone https://github.com/manishgovind/uniact-vla.git
cd UniLACT
pip install -r requirements.txt

3) (Optional) Set project root

export PROJECT_UNILACT_ROOT=/path/to/UniLACT

---

🤗 Model Zoo

Pretrained and finetuned checkpoints are hosted on Hugging Face : UniLACT-models. Each checkpoint directory contains config.yaml and pytorch_model.bin.

Model Name	HF Path	Note
unilarn-calvin	unilarn_trained_on_calvin	Stage-1 UniLARN trained on CALVIN RGB-D.
unilarn-oxe	unilarn_trained_on_oxe	Stage-1 UniLARN trained on Open X-Embodiment.
unilact-pretrain-calvin	unilact_pretrained_on_calvin	Stage-2 depth-aware latent pretraining on CALVIN.
unilact-pretrain-oxe	unilact_pretrained_on_oxe	Stage-2 depth-aware latent pretraining on OXE.
unilact-calvin	unilact_finetuned_on_calvin	Fine-tuned on CALVIN for ABC→D evaluation (in-domain pretraining).
unilact-calvin-ood	unilact_finetuned_on_calvin_from_pretrained_oxe	Fine-tuned on CALVIN after out-of-domain OXE pretraining.

Download all checkpoints

pip install -U "huggingface_hub[cli]"
export PROJECT_UNILACT_ROOT=/path/to/UniLACT

hf download mgovind7/UniLACT --local-dir ${PROJECT_UNILACT_ROOT}/checkpoints

Download a single checkpoint

# Stage 3 fine-tuned model used for CALVIN evaluation
hf download mgovind7/UniLACT \
  --local-dir ${PROJECT_UNILACT_ROOT}/checkpoints \
  --include "unilact_finetuned_on_calvin/*"

After downloading, point the training or evaluation configs at the local checkpoint directory. For example, set unilarn_path in unilact/configs/train/pretrain_unilact_on_calvin.yaml to ${PROJECT_UNILACT_ROOT}/checkpoints/unilarn_trained_on_calvin, or set UniLACT_PATH in scripts/evaluate_unilact_on_calvin.sh to ${PROJECT_UNILACT_ROOT}/checkpoints/unilact_finetuned_on_calvin.

---

🚀 Training (3 Stages)

UniLACT training consists of three stages:

1. Stage 1: Unified latent action learning (UniLARN)
2. Stage 2: Unified latent pretraining
3. Stage 3: Action fine-tuning

Training is driven by YAML configs under unilact/configs/train/ and configs under unilarn/configs/train.

---

Stage 1 — UniLARN (Unified Latent Action Learning)

# (Update this command to your UniLARN entrypoint/config if different)
cd ${PROJECT_UNILACT_ROOT}/unilarn
accelerate launch --main_process_port <master_port> train_unilarn.py --config_path "${PROJECT_UNILACT_ROOT}/unilarn/configs/train/train_unilarn_on_calvin.yaml"

---

Stage 2 — Unified latent Pretraining

# (Update this command to your pretraining config if different)
cd ${PROJECT_UNILACT_ROOT}/unilact/train
accelerate launch --main_process_port <master_port>  train_unilact.py --config_path "${PROJECT_UNILACT_ROOT}/unilact/configs/train/pretrain_unilact_on_calvin.yaml"

---

Stage 3 — Fine-tuning

cd ${PROJECT_UNILACT_ROOT}/unilact/train
accelerate launch --main_process_port <master_port> train_unilact.py   --config_path "${PROJECT_UNILACT_ROOT}/unilact/configs/train/finetune_unilact_on_calvin.yaml"

---

Evaluation on CALVIN (ABC→D) Benchmark

Install the CALVIN benchmark in the same conda environment (unilact) by following the official CALVIN repository instructions.

conda activate unilact
export PROJECT_UNILACT_ROOT=/path/to/UniLACT
cd ${PROJECT_UNILACT_ROOT}/scripts
bash evaluate_unilact_on_calvin.sh

---

⏳ To-Do

• Training Data preparation
• Support for OXE-pretraining
• Release pretrained and finetuned model checkpoints

🙏 Acknowledgements

This project builds on top of Moto, CALVIN. We thank the authors for their open-sourced work.

📝 Citation

If you find our work useful, please cite:

@article{govind2026unilactdepthawarergblatent,
  title= {UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models},
  author= {Manish Kumar Govind and Dominick Reilly and Pu Wang and Srijan Das},
  journal={arXiv preprint arXiv:2602.20231},
  year={2026}
}