README.md

May 20, 2025 · View on GitHub

Visuotactile-Based Learning for Insertion with Compliant Hands

@article{azulay2025visuotactile,
  title={Visuotactile-Based Learning for Insertion With Compliant Hands},
  author={Azulay, Osher and Ramesh, Dhruv Metha and Curtis, Nimrod and Sintov, Avishai},
  journal={IEEE Robotics and Automation Letters},
  volume={10},
  number={4},
  pages={4053--4060},
  year={2025},
  publisher={IEEE}
}

Overview
Getting Started
- Dependencies
- Installation
Usage
License

Tactile Rendering in IsaacGym

Built on top of TACTO
This project extends allsight_sim by integrating support for round-shaped tactile rendering for IsaacGym. > For more information about the base simulation framework of the tactile images, please refer to the original allsight_sim project.

Overview

This repository provides the official implementation of Visuotactile-Based Learning for Insertion with Compliant Hands.

Getting Started

Dependencies

Project was tested on:

ubuntu >=18.04
python >= 3.8
cuda >=11.7
built on ROS melodic\noetic

Installation

First, Create Mamba-Forge environment:

mamba create -n insertion python=3.8
mamba activate insertion

Install IsaacGym:

Download the Isaac Gym Preview 4 release from the website, then follow the installation instructions in the documentation.

Once Isaac Gym is installed and samples work within your current python environment, install this repo:
```
pip install -e .
```

Clone and install this repo:

cd isaacgym/python
git clone https://github.com/osheraz/IsaacGymInsertion
cd IsaacGymInsertion && pip install -e .

Usage

Brief

Train a teacher policy using privliged information with RL
Train a student policy using visual and\or tactile information
Deploy on real-robot

Note: All configs, logs, and model weights are saved in a folder within outputs/.

As noted before, this repository includes implementations for tactile sensing (built on top of TACTO) and point cloud observation.

To visualize each input, refer to the cfg/task/FactoryTaskInsertionTactile.yaml file and set True for each input you would like to enable.

To visualize tactile inputs, set the following parameters:

tactile=True  
tactile_display_viz=True

You can adjust the tactile parameters in the file allsight\conf\sensor\config_allsight_white.yml.

To visualize depth, point cloud, or masks, set the following parameters:

external_cam=True  
pcl_cam\depth_cam\seg_cam=True 
display=True

Training

Step 1: Teacher Policy training

cd isaacgyminsertion
scripts/train_s1.sh

Step 2: Evaluate the teacher policy

scripts/eval_s1.sh

Step 3: Visuotactile Student distillation

Modify scripts/train_s2.sh given which student policy you want to use:

# for segmented-pcl:
train.ppo.obs_info=True \ train.ppo.img_info=False \ train.ppo.seg_info=False \ train.ppo.pcl_info=True \ train.ppo.tactile_info=False \ task.env.tactile=False \ task.external_cam.external_cam=True \ task.external_cam.depth_cam=False \ task.external_cam.seg_cam=True \ task.external_cam.pcl_cam=True \```

# for segmented-depth:
train.ppo.obs_info=True \ train.ppo.img_info=True \ train.ppo.seg_info=True \ train.ppo.pcl_info=False \ train.ppo.tactile_info=False \ task.env.tactile=False \ task.external_cam.external_cam=True \ task.external_cam.depth_cam=True \ task.external_cam.seg_cam=True \ task.external_cam.pcl_cam=False \```

# for visualtactile (pcl+tactile):
train.ppo.obs_info=True \ train.ppo.img_info=False \ train.ppo.seg_info=False \ train.ppo.pcl_info=True \ train.ppo.tactile_info=True \ task.env.tactile=True \ task.external_cam.external_cam=True \ task.external_cam.depth_cam=False \ task.external_cam.seg_cam=True \ task.external_cam.pcl_cam=True \```

Next, train the policy:

scripts/train_s2.sh

Step 4: Evaluate the student

Evaluate with same arguments in eval_s2.sh:

scripts/eval_s2.sh

Deploy

See - deploy_instructions

Acknowledgements

License

This repository is licensed under the Apache License. Feel free to use, modify, and distribute the code as per the terms of this license.

↑ Return