HiFi-CS: Towards Open Vocabulary Visual Grounding For Robotic Grasping Using Vision-Language Models

March 12, 2025 · View on GitHub

This repository contains the training and testing scripts for HiFi-CS: a 2D visual grounding model for robotics. We also provide a method to create a custom dataset for training/testing models for more domain specific analysis. For more details, kindly refer to the main paper.

Abstract

Robots interacting with humans through natural language can unlock numerous applications such as Referring Grasp Synthesis (RGS), to generate a grasp pose for an object specified by a text query. RGS comprises two steps: visual grounding and grasp pose estimation. This paper introduces HiFi-CS, featuring hierarchical application of Featurewise Linear Modulation (FiLM) to fuse image and text embeddings, enhancing visual grounding for complex attribute rich text queries encountered in robotic grasping. Visual grounding associates an object in 2D/3D space with the natural language input and is studied in two scenarios: Closed Vocabulary (environments with a fixed set of known objects) and Open Vocabulary (previously unseen objects). Recent studies leverage powerful Vision-Language Models (VLMs) for visually grounding free flowing natural language in real-world robotic execution. However, comparisons in complex, cluttered environments with multiple instances of the same object have been lacking. HiFi-CS features a lightweight decoder combined with a frozen VLM and outperforms competitive baselines in closed vocabulary settings while being 100x smaller in size. Our model can be effectively used in parallel with open-set object detectors like GroundedSAM to enhance open-vocabulary performance. We validate our approach through real-world RGS experiments using a 7-DOF robotic arm, achieving a 90.33% visual grounding accuracy in 15 tabletop scenes.

Installation

Clone the repository:

git clone https://github.com/vineet2104/HiFiCS-VisualGrounding.git
cd HiFiCS-VisualGrounding-main/

Create and activate a conda environment:

conda env create -f hifics_env.yml
conda activate hifics

Install CLIP

pip install git+https://github.com/openai/CLIP.git

Training HiFi-CS

We offer support to train the model on 2 datasets - RoboRefIt and OCID-VLG. Please follow the steps below to download the datasets and save it in the required format for easy training and testing. Training configurations can be modified by making appropriate changes in experiments/config.yaml

Download the RoboRefIt dataset:

Download the dataset from link and move it to the ./datasets/ folder

Download and prepare the OCID-VLG dataset:

A. Download the dataset: Download the dataset from link and move it to the ./datasets/ folder.

B. Process the dataset: Run the following command to process the data and prepare train-test splits:

python datasets/prepare_ocidvlg.py

At this point, the ./datasets/ folder should look like:

./datasets/
├── dataloader.py
├── load_ocidvlg.py
├── prepare_ocidvlg.py
├── OCID-VLG/
└── final_dataset/
    ├── testA/
    ├── testB/
    ├── train/
        ├── depth/
        ├── depth_colormap/
        ├── image/
        ├── mask/
        roborefit_train.json
└── ocidvlg_final_dataset/
    ├── test/
    ├── train/
        ├── depth/
        ├── image/
        ├── mask/
        ocid_vlg_train.json

Train HiFi-CS on RoboRefIt:
```
python training.py config.yaml 0
```
Train HiFi-CS on OCID-VLG:
```
python training.py config.yaml 1
```

Testing the Model

Trained models are saved in ./logs folder. After training HiFi-CS on RoboRefIt and OCID-VLG separately, we offer support to evaluate these models on their respective test sets.

Evaluate hifics-roborefit-default on RoboRefIt Test A and Test B splits:
```
python score.py config.yaml 0 0
```
Note - change the test_split variable in test_configuration in the experiments/config.yaml file to evaluate on testA or testB
Evaluate hifics-ocidvlg-default on OCID-VLG Test split:
```
python score.py config.yaml 1 1
```

Creating your Custom 2D Visual Grounding dataset

We provide a simple framework to create your own 2D visual grounding dataset to either train HiFi-CS or evaluate trained models using a small test corpus. This method assumes you have captured RGB images using either a mobile or robot-mounted camera in the .png format. We provided a notebook: data-gen-2DVG.ipynb to load a sample image and generate masks of all objects in the image. Before running the notebook, please follow the steps below to load all supporting scripts -

Using Segment Anything Model (SAM) to generate candidate masks: SAM is a powerful model trained on a large dataset for object segmentation. We load this model from the Grounded-SAM github repository. Please follow the instructions below to import all code files needed (Run the commands in the same conda environment)

A. Download the model checkpoint: Download the file from link and move it to the main folder

B. Install Grounded-SAM and partial dependencies for running the notebook (Run these commands from the main working directory):
```
git clone https://github.com/IDEA-Research/Grounded-Segment-Anything.git
cd Grounded-Segment-Anything
python -m pip install -e segment_anything
```
Run the notebook: Run all cells in the data-gen-2DVG.ipynb to load a sample image and generate masks for all objects in the image. However, not all masks correspond to meaninful entities. We provide a helpful user interface to manually verify each generated mask from the RGB image. This results in (RGB-Mask) pairs that can be crowd sourced to generate referring text for each mask.

RoboRESTest Corpus:

The test corpus used in our paper can be downloaded from here. Please unzip the folder and save it in the ./datasets folder. The file ./datasets/RoboRESTest/robores_test.json contains a list of tuples, where each tuple corresponds to (RGB-Mask-Text) for training/testing HiFi-CS or other 2D visual grounding datasets. Each tuple has the following structure:

{
    "num": 0,
    "text": "Please grab the silver bowl on the left",
    "rgb_path": "./RoboRESTest/sample120.png",
    "mask_path": "./RoboRESTest/masks/sample120_mask_11.png",
    "lighting": "D",
    "clutter": "M",
    "setup": "TT"
}

Description of JSON variables:

 - **text:** Referring text corresponding to a given mask of the RGB image
 - **rgb_path:** Path to RGB image
 - **mask_path:** Path to B/W mask
 - **lighting:** Lighting category: D - Dim, M - Medium, B - Bright
 - **clutter:** Clutter category: L - Low, M - Medium, H - High
 - **setup:** Background setup: TT - Table Top, S - Multilayered Shelf, D - Drawers, C - Chair, M - Human hand

Citation

@article{bhat2024hifi,
  title={HiFi-CS: Towards Open Vocabulary Visual Grounding For Robotic Grasping Using Vision-Language Models},
  author={Bhat, Vineet and Krishnamurthy, Prashanth and Karri, Ramesh and Khorrami, Farshad},
  journal={arXiv preprint arXiv:2409.10419},
  year={2024}
}