Install

0. Preparation

If you have a CUDA environment, please make sure the environment variable CUDA_HOME is set. It will be compiled under CPU-only mode if no CUDA available.

Please make sure following the installation steps strictly, otherwise the program may produce:

NameError: name '_C' is not defined

how to check cuda:

echo $CUDA_HOME

If it print nothing, then it means you haven't set up the path/

Run this so the environment variable will be set under current shell.

export CUDA_HOME=/path/to/cuda-11.3

Notice the version of cuda should be aligned with your CUDA runtime, for there might exists multiple cuda at the same time.

If you want to set the CUDA_HOME permanently, store it using:

echo 'export CUDA_HOME=/path/to/cuda' >> ~/.bashrc

after that, source the bashrc file and check CUDA_HOME:

source ~/.bashrc
echo $CUDA_HOME

In this example, /path/to/cuda-11.3 should be replaced with the path where your CUDA toolkit is installed. You can find this by typing which nvcc in your terminal:

For instance, if the output is /usr/local/cuda/bin/nvcc, then:

export CUDA_HOME=/usr/local/cuda

1. Clone Repositories and Related Packages

1.1 image-textualization

git clone https://github.com/sterzhang/image-textualization.git
cd image-textualization
conda create --name image-textualization python=3.8 -y
conda activate image-textualization
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

1.2 detectron2

git clone https://github.com/facebookresearch/detectron2.git
cd detectron2
git checkout cc87e7ec
pip install -e .

1.3 GroundingDINO

cd ../image-textualization/filter
git clone https://github.com/IDEA-Research/GroundingDINO.git
cd GroundingDINO/
pip install -e .

1.4 SAM

Follow the SAM repo to install packages.

1.5 GRiT

Follow the instruction in GRiT repo.

1.6 LLama3

Follow the instruction in LLama3 repo.

2. Download Models

2.1 Create Folder

cd ../image-textualization
mkdir ckpt && cd ckpt

2.2 Checkpoint of GRiT

wget https://datarelease.blob.core.windows.net/grit/models/grit_b_densecap_objectdet.pth

2.3 Checkpoint of Groundingdino

wget -q https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth

2.4 Checkpoint of SAM

wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth