Cross-modality Visualization Tools

Heatmap Visualization

cd Visualizaztion/Cross_Modality_Transformer_Visualization
mkdir pretrained
cd pretrained && mkdir distillbert-base-uncased

Then download all files in /distilbert-base-uncased and place these file in the directory distillbert-base-uncased.

Image

python main_img.py

We provide both feature map visualization and cross-modality attention visualize.

Video

python main_video.py

Binary Map Visualization

If we ask the model to learn fine-grained align we can generate binary map as below:

Refer to file test_region_mem.py for details.