LaViSE

This is the official repository for paper "Explaining Deep Convolutional Neural Networks via Unsupervised Visual-Semantic Filter Attention" to appear in CVPR 2022.

Authors: Yu Yang, Seungbae Kim, Jungseock Joo

Datasets

Common Objects in Context (COCO)

• Please follow the instructions in the COCO API README and here to download and setup the COCO data.

Visual Genome (VG)

• Please follow the instructions in the README of the python wrapper for the Visual Genome API and here.

GloVe

• We load the pretrained GloVe word embeddings directly from the torchtext library.

Social Media Photographs of US Politicians (PoP)

• The list of entities used to discover new concepts is provided in data/entities.txt.

Getting started

Requirements

Required packages can be found in requirements.txt.

Usage

Train an explainer with

python train_explainer.py

Explain a target filter of any model with

python infer_filter.py

More features will be added soon! 🍻

Citation

@inproceedings{yang2022explaining,
    author    = {Yang, Yu and Kim, Seungbae and Joo, Jungseock},
    title     = {Explaining Deep Convolutional Neural Networks via Unsupervised Visual-Semantic Filter Attention},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    year      = {2022},
}