Massimo Bosetti¹  Shibingfeng Zhang³  Benedetta Liberatori¹  Giacomo Zara¹  Paolo Rota¹  Elisa Ricci^1,2

¹University of Trento, IT   ²Fondazione Bruno Kessler (FBK), IT   ³University of Bologna, IT

Abstract: Vision-language models (VLMs) have demonstrated remarkable performance across various visual tasks, leveraging joint learning of visual and textual representations. While these models excel in zero-shot image tasks, their application to zero-shot video action recognition (ZSVAR) remains challenging due to the dynamic and temporal nature of actions. Existing methods for ZS-VAR typically require extensive training on specific datasets, which can be resource-intensive and may introduce domain biases. In this work, we propose Text-Enhanced Action Recognition (TEAR), a simple approach to ZS-VAR that is training-free and does not require the availability of training data or extensive computational resources. Drawing inspiration from recent findings in vision and language literature, we utilize action descriptors for decomposition and contextual information to enhance zero-shot action recognition. Through experiments on UCF101, HMDB51, and Kinetics-600 datasets, we showcase the effectiveness and applicability of our proposed approach in addressing the challenges of ZS-VAR.

Link to arxiv paper: TEAR: Text-Enhanced Zero-Shot Action Recognition link

To start

# clone project
git clone https://github.com/MaXDL4Phys/tear.git
cd tear

# create environment and install requirements

conda create --name tear python=3.9
conda activate tear
pip install -r requirements.txt

Then go to https://github.com/openai/CLIP to install CLIP according to the instructions.

Dataset Downloading and Preparation

This Python script extracts frames from videos in a dataset (like UCF101, HMDB51, or K600) and saves them as images while also handling directory creation and management.

UCF101

The code is a Python command to extract frames from the "ucf101" dataset, specifying paths for input and output.

# Download the dataset from https://www.crcv.ucf.edu/data/UCF101/UCF101.rar, then unzip the file
#extract frames from videos
cd language_driven_action_recognition_localization/src/utils
python extract_frames.py --input <the path to downloaded dataset> --output <where you want to save the extracted frames> --dataset=ucf101

HMDB51

The code is a Python command to extract frames from the "hmdb51" dataset, specifying paths for input and output.

#Download the dataset from https://serre-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database/, then unzip the file
#extract frames from videos
cd language_driven_action_recognition_localization/src/utils
python extract_frames.py --input <the path to downloaded dataset> --output <where you want to save the extracted frames> --dataset=hmdb51

KINETICS 600

Kinetics-600 Download:

Clone repo and enter directory

git clone https://github.com/cvdfoundation/kinetics-dataset.git
cd kinetics-dataset

Download tar gzip files

This will create two directories, k600 and k600_targz. Tar gzips will be in k600_targz, you can delete k600_targz directory after extraction.

bash ./k600_downloader.sh

Extract tar gzip files

To extract the validation videos in folder related to classes use the script k600_video_extractor.sh. This script will extract the videos in the folder related to the classes.

bash ./k600_video_extractor.sh

Extract frames

cd language_driven_action_recognition_localization/src/utils
python extract_frames.py --input <the path to downloaded dataset> --output <where you want to save the extracted frames> --dataset=k600
#Create a symbolic link to the folder where extracted images are saved at language_driven_action_recognition_localization/data

Experiments

UCF101

python scripts/zsacr-ucf101-run.py

HMDB51

python scripts/zsacr-hmdb51-run.py

HMDB51 Llama

python scripts/zsacr-hmdb51-run.py

KINETICS 600

python scripts/zsacr-k600-run.py

Citation

Please consider citing our paper in your publications if the project helps your research.

@article{Bosetti_2024_TEAR,
        title={Text-Enhanced Zero-Shot Action Recognition: A training-free approach},
        author={Massimo Bosetti and Shibingfeng Zhang and Bendetta Liberatori and Giacomo Zara and Elisa Ricci and Paolo Rota},
        year={2024},
        journal={arXiv}
}