Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser

python ./utils/download_dataset.py --videos_saved_dir ./data/raw_videos --label_all_dataset ./data/example.csv

After downloading the videos to data/raw_videos/, one should extract video frames in 8fps from each video with the following command. We show the frame extraction of the video downloaded from the above command.

python ./utils/extract_frames.py --video_path ./data/raw_videos --out_dir ./data/video_frames

However, there are some potential issues when using these downloaded videos. For more details, please see here.

The visual features and pseudo labels can also be generated with the following command if the video frames from all videos are prepared in data/video_frames/:

python ./utils/clip_preprocess.py --print_progress

For example, the features and pseudo labels of the video downloaded above can be generated in data/example_v_feats/ and data/example_v_labels/, respectively, with the following command:

python ./utils/clip_preprocess.py --label_all_dataset ./data/example.csv --pseudo_labels_saved_dir ./data/example_v_labels --visual_feats_saved_dir ./data/example_v_feats --print_progress

The audio features and pseudo labels can also be generated with the following command if raw videos are prepared in data/raw_videos/:

python ./utils/clap_preprocess.py --print_progress

For example, the features and pseudo labels of the video downloaded above can be generated in data/example_a_feats/ and data/example_a_labels/, respectively, with the following command:

python ./utils/clap_preprocess.py --label_all_dataset ./data/example.csv --pseudo_labels_saved_dir ./data/example_a_labels --audio_feats_saved_dir ./data/example_a_feats --print_progress

> data/
    ├── AVVP_dataset_full.csv
    ├── AVVP_eval_audio.csv
    ├── AVVP_eval_visual.csv
    ├── AVVP_test_pd.csv
    ├── AVVP_train.csv
    ├── AVVP_val_pd.csv
    ├── feats/
    │     └── r2plus1d_18/
    │     └── res152/
    │     └── vggish/
    ├── CLIP/
    │     └── features/
    │     └── segment_pseudo_labels/
    ├── CLAP/
    │     └── features/
    │     └── segment_pseudo_labels/
    ├── video_frames/ (optional)
    │     └── 00BDwKBD5i8/
    │     └── 00fs8Gpipss/
    │     └── ...
    └── raw_videos/ (optional)
          └── 00BDwKBD5i8.mp4
          └── 00fs8Gpipss.mp4
          └── ...

> models/
    ├── model_VALOR/
    │        └── checkpoint_best.pt
    ├── model_VALOR+/
    │        └── checkpoint_best.pt
    └── model_VALOR++/
             └── checkpoint_best.pt