Video data pre-processing

June 13, 2017 · View on GitHub

This folder contains the following scripts:

add_frame_numbering.sh*: draws a huge number on each frame on a specific video;
dump_data_set.sh*: get images from videos for "traditional training";
objectify.sh*: convert sampled-data from video-indexed to object-indexed;
resize_and_split.sh*: see below;
resize_and_sample.sh*: see below;
sample_video.sh*: sample the <src video> into k time-subsampled <dst prefix>- videos;
VideoFolder.py: PyTorch data.Dataset's sub-class for video data loading.

Remove white spaces from file names

White spaces and scripts are not good friends. Replace all spaces with _ in your source data set with the following command, where you have to replace <root data dir> with the correct location.

rename -n "s/ /_/g" <root data dir>/*/*  # for a dry run
rename "s/ /_/g" <root data dir>/*/*     # to rename the files in <root data dir>!!!

In e-VDS35 we have already done this for you. If you plan to use your own data, this step is fundamental!

MatchNet mode

Resize and split videos in train-val

In order to speed up data loading, we shall resize the video shortest side to, say, 256. To do so run the following script.

./resize_and_split.sh <source dir> <destination dir>

By default, the script will

skip videos shorter than 144 frames (4.8s)
trim videos longer than 654 frames (21.8s)
use the last 2s for the validation split
resize the shortest side to 256px

These options can be varied and turned off by changing the header of the script, which now looks something like this.

################################################################################
# SETTINGS #####################################################################
################################################################################
# comment the next line if you don't want to skip short videos
min_frames=144
# comment the next line if you don't want to limit the max length
max_frames=564
# set split to 0 (seconds) if no splitting is required
split=2
################################################################################

File system

Our input data_set looks somehow like this.

data_set
├── barcode
│   ├── 20160613_140057.mp4
│   ├── 20160613_140115.mp4
│   ├── 20160613_140138.mp4
│   ├── 20160721_023437.mp4
├── bicycle
│   ├── 0707_2_(2).mov
│   ├── 0707_2_(4).mov

Running ./resize_and_sample.sh data_set/ processed-data yields

processed-data/
├── train
│   ├── barcode
│   │   ├── 20160613_140057.mp4
│   │   ├── 20160613_140115.mp4
│   │   ├── 20160613_140138.mp4
│   │   ├── 20160721_023437.mp4
│   ├── bicycle
│   │   ├── 0707_2_(2).mp4
│   │   ├── 0707_2_(4).mp4

TempoNet mode

Resize and sample videos, then splits in train-val

In order to speed up data loading, we shall resize the video shortest side to, say, 256. To do so run the following script.

./resize_and_sample.sh <source dir> <destination dir>

By default, the script will

skip videos shorter than 144 frames (4.8s)
trim videos longer than 654 frames (21.8s)
perform 5 subsamples, use 4 for training, 1 for validation
resize the shortest side to 256px

These options can be varied and turned off by changing the header of the script, which now looks something like this.

################################################################################
# SETTINGS #####################################################################
################################################################################
# comment the next line if you don't want to skip short videos
min_frames=144
# comment the next line if you don't want to limit the max length
max_frames=564
# set sampling interval: k - 1 train, 1 val
k=5
################################################################################

The output directory will contain as many folders as the total number of videos. Each folder will contain the individual splits.

From video-index- to class-major data organisation

If you would like to train against object classes instead of video indices (like explained in the paper), you also need to run

./objectify.sh <source dir> <destination dir>

This script generates a new directory containing as many folders as classes, filled with symbolic links from the source directory. You can use it for both videos and dumped-videos (images) data sets.

File system

Running ./resize_and_sample.sh data_set/ sampled-data yields

sampled-data/
├── train
│   ├── barcode-20160613_140057
│   │   ├── 1.mp4
│   │   ├── 2.mp4
│   │   ├── 3.mp4
│   │   └── 4.mp4
│   ├── barcode-20160613_140115
│   │   ├── 1.mp4
│   │   ├── 2.mp4

We can "objectify" the structure with ./objectify.sh sampled-data/ object-sampled-data and get

object-sampled-data/
├── train
│   ├── barcode
│   │   ├── 20160613_140057-1.mp4 -> ../../../sampled-data/train/barcode-20160613_140057/1.mp4
│   │   ├── 20160613_140057-2.mp4 -> ../../../sampled-data/train/barcode-20160613_140057/2.mp4
│   │   ├── 20160613_140057-3.mp4 -> ../../../sampled-data/train/barcode-20160613_140057/3.mp4
│   │   ├── 20160613_140057-4.mp4 -> ../../../sampled-data/train/barcode-20160613_140057/4.mp4
│   │   ├── 20160613_140115-1.mp4 -> ../../../sampled-data/train/barcode-20160613_140115/1.mp4
│   │   ├── 20160613_140115-2.mp4 -> ../../../sampled-data/train/barcode-20160613_140115/2.mp4
│   │   ├── 20160613_140115-3.mp4 -> ../../../sampled-data/train/barcode-20160613_140115/3.mp4

To train the discriminative feed-forward branch we need to dump our sampled-data with ./dump_data_set.sh sampled-data/ dumped-sampled-data. The file system will look like this

dumped-sampled-data/
├── train
│   ├── barcode-20160613_140057
│   │   ├── 1001.png
│   │   ├── 1002.png
│   │   ├── 1003.png
│   │   ├── 1004.png
│   │   ├── 1005.png
│   │   ├── 1006.png
│   │   ├── 1007.png

where the "training class" correspond to the video name. If we wish to train against object classes, then we can run ./objectify.sh dumped-sampled-data/ dumped-object-sampled-data and get the following

dumped-object-sampled-data/
├── train
│   ├── barcode
│   │   ├── 20160613_140057-1001.png -> ../../../dumped-sampled-data/train/barcode-20160613_140057/1001.png
│   │   ├── 20160613_140057-1002.png -> ../../../dumped-sampled-data/train/barcode-20160613_140057/1002.png
│   │   ├── 20160613_140057-1003.png -> ../../../dumped-sampled-data/train/barcode-20160613_140057/1003.png
│   │   ├── 20160613_140057-1004.png -> ../../../dumped-sampled-data/train/barcode-20160613_140057/1004.png
│   │   ├── 20160613_140057-1005.png -> ../../../dumped-sampled-data/train/barcode-20160613_140057/1005.png
│   │   ├── 20160613_140057-1006.png -> ../../../dumped-sampled-data/train/barcode-20160613_140057/1006.png
│   │   ├── 20160613_140057-1007.png -> ../../../dumped-sampled-data/train/barcode-20160613_140057/1007.png