GDumb

This repository contains simplified code for the paper:

GDumb: A Simple Approach that Questions Our Progress in Continual Learning, ECCV 2020 (Oral: Top 2%)
Ameya Prabhu, Philip Torr, Puneet Dokania

[PDF] [Slides] [Bibtex]

Installation and Dependencies

• Install all requirements required to run the code on a Python 3.x by:

# First, activate a new virtual environment
$ pip3 install -r requirements.txt

• Create two additional folders in the repository data/ and logs/ which will store the datasets and logs of experiments. Point --data_dir and --log_dir in src/opts.py to locations of these folders.

• Select Imagenet100 from Imagenet using this link and TinyImagenet from this link and convert them to ImageFolder format with train and test splits.

Usage

• To run the GDumb model you can simply specify conditions from argument, an example command below:

$ python main.py --dataset CIFAR100 --num_classes_per_task 5 --num_tasks 20 --memory_size 500 --num_passes 256 --regularization cutmix --model ResNet --depth 32 --exp_name my_experiment_name

Arguments you can freely tweak given a dataset and model:

• Number of classes per task (--num_classes_per_task)
• Number of tasks (--num_tasks)
• Maximum memory size (--memory_size)
• Number of classes to pretrain a dumb model (--num_pretrain_classes)
• Number of passes through the memory for learning the dumb model and pretraining (--num_passes and --num_pretrain_passes)

To add your favorite dataset:

• Convert it to ImageFolder format (as in imagenet) with train and test folders
• Add the dataset folder name exactly to src/opts.py
• Add dataset details to get_statistics() function in src/dataloader.py
• Run you model with --dataset your_fav_dataset!

Additional details and default hyperparameters can be found in src/opts.py

• To replicate the complete set of experiments, copy scripts/replicate.sh to src/ and run with substituting $SEED with {0,1,2}:

$ bash replicate.sh $SEED

Similarly, other scripts can replicate results for specific formulations.

Results

After running replicate.sh you should get results somewhat like these:

GDumb Model	Mem (k)	Table	Accuracy
MNIST-MLP-100	300	3,8	89.1 ± 0.4
MNIST-MLP-100	500	3	90.2 ± 0.4
MNIST-MLP-400	500	4	91.9 ± 0.5
MNIST-MLP-400	4400	5,6	97.8 ± 0.1
SVHN-ResNet18	4400	3	93.4 ± 0.1
CIFAR10-ResNet18	200	3	35.0 ± 0.4
CIFAR10-ResNet18	500	3,4,8	45.4 ± 1.9
CIFAR10-ResNet18	1000	3,4	61.2 ± 1.0
CIFAR100-ResNet32	2000	5	24.3 ± 0.4
TinyImageNet-DenseNet-100-12-BC	9000	6	57.32 (best of 3)

Extensibility to other setups

• Settings can be tweaked by adjusting the above parameters. Additionally, GDumb can be used in a wide-variety of settings beyond current CL formulations:
  • GDumb is sortof robust against drastic variations in sample orders given the same/similar set of samples land in memory, hence this implementation abstracts the sampling process out.
  • Masking is to be used to handle dynamic variations to likely subset of classes, adding class-priors, handling scenarios like cost-sensitive classification

If you discover any bugs in the code please contact me, I will cross-check them with my nightmares.

Citation

We hope GDumb is a strong baseline and comparison, and the sampler or masking introduced are useful for your cool CL formulation! To cite our work:

@inproceedings{prabhu2020greedy,
  title={GDumb: A Simple Approach that Questions Our Progress in Continual Learning},
  author={Prabhu, Ameya and Torr, Philip and Dokania, Puneet},
  booktitle={The European Conference on Computer Vision (ECCV)},
  month={August},
  year={2020}
}