Sequential Covariate Shift Detection
September 11, 2022 ยท View on GitHub
Description
This repository is for the implementation of our paper, Sequential Covariate Shift Detection Using Two Sample-Tests (ICML 2022).
How to Run
This repository includes different types of scripts; 1) Running experiments, 2) Drawing results, 3) Analyzing quantitative results, 4) Exporting dataset.
Running experiments
This repository includes the python implementations of multiple algorithms including ours. Each implementation has two types of scripts; one is for synthetic shift, another one is for natural shift. The file name format has the following pattern
detect_[ALGORITHM_NAME]_schedule[TYPE].py
[ALGORITHM_NAME] can be "dk", "icm", "kds", "mmd", "ours", "wald", and [TYPE] can be "" for a synthetic shift, and "_natural" for natural shift. For example, you need to run 'detect_ours_schedule.py' for the synthetic shift for our algorithm like the below.
python detect_ours_schedule.py
Each script file has different command line arguments, and most of them are self-explanatory. Here is the description for our algorithm.
| Name | Description |
|---|---|
| root_dir | Root directory of exported dataset file |
| schedule_file | Text file describing the schedule (c3, gradinc, gradincdec) |
| holdout_interval | Interval for constructing hold-out set. -1 for no hold-outset, and 2 and 5 for H2, H5 algorithm in our paper |
| CP_window_size | Window size (W) for constructing Clopper-Pearson Interval. W=100 is used. |
| n_samples | Number of samples (m) |
| perturb_ratio | Ignore this. 1.0 is fixed as default |
| batch_size | Batch size. B=5 is used |
| lr | Learning rate for updating source-target classifier |
| momentum | Momentum for updating source-target classifier |
| weight_decay | Weight Decay for updating source-target classifier |
| n_repeats | Number of repetition (R) |
| log_dir | Logging file directory |
| result_dir | Experiment results directory |
| alpha | Significance level (FPR Bound) |
| seed | Random seed |
| device | Which device will be used. (CPU or CUDA) |
Drawing Results
Two scripts are used for drawing results; one is for synthetic shifts (draw_seq_sch_synt.py), another one is for natural shifts (draw_seq_sch_natural.py).
Analyzing quantitative results
Similar to the drawing results, there are two scripts: 'analyze_data.py' (Synthetic Shifts) and 'analyze_data_natural.py' (Natural shifts).
Exporting datasets
Each dataset is exported to numpy files for each experiments.
| Script | Description |
|---|---|
| export_dataset.py | Synthetic shift using ImageNet |
| export_dogs.py | Natural shift using ImageNet (Dogs) |
| export_iwildcam.py | Natural shift using IWildCam |
| export_py150.py | Natural shift using Py150 |
Citation
@inproceedings{jang2022sequential,
title={Sequential Covariate Shift Detection Using Classifier Two-Sample Tests},
author={Jang, Sooyong and Park, Sangdon and Lee, Insup and Bastani, Osbert},
booktitle={International Conference on Machine Learning},
pages={9845--9880},
year={2022},
organization={PMLR}
}