README.md

A Practical Debugging Tool for Training Deep Neural Networks

Experiments

Cockpit Package • Docs • License

Experiments

To showcase the capabilities of Cockpit we performed several experiments illustrating the usefulness of our debugging tool in our paper. The code for all experiments, as well as the generated data is presented in this repository. For a discussion of those experiments please refer to our paper.

We provide an overivew of all experiments here, splitting them up in experiments that are presented in the main body of our paper and ones that are shown in the appendix.

All necessary requirements (besides Cockpit) can be found in requirements.txt

If you decide to re-run the code, you may want to delete the ~/temp/data_deepobs directory, that will be created to share the data sets among experiments, afterwards.

All experiments include some run.py file to reproduce the results. Our original results are also stored in each experiment folder in a results.zip file. The plot.py script produces the Figures shown in the paper.

Experiment	Description
01_benchmark	Benchmark the run time of individual instruments and configurations of Cockpit. Reproduces Figure 6 and 13-17.
02_LINE	Illustrative Example of why just monitoring the loss is not enough. Reproduces Figure 1.
03_scalar_deep	Concrete example why just monitoring the loss is not enough. Reproduces Figure 11.
04_benchmark_memory	Benchmark the memory consumption of the histograms. Reproduces Figure 12.
06_preprocessing	Study how incorrectly scaled data is represented in Cockpit. Reproduces Figure 3 and 10.
07_learning_rate_selection	Grid search the learning rate and compare the alpha values vs. final performance. Reproduces Figure 5.
09_layerwise	Analyzing the layerwise histogram for two different architectures. Reproduces Figure 4.
10_showcase	Showcase of the full Cockpit for DeepOBS problems. Reproduces Figure 2 and 19.
11_histogram2d	Benchmark the performance of different implementations for computing the 2D Histogram. Reproduces Figure 18.
12_alpha_explanation	Illustrative plot explaining the Alpha Quantity. Reproduces Figure 8.

Main Text Experiments

Figure 1: Loss is not Enough - 02_LINE

Figure 2: Showcase - 10_showcase

Figure 3: Misscaled Data - 06_preprocessing

Figure 4: Layerwise Histogram - 09_layerwise

Figure 5: Tuning the Learning Rate with Alpha - 07_learning_rate_selection

Figure 6: Run Time Benchmark - 01_benchmark

Appendix Experiments

Figure 7 is not included in this list, as it is a code example. The code is an adapted version of the basic and advanced examples from the examples directory and the documentation.

Figure 8: Motivational Sketch for Alpha - 12_alpha_explanation

Figure 10: Misscaled Data ImageNet - 06_preprocessing

Figure 11: Regularization Detection - 03_scalar_deep

Figure 12: Memory Consumption Histogram - 04_benchmark_memory

Figure 13 - 17: Additional Run Time Benchmarks - 01_benchmark

Figure 18: Implementation Performance 2D Histogram - 11_histogram2d

Figure 19: Additional Showcases - 10_showcase