Ablation Study

The ablation study results can be extracted from file ablation_study_results.zip hosted alongside the artifact. Download the file and extract the results. This command creates a new directory so is safe run without a new directory:

unzip ablation_study_results.zip

Prepare the results for plotting (this step can be skipped and ablation_results.json be used as the process is deterministic):

python prepare_results.py --root ablation_study_results

Expected output:

Collecting results for parameter: seq_len; Using config: results_01_04
Categorizing 128 files
Collecting results for parameter: seq_len; Using config: results_02_04
...
Aggregated results saved to ablation_results.json

The preparation takes 20-40 minutes.

Figure 3

Finally, the results can be plotted. To generate Figure 3 run:

python plot_ablation.py --input ablation_results.json --merge --export --format svg

This will create files merged_registers.svg and merged_sequence_length.svg. We provide the file ablation_results.json so that the generation is not needed.