FastAT Benchmark

May 6, 2026 · View on GitHub

🌐 Benchmark Webpage

The benchmark results can be viewed at: https://fzjcdt.github.io/FastAT_Benchmark/benchmark.html

The FastAT Benchmark provides a rigorous and fair evaluation framework for fast adversarial training methods. Unlike public leaderboards that allow diverse combinations of model architectures, data sources, and computational budgets, our benchmark establishes conditions where all methods compete on equal footing.

This platform implements over twenty representative FastAT methods with a unified codebase, ensuring fair and reproducible comparison across different algorithmic innovations. The benchmark systematically removes advantages from massive computational resources and unlimited external data, providing the research community with a transparent baseline for evaluating fast adversarial training techniques.

✨ Key Features

🎯 Unified Architecture: All methods are evaluated on identical network structures, eliminating performance differences arising from architectural advantages rather than training procedures.
⚙️ Standardized Settings: Consistent training schedules, optimizers, learning rate policies, and data augmentation strategies prevent the experimental setup from favoring any particular method.
🚫 No External Data: Strict prohibition of using additional or synthetic data beyond the original benchmark training set ensures observed gains stem solely from the learning algorithm.
📊 Dual-Metric Framework: Evaluates both robustness performance (accuracy against strong attacks) and computational cost (GPU hours and memory footprint).
🔬 Comprehensive Evaluation: Includes diverse attack methods: PGD with varying iterations, AutoAttack, and CR Attack for thorough robustness assessment.
🔄 Unified Implementation: Re-implemented over a dozen FastAT methods with a common interface for data loading, model initialization, training loops, and evaluation protocols.

📋 Design Principles and Controlled Constraints

Strict experimental control forms the foundation of the FastAT Benchmark. Several key constraints support this central design principle:

Unified Architecture: All methods undergo evaluation on identical network structures.
Standardized Training Settings: Consistent training schedules, optimizers, learning rate policies, and data augmentation strategies.
No External Data: Strictly prohibits using any additional or synthetic data beyond the original benchmark training set.

📊 Dual-Metric Evaluation Framework

FastAT methods aim to achieve robustness through efficient means, making it essential to evaluate both effectiveness and computational cost:

Robustness Performance: Measured as final robust accuracy against strong adversarial attacks.
Training Cost: Encompasses critical resource requirements such as total GPU hours and peak memory footprint.

🔄 Benchmark Training Flow

Load Configuration (common.yaml, method.yaml)
Initialization Phase: Data Loaders, Model Init, Optimizer
Select FastAT Method
Training Phase: Training Loop, Validation
Evaluation Phase: Final Evaluation with PGD Attack, AutoAttack, CR Attack
Log Results & Metrics

📈 Evaluation Setup

Datasets and Models

CIFAR-10: ResNet-18 architecture
CIFAR-100: ResNet-18 architecture
Tiny-ImageNet: PreActResNet-18 architecture

Each experiment runs three times with different random seeds, and the mean of these three runs constitutes the final reported result.

Training Configuration

Optimizer: SGD with initial learning rate of 0.1, momentum of 0.9, and weight decay of 5×10⁻⁴
Training Duration: 100 epochs with OneCycleLR scheduler
Batch Size: 128
Data Augmentation: 4-pixel padding followed by random cropping and horizontal flipping
Additional Techniques: Weight averaging (τ = 0.9995) and label smoothing
Validation Sets: 1,000 images for CIFAR-10, 1,000 images for CIFAR-100, 2,000 images for Tiny-ImageNet

🛡️ Supported Methods

🚀 Getting Started

If you want to reproduce these FastAT methods, you can use commands similar to the following:

python main.py --dataset cifar10 --seeds 0 1 2 --methods fgsm_rs

🤝 Contributing

Contributions to the FastAT Benchmark are welcome! Please feel free to submit pull requests or open issues to help improve the framework.

📚 References

For more details, please refer to the paper:

@article{pan2026fastat,
  title={FastAT Benchmark: A Comprehensive Framework for Fair Evaluation of Fast Adversarial Training Methods},
  author={Pan, Chao and Yao, Xin},
  journal={arXiv preprint arXiv:2604.22853},
  year={2026}
}