Beyond Local Sharpness: Communication-Efficient Global Sharpness-aware Minimization for Federated Learning

March 25, 2025 · View on GitHub

This repository contains the official implementation of

Caldarola, D., Cagnasso, P., Caputo, B., & Ciccone, M. "Beyond Local Sharpness: Communication-Efficient Global Sharpness-aware Minimization for Federated Learning". In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.

[ ArXiv ] [ Bibtex ]

Abstract

Federated learning (FL) enables collaborative model training with privacy preservation. Data heterogeneity across edge devices (clients) can cause models to converge to sharp minima, negatively impacting generalization and robustness. Recent approaches use client-side sharpness-aware minimization (SAM) to encourage flatter minima, but the discrepancy between local and global loss landscapes often undermines their effectiveness, as optimizing for local sharpness does not ensure global flatness.
This work introduces FedGloSS (Federated Global Server-side Sharpness), a novel FL approach that prioritizes the optimization of global sharpness on the server, using SAM. To reduce communication overhead, FedGloSS cleverly approximates sharpness using the previous global gradient, eliminating the need for additional client communication. Our extensive evaluations demonstrate that FedGloSS consistently reaches flatter minima and better performance compared to state-of-the-art FL methods across various federated vision benchmarks.

GIF showing the comparison between FedGloSS and FedAVG GIF showing the comparison between FedGloSS and FedSAM

GIFs comparing FedGloSS's loss landscape (net) against those of two well-known methods (solid), FedAVG on the left and FedSAM on the right. ResNet18 trained on CIFAR10 (α=0.05\alpha = 0.05).

Algorithm

FedGloSS proposes to apply SAM on the server side, while promoting consistency between local and global models by applying Alternating Direction Method of Multipliers (ADMM). The following algorithm summarizes FedGloSS, distinguishing between the use of SAM and SGD (achievable by setting ρl=0\rho_l = 0) on the client side.

Algorithm summarizing FedGloSS

Experiments

Setup

Environment

Set up the environment and install the required dependencies:

python -m venv ./fgvenv
source ./fgvenv/bin/activate
pip install -r requirements.txt -f https://download.pytorch.org/whl/torch_stable.html

Download the provided datasets

After activating the environment, you can download and prepare the datasets using the provided script:

cd data
chmod +x setup_datasets.sh
./setup_datasets.sh

Datasets

The provided datasets are CIFAR10 and CIFAR100. Both datasets are split equally among 100 clients, with varying degrees of heterogeneity controlled by Dirichlet's concentration parameter α\alpha. The available splits are:

  • CIFAR10: α∈{0,0.05,1,5,10,100}\alpha \in \{0, 0.05, 1, 5, 10, 100\}
  • CIFAR100: α∈{0,0.5,1000}\alpha \in \{0, 0.5, 1000\}

To add a new dataset, place the raw data in /data/{dataset_name}/raw/. Client-specific training splits should correspond to a Dirichlet concentration parameter and be stored in /data/{dataset_name}/train/ using the naming convention: federated_train_alpha_{alpha_value}.json. The test split should be placed in /data/{dataset_name}/test/test.json. Please add the dataset name to the list DATASETS in /codebase/utils/args.py. Ensure {dataset_name} matches the --dataset parameter when running an experiment.

Models

The provided model is a CNN inspired by the LeNet-5 architecture. It consists of:

  • Two convolutional layers (64 channels, $5 \times 5 kernels), each followed by a \2 \times 2$ max-pooling layer.
  • Two fully connected layers with 384 and 192 units.
  • A final output layer adjusted to the dataset's number of classes.

The implementation is available in /codebase/{dataset}/cnn.py.

To add a new model, place its implementation in /codebase/{dataset_name}/{model_name}.py. Ensure {model_name} matches the --model parameter when running an experiment.

CLI args

The table below reports the arguments that can be used to run experiments with this repository. New args can be added to /codebase/utils/args.py.

Argument NameDescriptionDefault Value
--dirRoot directory where the /results folder will be created.
--datasetDataset name used in the experiment
--dir-alphaDirichlet distribution concentration parameter
--modelModel name used in the experimentcnn
-TTotal number of experiment rounds
--eval-everyEvaluation interval (in rounds)
--C-tNumber of clients selected per round
--algorithmServer-side algorithmfedgloss
--client-algorithmClient-side algorithmfedgloss
--seedRandom seed for reproducibility0
--server-optServer optimizer (FedOpt)sgd
--server-lrLearning rate for the server optimizer
--server-momentumMomentum for the server optimizer0
--batch-sizeBatch size for local training64
-ENumber of local training epochs1
--lrLearning rate for local training
--weight-decayWeight decay for local training0
--momentumMomentum for local training0
--deviceDevice used for running the experimentcuda:0
--rho-lLocal SAM (Sharpness-Aware Minimization) parameter
--etaLearning rate adjustment factor for local SAM
--rhoServer-side SAM parameter
--betaHyperparameter for Lagrangian optimization
--T-sNumber of rounds for rho scheduling0
--rho0Initial rho value for scheduling0.001
--no-eigsDisables the computation of eigenvalues at the end of training
--plotsPlots the accuracy and loss trends in the results directory

Results

The execution generates the following output files:

  • {--dir}/results/{--model}_{--dataset}_alpha={--dir-alpha}_[YYYY][mm][dd]T[HH]:[MM]:[SS]/params.text: Summary of the parameters used for the run.
  • {--dir}/results/{--model}_{--dataset}_alpha={--dir-alpha}_[YYYY][mm][dd]T[HH]:[MM]:[SS]/trends.csv: Results in terms of test accuracy, test loss, global model norm, and pseudo-gradient norm.
  • {--dir}/results/{--model}_{--dataset}_alpha={--dir-alpha}_[YYYY][mm][dd]T[HH]:[MM]:[SS]/eigs.txt: First five eigenvalues and the ratio between the first and fifth eigenvalue.
  • {--dir}/results/{--model}_{--dataset}_alpha={--dir-alpha}_[YYYY][mm][dd]T[HH]:[MM]:[SS]/logs.txt: A file that reports all the lines printed on the terminal during the training.
  • {--dir}/results/{--model}_{--dataset}_alpha={--dir-alpha}_[YYYY][mm][dd]T[HH]:[MM]:[SS]/accuracy.pdf: A PDF representing the accuracy trend during the experiment.
  • {--dir}/results/{--model}_{--dataset}_alpha={--dir-alpha}_[YYYY][mm][dd]T[HH]:[MM]:[SS]/loss.pdf: A PDF representing the loss trend during the experiment.
  • {--dir}/results/{--model}_{--dataset}_alpha={--dir-alpha}_[YYYY][mm][dd]T[HH]:[MM]:[SS]/checkpoints/: Contains some checkpoints saved during training.

Examples

Below are examples of the paper's experiments that you can run from within the /codebase directory.

CIFAR10

python main.py --dataset cifar10 --dir-alpha 0 --where-loading init --model cnn -T 10000 --eval-every 100 --C-t 5 --algorithm fedgloss --seed 0 --server-opt sgd --server-lr 1 --server-momentum 0 --batch-size 64 -E 1 --lr 0.01 --weight-decay 0.0004 --momentum 0 --device cuda:0 --rho-l 0.15 --eta 0 --beta 10 --rho 0.15 --T-s 2000 --plots

CIFAR100

python main.py --dataset cifar100 --dir-alpha 0 --where-loading init --model cnn -T 20000 --eval-every 100 --C-t 5 --algorithm fedgloss --seed 0 --server-opt sgd --server-lr 1 --server-momentum 0 --batch-size 64 -E 1 --lr 0.01 --weight-decay 0.0004 --momentum 0 --device cuda:0 --rho-l 0.2 --eta 0 --beta 100 --rho 0.01 --T-s 15000 --plots

Paper Experiments

In the /paper_experiments directory, you can find scripts to run all the CNN experiments from the paper. Be sure to activate the appropriate Python environment before executing any of these scripts.

Bibtex citation

@inproceedings{caldarola2025beyond,
  title={Beyond Local Sharpness: Communication-Efficient Global Sharpness-aware Minimization for Federated Learning},
  author={Caldarola, Debora and Cagnasso, Pietro and Caputo, Barbara and Ciccone, Marco},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2025}
}

Acknowledgements

This repository builds upon the following projects: