Skip to content
CaMLSysUniversity of Cambridge
Home People Publications Projects Contact

← All publications

SparsyFed: Sparse Adaptive Federated Training

  • Adriano GuastellaUniversity of Bologna
  • Lorenzo SaniUniversity of CambridgeFlower Labs
  • Alex IacobUniversity of CambridgeFlower Labs
  • Alessio MoraUniversity of Bologna
  • Paolo BellavistaUniversity of Bologna
  • Nicholas D. LaneUniversity of CambridgeFlower Labs

International Conference on Learning Representations (ICLR), 2025

Abstract

Sparse training is often adopted in cross-device federated learning (FL) environments where constrained devices collaboratively train a machine learning model on private data by exchanging pseudo-gradients across heterogeneous networks. Although sparse training methods can reduce communication overhead and computational burden in FL, they are often not used in practice for the following key reasons: (1) data heterogeneity makes it harder for clients to reach consensus on sparse models compared to dense ones, requiring longer training; (2) methods for obtaining sparse masks lack adaptivity to accommodate very heterogeneous data distributions, crucial in cross-device FL; and (3) additional hyperparameters are required, which are notably challenging to tune in FL. This paper presents SparsyFed, a practical federated sparse training method that critically addresses the problems above. Previous works have only solved one or two of these challenges at the expense of introducing new trade-offs, such as clients’ consensus on masks versus sparsity pattern adaptivity. We show that SparsyFed simultaneously (1) can produce 95% sparse models, with negligible degradation in accuracy, while only needing a single hyperparameter, (2) achieves a per-round weight regrowth 200 times smaller than previous methods, and (3) allows the sparse masks to adapt to highly heterogeneous data distributions and outperform all baselines under such conditions.

Materials

  • PDF
BibTeX
@inproceedings{guastella2025sparsyfed,
  title = {{SparsyFed: Sparse Adaptive Federated Training}},
  author = {Guastella, Adriano and Sani, Lorenzo and Iacob, Alex and Mora, Alessio and Bellavista, Paolo and Lane, Nicholas D.},
  year = {2025},
  booktitle = {International Conference on Learning Representations (ICLR)}
}
Cambridge Machine Learning Systems Group
Department of Computer Science and Technology
University of Cambridge
ContactGitHub