Skip to content
CaMLSysUniversity of Cambridge
Home People Publications Projects Contact

← All publications

Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment

  • Yan GaoFlower LabsUniversity of Cambridge
  • Mohammad NaseriFlower Labs
  • Javier Fernandez-MarquesFlower LabsUniversity of Cambridge
  • Dimitris StripelisFlower Labs
  • Lorenzo SaniFlower LabsUniversity of Cambridge
  • Davide EynardMozilla.ai
  • Fan ZhangUniversity of Cambridge
  • Hong JiaUniversity of Auckland
  • Ting DangUniversity of Melbourne
  • D. B. EmersonVector Institute
  • Fatemeh TavakoliVector Institute
  • Ole WergerFraunhofer IMS
  • Lars WulfertFraunhofer IMS
  • Petros DemetrakopoulosNetCompany
  • Sofia TsekeridouNetCompany
  • InSeo SongGachon University
  • KangYoon LeeGachon University
  • Honghao LiOwkin
  • Lingjuan LyuSony AI
  • John P DickersonMozilla.ai
  • Daniel Janes BeutelFlower Labs
  • Nicholas D. LaneFlower LabsUniversity of Cambridge

Preprint arXiv, 2026

Abstract

Federated learning (FL) has emerged as a key approach for training models across decentralized data, yet benchmarking in FL remains difficult to reproduce, compare, and extend. Existing evaluations are often tied to custom infrastructure, released as incomplete research code, and conducted primarily in simulation, which limits portability and practical relevance. We present Flower Hub, a platform for publishing, discovering, and executing decentralized and federated applications. We show how it enables reproducible benchmarking by packaging benchmarks as executable, versioned applications with standardized metadata, pinned dependencies, and explicit evaluation workflows. We instantiate this approach with a multi-domain benchmark suite spanning cross-silo and cross-device settings, and including tasks in medical imaging, financial tabular learning, legal instruction tuning, phishing URL detection, and audio tagging. We further demonstrate that the same benchmarking application can run across both simulation and deployment runtimes without changing the application code, enabling unified evaluation across varying learning environments. Beyond model quality, our benchmark design supports system-aware reporting, including runtime and communication metrics. This work advances benchmarking in FL settings from ad hoc code artifacts towards portable, executable, and reusable benchmark applications.

Materials

  • PDF
BibTeX
@misc{gao2026flowerhub,
  title = {{Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment}},
  author = {Gao, Yan and Naseri, Mohammad and Fernandez-Marques, Javier and Stripelis, Dimitris and Sani, Lorenzo and Eynard, Davide and Zhang, Fan and Jia, Hong and Dang, Ting and Emerson, D. B. and Tavakoli, Fatemeh and Werger, Ole and Wulfert, Lars and Demetrakopoulos, Petros and Tsekeridou, Sofia and Song, InSeo and Lee, KangYoon and Li, Honghao and Lyu, Lingjuan and Dickerson, John P and Beutel, Daniel Janes and Lane, Nicholas D.},
  year = {2026},
  url = {https://arxiv.org/abs/2608.25114},
  eprint = {2608.25114},
  archiveprefix = {arXiv}
}
Cambridge Machine Learning Systems Group
Department of Computer Science and Technology
University of Cambridge
ContactGitHub