Skip to content
CaMLSysUniversity of Cambridge
Home People Publications Projects Contact

← All publications

The Future of Large Language Model Pre-training is Federated

  • Lorenzo SaniUniversity of CambridgeFlower Labs
  • Alex IacobUniversity of CambridgeFlower Labs
  • Zeyu CaoUniversity of Cambridge
  • Bill MarinoUniversity of Cambridge
  • Yan GaoUniversity of CambridgeFlower Labs
  • Tomas PaulikUniversity of Cambridge
  • Wanru ZhaoUniversity of Cambridge
  • William F. ShenUniversity of Cambridge
  • Preslav AleksandrovUniversity of Cambridge
  • Xinchi QiuUniversity of Cambridge
  • Nicholas D. LaneUniversity of CambridgeFlower Labs

International Workshop on Federated Foundation Models (FL@FM) at NeurIPS, 2024

Abstract

Generative pre-trained large language models (LLMs) have demonstrated impressive performance over a wide range of tasks, thanks to the unprecedented amount of data they have been trained on. As established scaling laws indicate, LLMs’ future performance improvement depends on the amount of computing and data sources they can leverage for pre-training. Federated learning (FL) has the potential to unleash the majority of the planet’s data and computational resources, which are underutilized by the data-center-focused training methodology of current LLM practice. Our work presents a robust, flexible, reproducible FL approach that enables large-scale collaboration across institutions to train LLMs. We propose a scalable deployment system called Photon to enable the investigation and development of this new training paradigm for LLM pre-training. We show that Photon can be used by organizations interested in collaborating with their private data sources and computational resources for pre-training LLMs with billions of parameters. This paradigm would mobilize more computational and data resources while matching or potentially exceeding centralized performance. We further show the effectiveness of the federated training scales with model size and present our approach for training billion-scale federated LLMs using limited resources. Thus far, we have used Photon to train LLM models to the size of 7B parameters and anticipate larger models being completed in the near future. Finally, we show that LLM training is highly resilient to the classical challenges of federated statistical and hardware heterogeneity. Furthermore, we show that convergence is robust to partial participation, opening the avenue for compute-efficient collaborative training. Photon will help data-rich actors to become the protagonists of LLMs pre-training instead of leaving the stage to compute-rich actors alone.

Materials

  • PDF
BibTeX
@inproceedings{sani2024future,
  title = {{The Future of Large Language Model Pre-training is Federated}},
  author = {Sani, Lorenzo and Iacob, Alex and Cao, Zeyu and Marino, Bill and Gao, Yan and Paulik, Tomas and Zhao, Wanru and Shen, William F. and Aleksandrov, Preslav and Qiu, Xinchi and Lane, Nicholas D.},
  year = {2024},
  booktitle = {International Workshop on Federated Foundation Models (FL@FM) at NeurIPS}
}
Cambridge Machine Learning Systems Group
Department of Computer Science and Technology
University of Cambridge
ContactGitHub