Skip to content
CaMLSysUniversity of Cambridge
Home People Publications Projects Contact

← All publications

Worldwide Federated Training of Language Models

  • Alex IacobUniversity of CambridgeFlower Labs
  • Lorenzo SaniUniversity of CambridgeFlower Labs
  • Bill MarinoUniversity of Cambridge
  • Preslav AleksandrovUniversity of Cambridge
  • William F. ShenUniversity of Cambridge
  • Nicholas Donald LaneUniversity of CambridgeFlower Labs

International Workshop on Federated Foundation Models (FL@FM) at NeurIPS, 2024

Abstract

The reliance of language model training on massive amounts of computation and vast datasets scraped from potentially low-quality, copyrighted, or sensitive data has come into question practically, legally, and ethically. Federated learning provides a plausible alternative by enabling previously untapped data to be voluntarily gathered from collaborating organizations. However, when scaled globally, federated learning requires collaboration across heterogeneous legal, security, and privacy regimes while accounting for the inherent locality of language data; this further exacerbates the established challenge of federated statistical heterogeneity. We propose a Worldwide Federated Language Model Training (WorldLM) system based on federations of federations, where each federation has the autonomy to account for factors such as its industry, operating jurisdiction, or competitive environment. WorldLM enables such autonomy in the presence of statistical heterogeneity via partial model localization by allowing sub-federations to attentively aggregate key layers from their constituents. Furthermore, it can adaptively share information across federations via residual layer embeddings. Evaluations of language modeling on naturally heterogeneous datasets show that WorldLM outperforms standard federations by up to 1.91×, approaches the personalized performance of fully local models, and maintains these advantages under privacy-enhancing techniques.

Materials

  • PDF
BibTeX
@inproceedings{iacob2024worldwidefederatedtraininglanguage,
  title = {{Worldwide Federated Training of Language Models}},
  author = {Iacob, Alex and Sani, Lorenzo and Marino, Bill and Aleksandrov, Preslav and Shen, William F. and Lane, Nicholas Donald},
  year = {2024},
  booktitle = {International Workshop on Federated Foundation Models (FL@FM) at NeurIPS}
}
Cambridge Machine Learning Systems Group
Department of Computer Science and Technology
University of Cambridge
ContactGitHub