Skip to content
CaMLSysUniversity of Cambridge
Home People Publications Projects Contact

← All publications

FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs

  • Lorenzo SaniUniversity of CambridgeFlower Labs
  • Zeyu CaoUniversity of Cambridge
  • Meghdad KurmanjiUniversity of Cambridge
  • Alex IacobUniversity of CambridgeFlower Labs
  • Andrej JovanovicUniversity of CambridgeFlower Labs
  • Yan GaoUniversity of CambridgeFlower Labs
  • Wanru ZhaoUniversity of Cambridge
  • Nicholas D. LaneUniversity of CambridgeFlower Labs

Preprint arXiv, 2026

Abstract

Pre-training Large Language Models (LLMs) typically demands large-scale infrastructure with tightly coupled hardware accelerators. Mixture-of-Experts (MoEs) architectures partially decouple model capacity from per-token compute. This efficiency alone does not make MoE training feasible over ordinary Internet links or loosely connected commodity hardware since active expert routing still assumes high-speed datacenter fabrics. Low-communication methods such as DiLoCo and Photon reduce synchronization frequency across distributed sites, mitigating bandwidth constraints, yet still require full model replicas at every site. This creates a mismatch: modern MoEs have sparse data paths, but their distributed training infrastructure remains communication-dense and memory-inefficient, limiting attempts to pool geographically distributed compute. In this work, we introduce FoMoE, a system that breaks the full-replica paradigm by partitioning expert layers across workers and skipping non-resident experts during local training. We demonstrate that FoMoE: (I) reduces communication costs by up to 1.42x over efficient baselines and 45.44x over Distributed Data Parallelism (DDP) via partial expert replication in controlled regimes; (II) achieves empirical throughput speedups of up to 1.4x through the skip-token mechanism; and (III) shows stable routing in the trained regimes and projects the communication/memory benefits to 100B-scale configurations through system modeling.

Materials

  • PDF
BibTeX
@misc{sani2026fomoe,
  title = {{FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs}},
  author = {Sani, Lorenzo and Cao, Zeyu and Kurmanji, Meghdad and Iacob, Alex and Jovanovic, Andrej and Gao, Yan and Zhao, Wanru and Lane, Nicholas D.},
  year = {2026},
  url = {https://arxiv.org/abs/2606.19025},
  eprint = {2606.19025},
  archiveprefix = {arXiv}
}
Cambridge Machine Learning Systems Group
Department of Computer Science and Technology
University of Cambridge
ContactGitHub