Skip to content
CaMLSysUniversity of Cambridge
Home People Publications Projects Contact

← All publications

Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languages

  • Wanru ZhaoUniversity of Cambridge
  • Yihong ChenUniversity College London
  • Royson LeeUniversity of CambridgeSamsung AI Center
  • Xinchi QiuUniversity of Cambridge
  • Yan GaoUniversity of CambridgeFlower Labs
  • Hongxiang FanUniversity of CambridgeSamsung AI Center
  • Nicholas D. LaneUniversity of CambridgeFlower Labs

International Conference on Learning Representations (ICLR), 2024

Abstract

Pre-trained large language models (LLMs) have become a cornerstone of modern natural language processing, with their capabilities extending across a wide range of applications and languages. However, the fine-tuning of multilingual LLMs, especially for low-resource languages, faces significant challenges arising from data-sharing restrictions (the physical border) and inherent linguistic differences (the linguistic border). These barriers hinder users of various languages, particularly those in low-resource regions, from fully benefiting from the advantages of LLMs. To address these challenges, we propose the Federated Prompt Tuning Paradigm for multilingual scenarios, which utilizes parameter-efficient fine-tuning while adhering to data sharing restrictions. We design a comprehensive set of experiments and analyze them using a novel notion of language distance to highlight the strengths of our paradigm: Even under computational constraints, our method not only improves data efficiency but also facilitates mutual enhancements across languages, particularly benefiting low-resource ones. Compared to traditional local cross-lingual transfer tuning methods, our approach achieves 6.9% higher accuracy with improved data efficiency, and demonstrates greater stability and generalization. These findings underscore the potential of our approach to promote social equality and champion linguistic diversity, ensuring that no language is left behind.

Materials

  • PDF
BibTeX
@inproceedings{zhao2024breaking,
  title = {{Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languages}},
  author = {Zhao, Wanru and Chen, Yihong and Lee, Royson and Qiu, Xinchi and Gao, Yan and Fan, Hongxiang and Lane, Nicholas D.},
  year = {2024},
  booktitle = {International Conference on Learning Representations (ICLR)},
  url = {https://openreview.net/forum?id=HyRwexERAo}
}
Cambridge Machine Learning Systems Group
Department of Computer Science and Technology
University of Cambridge
ContactGitHub