A First Look into the Carbon Footprint of Federated Learning
Abstract
Despite impressive results, deep learning-based technologies also raise severe privacy and environmental concerns induced by the training procedure often conducted in data centers. In response, alternatives to centralized training such as Federated Learning (FL) have emerged. FL is now starting to be deployed at a global scale by companies that must adhere to new legal demands and policies originating from governments and social groups advocating for privacy protection. However, the potential environmental impact related to FL remains unclear and unexplored. This article offers the first-ever systematic study of the carbon footprint of FL. We propose a rigorous model to quantify the carbon footprint, hence facilitating the investigation of the relationship between FL design and carbon emissions. We also compare the carbon footprint of FL to traditional centralized learning. Our findings show that, depending on the configuration, FL can emit up to two orders of magnitude more carbon than centralized training. However, in certain settings, it can be comparable to centralized learning due to the reduced energy consumption of embedded devices. Finally, we highlight and connect the results to the future challenges and trends in FL to reduce its environmental impact, including algorithms efficiency, hardware capabilities, and stronger industry transparency.
Carbon Dioxide in Federated Learning
For the better part of a decade, Deep Learning has been defining the state of the art in various machine learning tasks. Ranging from Computer Vision [1] to Natural Language Processing [2], models have achieved remarkable results at the cost of doubling computation resources every 3.4 months [3]. As hardware efficiency struggles to keep up with the exponential growth in architecture complexity, a similar trend in energy consumption is created as more hardware and cooling systems are required to train modern models. In fact, recent studies [4][5] have shown that training large models in conventional data centres, a sector that already amounts for 0.3% of the world’s carbon emission [6], can cause significant increase in CO2eq production.
How Green is Federated Learning?
Fortunately, not all hope is lost and a more carbon-friendly way to train neural networks exists. In Federated Learning (FL), training is performed not inside large data centres, but distributed over thousands of mobile devices, such as smartphones, where data is usually collected by the end-users themselves.
An example of an application currently using FL is the next-word prediction in mobile phones [7]. In this application, each smartphone (client) trains a local network (model) to predict which word the user will type next based on their previous text messages. Trained local models are then sent to a server to perform a much simpler task called aggregation in which a final model will be generated and sent back to all users.
Besides the privacy-related gains of not having to send user-data to a centralised server, in our recent work [8], [9] we show that FL can also have a positive impact in reducing carbon emissions derived from Deep Learning. Although mobile devices are much less powerful than server GPUs, FL benefits from not needing any cooling and from having wide pools of devices for training.
The Federated Learning Carbon Calculator
So how much carbon does Federated Learning produce? The following calculator allows you to estimate how much CO2eq is produced by a given pool of devices.
To help you define your training scenario here are a few definitions and hyperparameters used in FL, followed by realistic input values.
- Devices: Selects the hardware profile being used.
- Country: Defines the energy mix and Internet Upload/Download speeds of a given pool of devices. The former parameter defines the electricity/carbon conversion rate while the latter parameters help estimate emissions due to client-server communication.
- Dataset: Defines the balanced, non-IID dataset being used.
- Number of rounds: Total number of times the server will aggregate results from clients (10-100).
- Number of local epochs: Number of times each client will train on their on their local data before sending their model for aggregation (1-5).
- Number of active devices: Number of devices participating in each round. This should be a small fraction of the total number of devices available (10-1000).
Device pool:
(#Devices, Type, Country, Download(Mbps), Upload(Mbps), #Epochs, #Rounds)
Wow! Your FL setup produces: g CO2eq.
To put this into perspective...
- Sending an email from laptop to laptop generates 0.3g CO2eq[1].
- Making a Zoom call from a regular laptop generates 10g CO2eq[1].
- Training a neural network in a data centre in China to reach 70% of accuracy on the Speech Commands dataset may produce 17.5g CO2eq[2].
- Training a neural network in a data centre in the US to reach 50% of accuracy on the ImageNet dataset may produce 889g CO2eq[2].
[1] "How Bad Are Bananas? The carbon footprint of everything" by Mike Berners-Lee.
[2] "A first look into the Carbon footprint of Federated Learning" by Qiu et al..
References
BibTeX
@online{openai,
title = {AI and Compute},
author = {Amodei, Dario and Hernandez, Danny and Sastry, Girish and Clark, Jack and Brockman, Greg and Sutskever, Ilya},
year = {2019},
addendum = {(accessed: 25.02.2021)},
url = {https://openai.com/blog/ai-and-compute/}
}
BibTeX
@article{strubell2019energy,
title = {Energy and policy considerations for deep learning in NLP},
author = {Strubell, Emma and Ganesh, Ananya and McCallum, Andrew},
journal = {arXiv preprint arXiv:1906.02243},
year = {2019}
}
BibTeX
@misc{carbon_footprint,
author = {Qiu, Xinchi and Parcollet, Titouan and Fernandez-Marques, Javier and de Gusmao, Pedro Porto Buarque and Beutel, Daniel J. and Topal, Taner and Mathur, Akhil and Lane, Nicholas D.},
title = {A first look into the carbon footprint of federated learning},
year = {2021},
eprint = {arXiv:2102.07627}
}
BibTeX
@article{brown2020language,
title = {Language models are few-shot learners},
author = {Brown, Tom B and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and others},
journal = {arXiv preprint arXiv:2005.14165},
year = {2020}
}
BibTeX
@article{krizhevsky2012imagenet,
title = {Imagenet classification with deep convolutional neural networks},
author = {Krizhevsky, Alex and Sutskever, Ilya and Hinton, Geoffrey E},
journal = {Advances in neural information processing systems},
volume = {25},
pages = {1097--1105},
year = {2012}
}
BibTeX
@article{lacoste1910quantifying,
title = {Quantifying the Carbon Emissions of Machine Learning. arXiv 2019},
author = {Lacoste, A and Luccioni, A and Schmidt, V and Dandres, T},
journal = {arXiv preprint arXiv:1910.09700},
year = {2019}
}
BibTeX
@article{jones2018stop,
title = {How to stop data centres from gobbling up the world's electricity.},
author = {Jones, Nicola},
journal = {Nature},
volume = {561},
number = {7722},
pages = {163--167},
year = {2018},
publisher = {Nature Publishing Group}
}
BibTeX
@misc{google_keyboard,
title = {Federated Learning for Mobile Keyboard Prediction},
author = {Hard, Andrew and Kiddon, Chloé M and Ramage, Daniel and Beaufays, Francoise and Eichner, Hubert and Rao, Kanishka and Mathews, Rajiv and Augenstein, Sean},
year = {2018},
url = {https://arxiv.org/abs/1811.03604}
}
BibTeX
@inproceedings{qiu2020can,
title = {Can Federated Learning Save the Planet?},
author = {Qiu, Xinchi and Parcollet, Titouan and Beutel, Daniel and Topal, Taner and Mathur, Akhil and Lane, Nicholas},
booktitle = {NeurIPS-Tackling Climate Change with Machine Learning},
year = {2020}
}