Minimal Random Coding for Stochastic Federated Learning
Description
Communication efficiency is a major and well-studied premise in federated learning (FL). Various compression schemes have been proposed to alleviate prohibitively high communication costs. In stochastic formulations of FL, principled approaches to compression can substantially improve existing measures. Minimal random coding (MRC), introduced in [1], leverages side information, common randomness, and tools from importance-sampling. The authors of [2] proposed to use MRC for stochastic FL and show substantial improvements in various domains.
The task of the student is to understand the principles of the proposed compression schemes and their application to stochastic FL for different learning narratives. The student should analyze where the substantial communication improvements originate from, and what open problems remain.
[1] Marton Havasi, Robert Peharz, and Jose Miguel ´ Hernandez-Lobato. Minimal random code learning: ´ Getting bits back from compressed model parameters. In International Conference on Learning Representations, 2019.
[2] Berivan Isik, Francesco Pase, Deniz Gunduz, Sanmi Koyejo, Tsachy Weissman, and Michele Zorzi. Adaptive compression in federated learning via side information. In International Conference on Artificial Intelligence and Statistics, pp. 487–495, 2024.
Supervisor:
Private and Secure Federated Learning
Description
In federated learning, a machine learning model shall be trained on private user data with the help of a central server, the so-called federator. This setting differs from other machine learning settings in that the user data shall not be shared with the federator for privacy reasons and/or to decrease the communication load of the system.
Even though only intermediate results are shared, extra care is necessary to guarantee data privacy. An additional challenge arises if the system includes malicious users that breach protocol and send corrupt computation results.
The goal of this work is to design, implement and analyze coding- and information-theoretic solutions for privacy and security in federated learning.
Prerequisites
- Information Theory
- Coding Theory (e.g., Channel Coding)
- Machine Learning (Theory and Practice)