Authors:
C. Nagaraju
1
;
Mrinmay Sen
2
and
C. Mohan
3
Affiliations:
1
Deparment of Computer Science, IIT Hyderabad,, India
;
2
Department of Artificial Intelligence, IIT Hyderabad, India
;
3
Deparment of Computer Science, IIT Hyderabad, India
Keyword(s):
Data Heterogeneity in Federated Learning, Global Data Distribution with Gaussian Mixture Model.
Abstract:
Federated learning, a different direction of distributed optimization, is very much important when there are re- strictions of data sharing due to privacy and communication overhead. In federated learning, instead of sharing raw data, information from different sources are gathered in terms of model parameters or gradients of local loss functions and these information is fused in such way that we can find the optima of average of all the local loss functions (global objective). Exiting analyses on federated learning show that federated optimization gets slow convergence when data distribution across all the clients or sources are not homogeneous. Heterogeneous data distribution in federated learning causes objective inconsistency which means global model converges to a another stationary point which is not same as the optima of the global objective which results in poor per- formance of the global model. In this paper, we propose a federated Learning(FL) algorithm in heterogeneous da
ta distribution. To handle data heterogeneity during collaborative training, we generate data in local clients with the help of a globally trained Gaussian Mixture Models(GMM). We update each local model with the help of both original and generated local data and then perform the similar operations of the most popular algorithm called FedAvg. We compare our proposed method with exiting FedAvg and FedProx algorithms with CIFAR10 and FashionMNIST Non-IID data. Our experimental results show that our proposed method performs better than the exiting FedAvg and FedProx algorithm in terms of training loss, test loss and test accuracy in heterogeneous system.
(More)