FedFa: A Fully Asynchronous Training Paradigm for Federated Learning
Authors:
Haotian Xu,
Zhaorui Zhang,
Sheng Di,
Benben Liu,
Khalid Ayed Alharthi,
Jiannong Cao
Abstract:
Federated learning has been identified as an efficient decentralized training paradigm for scaling the machine learning model training on a large number of devices while guaranteeing the data privacy of the trainers. FedAvg has become a foundational parameter update strategy for federated learning, which has been promising to eliminate the effect of the heterogeneous data across clients and guaran…
▽ More
Federated learning has been identified as an efficient decentralized training paradigm for scaling the machine learning model training on a large number of devices while guaranteeing the data privacy of the trainers. FedAvg has become a foundational parameter update strategy for federated learning, which has been promising to eliminate the effect of the heterogeneous data across clients and guarantee convergence. However, the synchronization parameter update barriers for each communication round during the training significant time on waiting, slowing down the training procedure. Therefore, recent state-of-the-art solutions propose using semi-asynchronous approaches to mitigate the waiting time cost with guaranteed convergence. Nevertheless, emerging semi-asynchronous approaches are unable to eliminate the waiting time completely.
We propose a full asynchronous training paradigm, called FedFa, which can guarantee model convergence and eliminate the waiting time completely for federated learning by using a few buffered results on the server for parameter updating. Further, we provide theoretical proof of the convergence rate for our proposed FedFa. Extensive experimental results indicate our approach effectively improves the training performance of federated learning by up to 6x and 4x speedup compared to the state-of-the-art synchronous and semi-asynchronous strategies while retaining high accuracy in both IID and Non-IID scenarios.
△ Less
Submitted 20 April, 2024; v1 submitted 16 April, 2024;
originally announced April 2024.
A Survey on Error-Bounded Lossy Compression for Scientific Datasets
Authors:
Sheng Di,
Jinyang Liu,
Kai Zhao,
Xin Liang,
Robert Underwood,
Zhaorui Zhang,
Milan Shah,
Yafan Huang,
Jiajun Huang,
Xiaodong Yu,
Congrong Ren,
Hanqi Guo,
Grant Wilkins,
Dingwen Tao,
Jiannan Tian,
Sian Jin,
Zizhe Jian,
Daoce Wang,
MD Hasanur Rahman,
Boyuan Zhang,
Jon C. Calhoun,
Guanpeng Li,
Kazutomo Yoshii,
Khalid Ayed Alharthi,
Franck Cappello
Abstract:
Error-bounded lossy compression has been effective in significantly reducing the data storage/transfer burden while preserving the reconstructed data fidelity very well. Many error-bounded lossy compressors have been developed for a wide range of parallel and distributed use cases for years. These lossy compressors are designed with distinct compression models and design principles, such that each…
▽ More
Error-bounded lossy compression has been effective in significantly reducing the data storage/transfer burden while preserving the reconstructed data fidelity very well. Many error-bounded lossy compressors have been developed for a wide range of parallel and distributed use cases for years. These lossy compressors are designed with distinct compression models and design principles, such that each of them features particular pros and cons. In this paper we provide a comprehensive survey of emerging error-bounded lossy compression techniques for different use cases each involving big data to process. The key contribution is fourfold. (1) We summarize an insightful taxonomy of lossy compression into 6 classic compression models. (2) We provide a comprehensive survey of 10+ commonly used compression components/modules used in error-bounded lossy compressors. (3) We provide a comprehensive survey of 10+ state-of-the-art error-bounded lossy compressors as well as how they combine the various compression modules in their designs. (4) We provide a comprehensive survey of the lossy compression for 10+ modern scientific applications and use-cases. We believe this survey is useful to multiple communities including scientific applications, high-performance computing, lossy compression, and big data.
△ Less
Submitted 3 April, 2024;
originally announced April 2024.
