Privacy-preserving distributed machine learning for artificial intelligence of things
Doctoral thesis
Permanent lenke
https://hdl.handle.net/11250/3042068Utgivelsesdato
2023Metadata
Vis full innførselSamlinger
Sammendrag
This thesis proposes machine learning algorithms that can be fully distributed over ad-hoc networks of machines/agents. Developing distributed algorithms for artificial intelligence is necessary since running machine-learning-based data analytics on a single central hub may be unfeasible due to computing/communication costs. In the context of distributed learning, privacy violation risks due to curious members of the network or eavesdroppers make the development of privacy-preserving distributed algorithms imperative.
The main contributions of the thesis are around developing distributed machine learning algorithms for artificial intelligence of things including distributed algorithms with intrinsic privacy-preserving properties. In particular, the contributions can be grouped in the following categories:
• distributed learning over networks with horizontal/row partitioning of data
• intrinsically privacy-preserving distributed learning with zeroth-order optimization
• distributed learning over networks with vertical/feature partitioning of data.
In the context of distributed learning with horizontal partitioning of data, we propose a new distributed algorithm to solve the total least-squares (TLS) problem and a privacy-preserving distributed algorithm to minimize a regularized empirical risk function when the first-order information is not available.
We show that the latter algorithm has intrinsic privacy-preserving properties. Most existing privacy-preserving distributed optimization/estimation algorithms exploit some perturbation mechanism to preserve privacy, which comes at the cost of reduced accuracy. Contrarily, we exploit the inherent randomness due to the use of a zeroth-order method and show that this stochasticity is sufficient to ensure differential privacy. Moreover, we demonstrate that the proposed algorithm outperforms the existing differentially-private ones in terms of accuracy while yielding similar privacy guarantees.
In the context of distributed learning with feature partitioning of data, we develop a new distributed algorithm to solve the ridge regression problem. Subsequently, we develop a new algorithm that is designed for an ‘2-norm-square cost function with non-smooth regularizers. Finally, we develop a new consensus-based distributed algorithm for solving learning problems when the data is distributed among agents in feature partitions and computing the conjugate of the possibly non-smooth cost or regularizer functions is challenging or unfeasible. The proposed algorithm is designed for optimizing generic non-smooth objective functions over arbitrary graphs without using or computing any conjugate function. All the above-mentioned algorithms are fully-distributed and based on the alternating direction method of multipliers (ADMM) that is suitable for distributed optimization thanks to its scalability and robustness properties. We prove theoretically that the proposed algorithms converge. We also confirm their network-wide convergence via simulations.
Består av
Paper 1: Gratton, Cristiano; Dasanadoddi Venkategowda, Naveen Kumar; Arablouei, Reza; Werner, Stefan. Consensus-based Distributed Total Least-squares Estimation Using Parametric Semidefinite Programming. I: 2019 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). IEEE 2019 ISBN 978-1-4799-8131-1. s. 5227-5231. © 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.Paper 2: Gratton, Cristiano; Dasanadoddi Venkategowda, Naveen Kumar; Arablouei, Reza; Werner, Stefan. Distributed Learning with Non-Smooth Objective Functions. I: 28th European Signal Processing Conference (EUSIPCO 2020). : European Signal Processing Conference 2020 ISBN 978-9-0827-9705-3. s. 2180-2184. © 2020 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Paper 3: Gratton, Cristiano; Kumar Dasanadoddi Venkategowda, Naveen; Arablouei, Reza; Werner, Stefan. Privacy-Preserved Distributed Learning With Zeroth-Order Optimization. IEEE Transactions on Information Forensics and Security 2022 ;Volum 17. s. 265-279 © 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Paper 4: Gratton, Cristiano; Dasanadoddi Venkategowda, Naveen Kumar; Arablouei, Reza; Werner, Stefan. Distributed Ridge Regression with Feature Partitioning. I: 2018 52nd Asilomar Conference on Signals, Systems, and Computers. IEEE conference proceedings 2018 ISBN 978-1-5386-9218-9. s. 1423-1427. © 2018 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Paper 5: Gratton, Cristiano; Kumar Dasanadoddi Venkategowda, Naveen; Arablouei, Reza; Werner, Stefan. Distributed Learning over Networks with Non-Smooth Regularizers and Feature Partitioning. I: 2021 29th European Signal Processing Conference (EUSIPCO). IEEE conference proceedings 2021 ISBN 978-9-0827-9706-0 © 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Paper 6: Gratton, Cristiano; Kumar Dasanadoddi Venkategowda, Naveen; Arablouei, Reza; Werner, Stefan. “Decentralized optimization with distributed features and non-smooth objective functions,” 2022, arXiv: 2208.11224.