Privacy-preserving distributed machine learning for artificial intelligence of things

Abstract

This thesis proposes machine learning algorithms that can be fully distributed over ad-hoc networks of machines/agents. Developing distributed algorithms for artificial intelligence is necessary since running machine-learning-based data analytics on a single central hub may be unfeasible due to computing/communication costs. In the context of distributed learning, privacy violation risks due to curious members of the network or eavesdroppers make the development of privacy-preserving distributed algorithms imperative. The main contributions of the thesis are around developing distributed machine learning algorithms for artificial intelligence of things including distributed algorithms with intrinsic privacy-preserving properties. In particular, the contributions can be grouped in the following categories: • distributed learning over networks with horizontal/row partitioning of data • intrinsically privacy-preserving distributed learning with zeroth-order optimization • distributed learning over networks with vertical/feature partitioning of data. In the context of distributed learning with horizontal partitioning of data, we propose a new distributed algorithm to solve the total least-squares (TLS) problem and a privacy-preserving distributed algorithm to minimize a regularized empirical risk function when the first-order information is not available. We show that the latter algorithm has intrinsic privacy-preserving properties. Most existing privacy-preserving distributed optimization/estimation algorithms exploit some perturbation mechanism to preserve privacy, which comes at the cost of reduced accuracy. Contrarily, we exploit the inherent randomness due to the use of a zeroth-order method and show that this stochasticity is sufficient to ensure differential privacy. Moreover, we demonstrate that the proposed algorithm outperforms the existing differentially-private ones in terms of accuracy while yielding similar privacy guarantees. In the context of distributed learning with feature partitioning of data, we develop a new distributed algorithm to solve the ridge regression problem. Subsequently, we develop a new algorithm that is designed for an ‘2-norm-square cost function with non-smooth regularizers. Finally, we develop a new consensus-based distributed algorithm for solving learning problems when the data is distributed among agents in feature partitions and computing the conjugate of the possibly non-smooth cost or regularizer functions is challenging or unfeasible. The proposed algorithm is designed for optimizing generic non-smooth objective functions over arbitrary graphs without using or computing any conjugate function. All the above-mentioned algorithms are fully-distributed and based on the alternating direction method of multipliers (ADMM) that is suitable for distributed optimization thanks to its scalability and robustness properties. We prove theoretically that the proposed algorithms converge. We also confirm their network-wide convergence via simulations.