arXiv subjects

Salim El Rouayheb

Publications and source records attributed to Salim El Rouayheb.

At least 37 records · Page 2Linked to original sources

arXiv2021

Degree Tables for Secure Distributed Matrix Multiplication

We consider the problem of secure distributed matrix multiplication (SDMM) in which a user wishes to compute the product of two matrices with the assistance of honest but curious servers. We construct polynomial codes for SDMM by studying a recently introduced combinatorial tool called the degree table. For a fixed partitioning, minimizing the total communication cost of a polynomial code for SDMM is equivalent to minimizing $N$, the number of distinct elements in the corresponding degree table. We propose new constructions of degree tables with a low number of distinct elements. These new constructions lead to a general family of polynomial codes for SDMM, which we call $\mathsf{GASP}_{r}$ (Gap Additive Secure Polynomial codes) parametrized by an integer $r$. $\mathsf{GASP}_{r}$ outperforms all previously known polynomial codes for SDMM under an outer product partitioning. We also present lower bounds on $N$ and prove the optimality or asymptotic optimality of our constructions for certain regimes. Moreover, we formulate the construction of optimal degree tables as an integer linear program and use it to prove the optimality of $\mathsf{GASP}_{r}$ for all the system parameters that we were able to test.

cs.IT↗

arXiv2021

Private Multi-Group Aggregation

We study the differentially private multi group aggregation (PMGA) problem. This setting involves a single server and $n$ users. Each user belongs to one of $k$ distinct groups and holds a discrete value. The goal is to design schemes that allow the server to find the aggregate (sum) of the values in each group (with high accuracy) under communication and local differential privacy constraints. The privacy constraint guarantees that the user's group remains private. This is motivated by applications where a user's group can reveal sensitive information, such as his religious and political beliefs, health condition, or race. We propose a novel scheme, dubbed Query and Aggregate (Q\&A) for PMGA. The novelty of Q\&A is that it is an interactive aggregation scheme. In Q\&A, each user is assigned a random query matrix, to which he sends the server an answer based on his group and value. We characterize the Q\&A scheme's performance in terms of accuracy (MSE), privacy, and communication. We compare Q\&A to the Randomized Group (RG) scheme, which is non-interactive and adapts existing randomized response schemes to the PMGA setting. We observe that typically Q\&A outperforms RG, in terms of privacy vs. utility, in the high privacy regime.

cs.IT↗

arXiv2021

ON-OFF Privacy Against Correlation Over Time

We consider the problem of ON-OFF privacy in which a user is interested in the latest message generated by one of n sources available at a server. The user has the choice to turn privacy ON or OFF depending on whether he wants to hide his interest at the time or not. The challenge of allowing the privacy to be toggled between ON and OFF is that the user's online behavior is correlated over time. Therefore, the user cannot simply ignore the privacy requirement when privacy is OFF. We represent the user's correlated requests by an n-state Markov chain. Our goal is to design ON-OFF privacy schemes with optimal download rate that ensure privacy for past and future requests. We devise a polynomial-time algorithm to construct an ON-OFF privacy scheme. Moreover, we present an upper bound on the achievable rate. We show that the proposed scheme is optimal and the upper bound is tight for some special families of Markov chains. We also give an implicit characterization of the optimal achievable rate as a linear programming (LP).

cs.IT↗

arXiv2021

Private Weighted Random Walk Stochastic Gradient Descent

We consider a decentralized learning setting in which data is distributed over nodes in a graph. The goal is to learn a global model on the distributed data without involving any central entity that needs to be trusted. While gossip-based stochastic gradient descent (SGD) can be used to achieve this learning objective, it incurs high communication and computation costs, since it has to wait for all the local models at all the nodes to converge. To speed up the convergence, we propose instead to study random walk based SGD in which a global model is updated based on a random walk on the graph. We propose two algorithms based on two types of random walks that achieve, in a decentralized way, uniform sampling and importance sampling of the data. We provide a non-asymptotic analysis on the rate of convergence, taking into account the constants related to the data and the graph. Our numerical results show that the weighted random walk based algorithm has a better performance for high-variance data. Moreover, we propose a privacy-preserving random walk algorithm that achieves local differential privacy based on a Gamma noise mechanism that we propose. We also give numerical results on the convergence of this algorithm and show that it outperforms additive Laplace-based privacy mechanisms.

cs.IT↗

arXiv2021

Advances and Open Problems in Federated Learning

Federated learning (FL) is a machine learning setting where many clients (e.g. mobile devices or whole organizations) collaboratively train a model under the orchestration of a central server (e.g. service provider), while keeping the training data decentralized. FL embodies the principles of focused data collection and minimization, and can mitigate many of the systemic privacy risks and costs resulting from traditional, centralized machine learning and data science approaches. Motivated by the explosive growth in FL research, this paper discusses recent advances and presents an extensive collection of open problems and challenges.

cs.LG↗

arXiv2021

Codes for Correcting Localized Deletions

We consider the problem of constructing binary codes for correcting deletions that are localized within certain parts of the codeword that are unknown a priori. The model that we study is when $δ\leq w$ deletions are localized in a window of size $w$ bits. These $δ$ deletions do not necessarily occur in consecutive positions, but are restricted to the window of size $w$. The localized deletions model is a generalization of the bursty model, in which all the deleted bits are consecutive. In this paper, we construct new explicit codes for the localized model, based on the family of Guess & Check codes which was previously introduced by the authors. The codes that we construct can correct, with high probability, $δ\leq w$ deletions that are localized in a single window of size $w$, where $w$ grows with the block length. Moreover, these codes are systematic; have low redundancy; and have efficient deterministic encoding and decoding algorithms. We also generalize these codes to deletions that are localized within multiple windows in the codeword.

cs.IT↗

arXiv2020

Notes on Communication and Computation in Secure Distributed Matrix Multiplication

We consider the problem of secure distributed matrix multiplication in which a user wishes to compute the product of two matrices with the assistance of honest but curious servers. In this paper, we answer the following question: Is it beneficial to offload the computations if security is a concern? We answer this question in the affirmative by showing that by adjusting the parameters in a polynomial code we can obtain a trade-off between the user's and the servers' computational time. Indeed, we show that if the computational time complexity of an operation in $\mathbb{F}_q$ is at most $\mathcal{Z}_q$ and the computational time complexity of multiplying two $n\times n$ matrices is $\mathcal{O}(n^ω\mathcal{Z}_q)$ then, by optimizing the trade-off, the user together with the servers can compute the multiplication in $\mathcal{O}(n^{4-\frac{6}{ω+1}} \mathcal{Z}_q)$ time. We also show that if the user is only concerned in optimizing the download rate, a common assumption in the literature, then the problem can be converted into a simple private information retrieval problem by means of a scheme we call Private Oracle Querying. However, this comes at large upload and computational costs for both the user and the servers.

cs.IT↗

arXiv2020

Adaptive Distributed Stochastic Gradient Descent for Minimizing Delay in the Presence of Stragglers

We consider the setting where a master wants to run a distributed stochastic gradient descent (SGD) algorithm on $n$ workers each having a subset of the data. Distributed SGD may suffer from the effect of stragglers, i.e., slow or unresponsive workers who cause delays. One solution studied in the literature is to wait at each iteration for the responses of the fastest $k<n$ workers before updating the model, where $k$ is a fixed parameter. The choice of the value of $k$ presents a trade-off between the runtime (i.e., convergence rate) of SGD and the error of the model. Towards optimizing the error-runtime trade-off, we investigate distributed SGD with adaptive $k$. We first design an adaptive policy for varying $k$ that optimizes this trade-off based on an upper bound on the error as a function of the wall-clock time which we derive. Then, we propose an algorithm for adaptive distributed SGD that is based on a statistical heuristic. We implement our algorithm and provide numerical simulations which confirm our intuition and theoretical analysis.

cs.LG↗

arXiv2020

GASP Codes for Secure Distributed Matrix Multiplication

We consider the problem of secure distributed matrix multiplication (SDMM) in which a user wishes to compute the product of two matrices with the assistance of honest but curious servers. We construct polynomial codes for SDMM by studying a combinatorial problem on a special type of addition table, which we call the degree table. The codes are based on arithmetic progressions, and are thus named GASP (Gap Additive Secure Polynomial) Codes. GASP Codes are shown to outperform all previously known polynomial codes for secure distributed matrix multiplication in terms of download rate.

cs.IT↗

arXiv2019

Private and Rateless Adaptive Coded Matrix-Vector Multiplication

Edge computing is emerging as a new paradigm to allow processing data near the edge of the network, where the data is typically generated and collected. This enables critical computations at the edge in applications such as Internet of Things (IoT), in which an increasing number of devices (sensors, cameras, health monitoring devices, etc.) collect data that needs to be processed through computationally intensive algorithms with stringent reliability, security and latency constraints. Our key tool is the theory of coded computation, which advocates mixing data in computationally intensive tasks by employing erasure codes and offloading these tasks to other devices for computation. Coded computation is recently gaining interest, thanks to its higher reliability, smaller delay, and lower communication costs. In this paper, we develop a private and rateless adaptive coded computation (PRAC) algorithm for distributed matrix-vector multiplication by taking into account (i) the privacy requirements of IoT applications and devices, and (ii) the heterogeneous and time-varying resources of edge devices. We show that PRAC outperforms known secure coded computing methods when resources are heterogeneous. We provide theoretical guarantees on the performance of PRAC and its comparison to baselines. Moreover, we confirm our theoretical results through simulations and implementations on Android-based smartphones.

cs.IT↗

arXiv2019

Secure Coded Cooperative Computation at the Heterogeneous Edge against Byzantine Attacks

Edge computing is emerging as a new paradigm to allow processing data at the edge of the network, where data is typically generated and collected, by exploiting multiple devices at the edge collectively. However, offloading tasks to other devices leaves the edge computing applications at the complete mercy of an attacker. One of the attacks, which is also the focus of this work, is Byzantine attacks, where one or more devices can corrupt the offloaded tasks. Furthermore, exploiting the potential of edge computing is challenging mainly due to the heterogeneous and time-varying nature of the devices at the edge. In this paper, we develop a secure coded cooperative computation mechanism (SC3) that provides both security and computation efficiency guarantees by gracefully combining homomorphic hash functions and coded cooperative computation. Homomorphic hash functions are used against Byzantine attacks and coded cooperative computation is used to improve computation efficiency when edge resources are heterogeneous and time-varying. Simulations results show that SC3 improves task completion delay significantly.

cs.DC↗

arXiv2019

Preserving ON-OFF Privacy for Past and Future Requests

We study the ON-OFF privacy problem. At each time, the user is interested in the latest message of one of $N$ sources. Moreover, the user is assumed to be incentivized to turn privacy ON or OFF whether he/she needs it or not. When privacy is ON, the user wants to keep private which source he/she is interested in. The challenge here is that the user's behavior is correlated over time. Therefore, the user cannot simply ignore privacy when privacy is OFF, because this may leak information about his/her behavior when privacy was ON due to correlation. We model the user's requests by a Markov chain. The goal is to design ON-OFF privacy schemes with optimal download rate that ensure privacy for past and future requests. The user is assumed to know future requests within a window of positive size $ω$ and uses it to construct privacy-preserving queries. In this paper, we construct ON-OFF privacy schemes for $N=2$ sources and prove their optimality.

cs.IT↗

arXiv2019

Stochastic Gradient Coding for Straggler Mitigation in Distributed Learning

We consider distributed gradient descent in the presence of stragglers. Recent work on \em gradient coding \em and \em approximate gradient coding \em have shown how to add redundancy in distributed gradient descent to guarantee convergence even if some workers are \em stragglers\em---that is, slow or non-responsive. In this work we propose an approximate gradient coding scheme called \em Stochastic Gradient Coding \em (SGC), which works when the stragglers are random. SGC distributes data points redundantly to workers according to a pair-wise balanced design, and then simply ignores the stragglers. We prove that the convergence rate of SGC mirrors that of batched Stochastic Gradient Descent (SGD) for the $\ell_2$ loss function, and show how the convergence rate can improve with the redundancy. We also provide bounds for more general convex loss functions. We show empirically that SGC requires a small amount of redundancy to handle a large number of stragglers and that it can outperform existing approximate gradient codes when the number of stragglers is large.

cs.IT↗

arXiv2019

Codes with Locality in the Rank and Subspace Metrics

We extend the notion of locality from the Hamming metric to the rank and subspace metrics. Our main contribution is to construct a class of array codes with locality constraints in the rank metric. Our motivation for constructing such codes stems from designing codes for efficient data recovery from correlated and/or mixed (i.e., complete and partial) failures in distributed storage systems. Specifically, the proposed local rank-metric codes can recover locally from 'crisscross errors and erasures', which affect a limited number of rows and/or columns of the storage system. We also derive a Singleton-like upper bound on the minimum rank distance of (linear) codes with 'rank-locality' constraints. Our proposed construction achieves this bound for a broad range of parameters. The construction builds upon Tamo and Barg's method for constructing locally repairable codes with optimal minimum Hamming distance. Finally, we construct a class of constant-dimension subspace codes (also known as Grassmannian codes) with locality constraints in the subspace metric. The key idea is to show that a Grassmannian code with locality can be easily constructed from a rank-metric code with locality by using the lifting method proposed by Silva et al. We present an application of such codes for distributed storage systems, wherein nodes are connected over a network that can introduce errors and erasures.

cs.IT↗

arXiv2019

ON-OFF Privacy with Correlated Requests

We introduce the ON-OFF privacy problem. At each time, the user is interested in the latest message of one of N online sources chosen at random, and his privacy status can be ON or OFF for each request. Only when privacy is ON the user wants to hide the source he is interested in. The problem is to design ON-OFF privacy schemes with maximum download rate that allow the user to obtain privately his requested messages. In many realistic scenarios, the user's requests are correlated since they depend on his personal attributes such as age, gender, political views, or geographical location. Hence, even when privacy is OFF, he cannot simply reveal his request since this will leak information about his requests when privacy was ON. We study the case when the users's requests can be modeled by a Markov chain and N=2 sources. In this case, we propose an ON-OFF privacy scheme and prove its optimality.

cs.IT↗

arXiv2019

List Decoding of Deletions Using Guess & Check Codes

Guess & Check (GC) codes are systematic binary codes that can correct multiple deletions, with high probability. GC codes have logarithmic redundancy in the length of the message $k$, and the encoding and decoding algorithms of these codes are deterministic and run in polynomial time for a constant number of deletions $δ$. The unique decoding properties of GC codes were examined in a previous work by the authors. In this paper, we investigate the list decoding performance of these codes. Namely, we study the average size and the maximum size of the list obtained by a GC decoder for a constant number of deletions $δ$. The theoretical results show that: (i) the average size of the list approaches $1$ as $k$ grows; and (ii) there exists an infinite sequence of GC codes indexed by $k$, whose maximum list size in upper bounded by a constant that is independent of $k$. We also provide numerical simulations on the list decoding performance of GC codes for multiple values of $k$ and $δ$.

cs.IT↗

arXiv2019

Single-Server Multi-Message Individually-Private Information Retrieval with Side Information

We consider a multi-user variant of the private information retrieval problem described as follows. Suppose there are $D$ users, each of which wants to privately retrieve a distinct message from a server with the help of a trusted agent. We assume that the agent has a random subset of $M$ messages that is not known to the server. The goal of the agent is to collectively retrieve the users' requests from the server. For protecting the privacy of users, we introduce the notion of individual-privacy -- the agent is required to protect the privacy only for each individual user (but may leak some correlations among user requests). We refer to this problem as Individually-Private Information Retrieval with Side Information (IPIR-SI). We first establish a lower bound on the capacity, which is defined as the maximum achievable download rate, of the IPIR-SI problem by presenting a novel achievability protocol. Next, we characterize the capacity of IPIR-SI problem for $M = 1$ and $D = 2$. In the process of characterizing the capacity for arbitrary $M$ and $D$ we present a novel combinatorial conjecture, that may be of independent interest.

cs.IT↗

arXiv2018

One-Shot PIR: Refinement and Lifting

We study a class of private information retrieval (PIR) methods that we call one-shot schemes. The intuition behind one-shot schemes is the following. The user's query is regarded as a dot product of a query vector and the message vector (database) stored at multiple servers. Privacy, in an information theoretic sense, is then achieved by encrypting the query vector using a secure linear code, such as secret sharing. Several PIR schemes in the literature, in addition to novel ones constructed here, fall into this class. One-shot schemes provide an insightful link between PIR and data security against eavesdropping. However, their download rate is not optimal, i.e., they do not achieve the PIR capacity. Our main contribution is two transformations of one-shot schemes, which we call refining and lifting. We show that refining and lifting one-shot schemes gives capacity-achieving schemes for the cases when the PIR capacity is known. In the other cases, when the PIR capacity is still unknown, refining and lifting one-shot schemes gives the best download rate so far.

cs.IT↗