arXiv subjects

Antonious M. Girgis

Publications and source records attributed to Antonious M. Girgis.

14 recordsLinked to original sources

arXiv2024

Learning from straggler clients in federated learning

How well do existing federated learning algorithms learn from client devices that return model updates with a significant time delay? Is it even possible to learn effectively from clients that report back minutes, hours, or days after being scheduled? We answer these questions by developing Monte Carlo simulations of client latency that are guided by real-world applications. We study synchronous optimization algorithms like FedAvg and FedAdam as well as the asynchronous FedBuff algorithm, and observe that all these existing approaches struggle to learn from severely delayed clients. To improve upon this situation, we experiment with modifications, including distillation regularization and exponential moving averages of model weights. Finally, we introduce two new algorithms, FARe-DUST and FeAST-on-MSG, based on distillation and averaging, respectively. Experiments with the EMNIST, CIFAR-100, and StackOverflow benchmark federated learning tasks demonstrate that our new algorithms outperform existing ones in terms of accuracy for straggler clients, while also providing better trade-offs between training time and total accuracy.

cs.LG↗

arXiv2023

Multi-Message Shuffled Privacy in Federated Learning

We study differentially private distributed optimization under communication constraints. A server using SGD for optimization aggregates the client-side local gradients for model updates using distributed mean estimation (DME). We develop a communication-efficient private DME, using the recently developed multi-message shuffled (MMS) privacy framework. We analyze our proposed DME scheme to show that it achieves the order-optimal privacy-communication-performance tradeoff resolving an open question in [1], whether the shuffled models can improve the tradeoff obtained in Secure Aggregation. This also resolves an open question on the optimal trade-off for private vector sum in the MMS model. We achieve it through a novel privacy mechanism that non-uniformly allocates privacy at different resolutions of the local gradient vectors. These results are directly applied to give guarantees on private distributed learning algorithms using this for private gradient aggregation iteratively. We also numerically evaluate the private DME algorithms.

cs.LG↗

arXiv2022

Differentially Private Stochastic Linear Bandits: (Almost) for Free

In this paper, we propose differentially private algorithms for the problem of stochastic linear bandits in the central, local and shuffled models. In the central model, we achieve almost the same regret as the optimal non-private algorithms, which means we get privacy for free. In particular, we achieve a regret of $\tilde{O}(\sqrt{T}+\frac{1}ε)$ matching the known lower bound for private linear bandits, while the best previously known algorithm achieves $\tilde{O}(\frac{1}ε\sqrt{T})$. In the local case, we achieve a regret of $\tilde{O}(\frac{1}ε{\sqrt{T}})$ which matches the non-private regret for constant $ε$, but suffers a regret penalty when $ε$ is small. In the shuffled model, we also achieve regret of $\tilde{O}(\sqrt{T}+\frac{1}ε)$ %for small $ε$ as in the central case, while the best previously known algorithm suffers a regret of $\tilde{O}(\frac{1}ε{T^{3/5}})$. Our numerical evaluation validates our theoretical results.

cs.LG↗

arXiv2022

A Generative Framework for Personalized Learning and Estimation: Theory, Algorithms, and Privacy

A distinguishing characteristic of federated learning is that the (local) client data could have statistical heterogeneity. This heterogeneity has motivated the design of personalized learning, where individual (personalized) models are trained, through collaboration. There have been various personalization methods proposed in literature, with seemingly very different forms and methods ranging from use of a single global model for local regularization and model interpolation, to use of multiple global models for personalized clustering, etc. In this work, we begin with a generative framework that could potentially unify several different algorithms as well as suggest new algorithms. We apply our generative framework to personalized estimation, and connect it to the classical empirical Bayes' methodology. We develop private personalized estimation under this framework. We then use our generative framework for learning, which unifies several known personalized FL algorithms and also suggests new ones; we propose and study a new algorithm AdaPeD based on a Knowledge Distillation, which numerically outperforms several known algorithms. We also develop privacy for personalized learning methods with guarantees for user-level privacy and composition. We numerically evaluate the performance as well as the privacy for both the estimation and learning problems, demonstrating the advantages of our proposed methods.

cs.LG↗

arXiv2021

Renyi Differential Privacy of the Subsampled Shuffle Model in Distributed Learning

We study privacy in a distributed learning framework, where clients collaboratively build a learning model iteratively through interactions with a server from whom we need privacy. Motivated by stochastic optimization and the federated learning (FL) paradigm, we focus on the case where a small fraction of data samples are randomly sub-sampled in each round to participate in the learning process, which also enables privacy amplification. To obtain even stronger local privacy guarantees, we study this in the shuffle privacy model, where each client randomizes its response using a local differentially private (LDP) mechanism and the server only receives a random permutation (shuffle) of the clients' responses without their association to each client. The principal result of this paper is a privacy-optimization performance trade-off for discrete randomization mechanisms in this sub-sampled shuffle privacy model. This is enabled through a new theoretical technique to analyze the Renyi Differential Privacy (RDP) of the sub-sampled shuffle model. We numerically demonstrate that, for important regimes, with composition our bound yields significant improvement in privacy guarantee over the state-of-the-art approximate Differential Privacy (DP) guarantee (with strong composition) for sub-sampled shuffled models. We also demonstrate numerically significant improvement in privacy-learning performance operating point using real data sets.

cs.LG↗

arXiv2021

A Field Guide to Federated Optimization

Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy protection. The distributed learning process can be formulated as solving federated optimization problems, which emphasize communication efficiency, data heterogeneity, compatibility with privacy and system requirements, and other constraints that are not primary considerations in other problem settings. This paper provides recommendations and guidelines on formulating, designing, evaluating and analyzing federated optimization algorithms through concrete examples and practical implementation, with a focus on conducting effective simulations to infer real-world performance. The goal of this work is not to survey the current literature, but to inspire researchers and practitioners to design federated learning algorithms that can be used in various practical applications.

cs.LG↗

arXiv2021

On the Renyi Differential Privacy of the Shuffle Model

The central question studied in this paper is Renyi Differential Privacy (RDP) guarantees for general discrete local mechanisms in the shuffle privacy model. In the shuffle model, each of the $n$ clients randomizes its response using a local differentially private (LDP) mechanism and the untrusted server only receives a random permutation (shuffle) of the client responses without association to each client. The principal result in this paper is the first non-trivial RDP guarantee for general discrete local randomization mechanisms in the shuffled privacy model, and we develop new analysis techniques for deriving our results which could be of independent interest. In applications, such an RDP guarantee is most useful when we use it for composing several private interactions. We numerically demonstrate that, for important regimes, with composition our bound yields an improvement in privacy guarantee by a factor of $8\times$ over the state-of-the-art approximate Differential Privacy (DP) guarantee (with standard composition) for shuffled models. Moreover, combining with Poisson subsampling, our result leads to at least $10\times$ improvement over subsampled approximate DP with standard composition.

cs.CR↗

arXiv2020

Shuffled Model of Federated Learning: Privacy, Communication and Accuracy Trade-offs

We consider a distributed empirical risk minimization (ERM) optimization problem with communication efficiency and privacy requirements, motivated by the federated learning (FL) framework. Unique challenges to the traditional ERM problem in the context of FL include (i) need to provide privacy guarantees on clients' data, (ii) compress the communication between clients and the server, since clients might have low-bandwidth links, (iii) work with a dynamic client population at each round of communication between the server and the clients, as a small fraction of clients are sampled at each round. To address these challenges we develop (optimal) communication-efficient schemes for private mean estimation for several $\ell_p$ spaces, enabling efficient gradient aggregation for each iteration of the optimization solution of the ERM. We also provide lower and upper bounds for mean estimation with privacy and communication constraints for arbitrary $\ell_p$ spaces. To get the overall communication, privacy, and optimization performance operation point, we combine this with privacy amplification opportunities inherent to this setup. Our solution takes advantage of the inherent privacy amplification provided by client sampling and data sampling at each client (through Stochastic Gradient Descent) as well as the recently developed privacy framework using anonymization, which effectively presents to the server responses that are randomly shuffled with respect to the clients. Putting these together, we demonstrate that one can get the same privacy, optimization-performance operating point developed in recent methods that use full-precision communication, but at a much lower communication cost, i.e., effectively getting communication efficiency for "free".

cs.LG↗

arXiv2020

Successive Refinement of Privacy

This work examines a novel question: how much randomness is needed to achieve local differential privacy (LDP)? A motivating scenario is providing {\em multiple levels of privacy} to multiple analysts, either for distribution or for heavy-hitter estimation, using the \emph{same} (randomized) output. We call this setting \emph{successive refinement of privacy}, as it provides hierarchical access to the raw data with different privacy levels. For example, the same randomized output could enable one analyst to reconstruct the input, while another can only estimate the distribution subject to LDP requirements. This extends the classical Shannon (wiretap) security setting to local differential privacy. We provide (order-wise) tight characterizations of privacy-utility-randomness trade-offs in several cases for distribution estimation, including the standard LDP setting under a randomness constraint. We also provide a non-trivial privacy mechanism for multi-level privacy. Furthermore, we show that we cannot reuse random keys over time while preserving privacy of each user.

cs.CR↗

arXiv2018

A Converse Bound for Cache-Aided Interference Networks

In this paper, an interference network with arbitrary number of transmitters and receivers is studied, where each transmitter is equipped with a finite size cache. We obtain an information-theoretic lower bound on both the peak normalized delivery time (NDT), and the \textit{expected} NDT of cache-aided interference networks with uniform content popularity. For the peak NDT, we show that our lower bound is strictly tighter than the bound in the literature for small cache sizes. Moreover, we show that the feasibility region on the expected NDT is bigger than that of the peak NDT.

cs.IT↗

arXiv2017

Degrees of Freedom of Interference Networks with Transmitter-Side Caches

This paper studies cache-aided interference networks with arbitrary number of transmitters and receivers, whereby each transmitter has a cache memory of finite size. Each transmitter fills its cache memory from a content library of files in the placement phase. In the subsequent delivery phase, each receiver requests one of the library files, and the transmitters are responsible for delivering the requested files from their caches to the receivers. The objective is to design schemes for the placement and delivery phases to maximize the sum degrees of freedom (sum-DoF) which expresses the capacity of the interference network at the high signal-to-noise ratio regime. Our work mainly focuses on a commonly used uncoded placement strategy. We provide an information-theoretic bound on the sum-DoF for this placement strategy. We demonstrate by an example that the derived bound is tighter than the bounds existing in the literature for small cache sizes. We propose a novel delivery scheme with a higher achievable sum-DoF than those previously given in the literature. The results reveal that the reciprocal of sum-DoF decreases linearly as the transmitter cache size increases. Therefore, increasing cache sizes at transmitters translates to increasing the sum-DoF and, hence, the capacity of the interference networks.

cs.IT↗

arXiv2017

Decentralized Coded Caching in Wireless Networks: Trade-off between Storage and Latency

This paper studies the decentralized coded caching for a Fog Radio Access Network (F-RAN), whereby two edge-nodes (ENs) connected to a cloud server via fronthaul links with limited capacity are serving the requests of $K_r$ users. We consider all ENs and users are equipped with caches. A decentralized content placement is proposed to independently store contents at each network node during the off-peak hours. After that, we design a coded delivery scheme in order to deliver the user demands during the peak-hours under the objective of minimizing the normalized delivery time (NDT), which refers to the worst case delivery latency. An information-theoretic lower bound on the minimum NDT is derived for arbitrary number of ENs and users. We evaluate numerically the performance of the decentralized scheme. Additionally, we prove the approximate optimality of the decentralized scheme for a special case when the caches are only available at the ENs.

cs.IT↗

arXiv2016

Optimal Energy Allocation For Delay-Constrained Traffic Over Fading Multiple Access Channels

In this paper, we consider a multiple-access fading channel where $N$ users transmit to a single base station (BS) within a limited number of time slots. We assume that each user has a fixed amount of energy available to be consumed over the transmission window. We derive the optimal energy allocation policy for each user that maximizes the total system throughput under two different assumptions on the channel state information. First, we consider the offline allocation problem where the channel states are known a priori before transmission. We solve a convex optimization problem to maximize the sum-throughput under energy and delay constraints. Next, we consider the online allocation problem, where the channels are causally known to the BS and obtain the optimal energy allocation via dynamic programming when the number of users is small. We also develop a suboptimal resource allocation algorithm whose performance is close to the optimal one. Numerical results are presented showing the superiority of the proposed algorithms over baseline algorithms in various scenarios.

cs.IT↗

arXiv2016

Proactive Location-Based Scheduling of Delay-Constrained Traffic Over Fading Channels

In this paper, proactive resource allocation based on user location for point-to-point communication over fading channels is introduced, whereby the source must transmit a packet when the user requests it within a deadline of a single time slot. We introduce a prediction model in which the source predicts the request arrival $T_p$ slots ahead, where $T_p$ denotes the prediction window (PW) size. The source allocates energy to transmit some bits proactively for each time slot of the PW with the objective of reducing the transmission energy over the non-predictive case. The requests are predicted based on the user location utilizing the prior statistics about the user requests at each location. We also assume that the prediction is not perfect. We propose proactive scheduling policies to minimize the expected energy consumption required to transmit the requested packets under two different assumptions on the channel state information at the source. In the first scenario, offline scheduling, we assume the channel states are known a-priori at the source at the beginning of the PW. In the second scenario, online scheduling, it is assumed that the source has causal knowledge of the channel state. Numerical results are presented showing the gains achieved by using proactive scheduling policies compared with classical (reactive) networks. Simulation results also show that increasing the PW size leads to a significant reduction in the consumed transmission energy even with imperfect prediction.

cs.IT↗