arXiv subjects

Arsany Guirguis

Publications and source records attributed to Arsany Guirguis.

8 recordsLinked to original sources

arXiv2024

Accelerating Transfer Learning with Near-Data Computation on Cloud Object Stores

Storage disaggregation underlies today's cloud and is naturally complemented by pushing down some computation to storage, thus mitigating the potential network bottleneck between the storage and compute tiers. We show how ML training benefits from storage pushdowns by focusing on transfer learning (TL), the widespread technique that democratizes ML by reusing existing knowledge on related tasks. We propose HAPI, a new TL processing system centered around two complementary techniques that address challenges introduced by disaggregation. First, applications must carefully balance execution across tiers for performance. HAPI judiciously splits the TL computation during the feature extraction phase yielding pushdowns that not only improve network time but also improve total TL training time by overlapping the execution of consecutive training iterations across tiers. Second, operators want resource efficiency from the storage-side computational resources. HAPI employs storage-side batch size adaptation allowing increased storage-side pushdown concurrency without affecting training accuracy. HAPI yields up to 2.5x training speed-up while choosing in 86.8% of cases the best performing split point or one that is at most 5% off from the best.

cs.LG↗

arXiv2021

Collaborative Learning in the Jungle (Decentralized, Byzantine, Heterogeneous, Asynchronous and Nonconvex Learning)

We study Byzantine collaborative learning, where $n$ nodes seek to collectively learn from each others' local data. The data distribution may vary from one node to another. No node is trusted, and $f < n$ nodes can behave arbitrarily. We prove that collaborative learning is equivalent to a new form of agreement, which we call averaging agreement. In this problem, nodes start each with an initial vector and seek to approximately agree on a common vector, which is close to the average of honest nodes' initial vectors. We present two asynchronous solutions to averaging agreement, each we prove optimal according to some dimension. The first, based on the minimum-diameter averaging, requires $ n \geq 6f+1$, but achieves asymptotically the best-possible averaging constant up to a multiplicative constant. The second, based on reliable broadcast and coordinate-wise trimmed mean, achieves optimal Byzantine resilience, i.e., $n \geq 3f+1$. Each of these algorithms induces an optimal Byzantine collaborative learning protocol. In particular, our equivalence yields new impossibility theorems on what any collaborative learning algorithm can achieve in adversarial and heterogeneous environments.

cs.LG↗

arXiv2020

Garfield: System Support for Byzantine Machine Learning

We present Garfield, a library to transparently make machine learning (ML) applications, initially built with popular (but fragile) frameworks, e.g., TensorFlow and PyTorch, Byzantine-resilient. Garfield relies on a novel object-oriented design, reducing the coding effort, and addressing the vulnerability of the shared-graph architecture followed by classical ML frameworks. Garfield encompasses various communication patterns and supports computations on CPUs and GPUs, allowing addressing the general question of the very practical cost of Byzantine resilience in SGD-based ML applications. We report on the usage of Garfield on three main ML architectures: (a) a single server with multiple workers, (b) several servers and workers, and (c) peer-to-peer settings. Using Garfield, we highlight several interesting facts about the cost of Byzantine resilience. In particular, (a) Byzantine resilience, unlike crash resilience, induces an accuracy loss, (b) the throughput overhead comes more from communication than from robust aggregation, and (c) tolerating Byzantine servers costs more than tolerating Byzantine workers.

cs.LG↗

arXiv2020

Fast Machine Learning with Byzantine Workers and Servers

Machine Learning (ML) solutions are nowadays distributed and are prone to various types of component failures, which can be encompassed in so-called Byzantine behavior. This paper introduces LiuBei, a Byzantine-resilient ML algorithm that does not trust any individual component in the network (neither workers nor servers), nor does it induce additional communication rounds (on average), compared to standard non-Byzantine resilient algorithms. LiuBei builds upon gradient aggregation rules (GARs) to tolerate a minority of Byzantine workers. Besides, LiuBei replicates the parameter server on multiple machines instead of trusting it. We introduce a novel filtering mechanism that enables workers to filter out replies from Byzantine server replicas without requiring communication with all servers. Such a filtering mechanism is based on network synchrony, Lipschitz continuity of the loss function, and the GAR used to aggregate workers' gradients. We also introduce a protocol, scatter/gather, to bound drifts between models on correct servers with a small number of communication messages. We theoretically prove that LiuBei achieves Byzantine resilience to both servers and workers and guarantees convergence. We build LiuBei using TensorFlow, and we show that LiuBei tolerates Byzantine behavior with an accuracy loss of around 5% and around 24% convergence overhead compared to vanilla TensorFlow. We moreover show that the throughput gain of LiuBei compared to another state-of-the-art Byzantine-resilient ML algorithm (that assumes network asynchrony) is 70%.

cs.LG↗

arXiv2020

Genuinely Distributed Byzantine Machine Learning

Machine Learning (ML) solutions are nowadays distributed, according to the so-called server/worker architecture. One server holds the model parameters while several workers train the model. Clearly, such architecture is prone to various types of component failures, which can be all encompassed within the spectrum of a Byzantine behavior. Several approaches have been proposed recently to tolerate Byzantine workers. Yet all require trusting a central parameter server. We initiate in this paper the study of the ``general'' Byzantine-resilient distributed machine learning problem where no individual component is trusted. We show that this problem can be solved in an asynchronous system, despite the presence of $\frac{1}{3}$ Byzantine parameter servers and $\frac{1}{3}$ Byzantine workers (which is optimal). We present a new algorithm, ByzSGD, which solves the general Byzantine-resilient distributed machine learning problem by relying on three major schemes. The first, Scatter/Gather, is a communication scheme whose goal is to bound the maximum drift among models on correct servers. The second, Distributed Median Contraction (DMC), leverages the geometric properties of the median in high dimensional spaces to bring parameters within the correct servers back close to each other, ensuring learning convergence. The third, Minimum-Diameter Averaging (MDA), is a statistically-robust gradient aggregation rule whose goal is to tolerate Byzantine workers. MDA requires loose bound on the variance of non-Byzantine gradient estimates, compared to existing alternatives (e.g., Krum). Interestingly, ByzSGD ensures Byzantine resilience without adding communication rounds (on a normal path), compared to vanilla non-Byzantine alternatives. ByzSGD requires, however, a larger number of messages which, we show, can be reduced if we assume synchrony.

cs.DC↗

arXiv2016

Channel Selection Scheme for Cooperative Routing Protocols in Cognitive Radio Networks

Cognitive radio networks (CRNs) propose a smart solution for spectrum usage inefficiency. Routing protocols for CRNs follow different criteria to choose the best route to the destination and to avoid the interference with primary users. Some protocols use cooperative communication techniques to achieve the full coexistence between secondary users (SUs) and primary users (PUs). Although using such cross-layer techniques have a great impact on the quality of the chosen routes, the existing work do not choose the best channel to send the data over. Thus, the available spectrum is not utilized efficiently. In this work, we propose CSCR, a channel selection scheme for cooperation-based routing protocols in CRNs. The proposed scheme increases the spectrum utilization through integrating the channels selection in the route discovery phase of the cooperation-based routing protocols. The best channels, that are less congested with primary users and that lead to minimum switching overhead, are chosen while constructing the cooperative group. Evaluating CSCR via NS2 simulations shows that it outperforms its counterparts in terms of goodput, end-to-end delay, and packet delivery ratio. The proposed scheme can enhance the network goodput, in some cases, by more than 150%, as compared to other related protocols.

cs.NI↗

arXiv2016

Cooperation-based Routing in Cognitive Radio Networks

Primary user activity is a major bottleneck for existing routing protocols in cognitive radio networks. Typical routing protocols avoid areas that are highly congested with primary users, leaving only a small fragment of available links for secondary route construction. In addition, wireless links are prone to channel impairments such as multipath fading; which renders the quality of the available links highly fluctuating. In this paper, we investigate using cooperative communication mechanisms to reveal new routing opportunities, enhance route qualities, and enable true coexistence of primary and secondary networks. As a result, we propose Undercover: a cooperative routing protocol that utilizes the available location information to assist in the routing process. Specifically, our protocol revisits a fundamental assumption taken by the state of the art routing protocols designed for cognitive radio networks. Using Undercover, secondary users can transmit in the regions of primary users activity through utilizing cooperative communication techniques to null out transmission at primary receivers via beamforming. In addition, the secondary links qualities are enhanced using cooperative diversity. To account for the excessive levels of interference typically incurred due to cooperative transmissions, we allow our protocol to be interference-aware. Thus, cooperative transmissions are penalized in accordance to the amount of negatively affected secondary flows. We evaluate the performance of our proposed protocol via NS2 simulations which show that our protocol can enhance the network goodput by a ratio reaches up to 250% compared to other popular cognitive routing protocols with minimal added overhead.

cs.NI↗

arXiv2014

Primary User-aware Network Coding for Multi-hop Cognitive Radio Networks

Network coding has proved its efficiency in increasing the network performance for traditional ad-hoc networks. In this paper, we investigate using network coding for enhancing the throughput of multi-hop cognitive radio networks. We formulate the network coding throughput maximization problem as a graph theory problem, where different constraints and primary users' characteristics are mapped to the graph structure. We then show that the optimal solution to this problem in NP-hard and propose a heuristic algorithm to efficiently solve it. Evaluation of the proposed algorithm through NS2 simulations shows that we can increase the throughput of the constrained secondary users' network by 150\% to 200\% for a wide range of scenarios covering different primary users' densities, traffic loads, and spectrum availability.

cs.NI↗