arXiv subjects

Amir Salman Avestimehr

Publications and source records attributed to Amir Salman Avestimehr.

15 recordsLinked to original sources

arXiv2021

CodedReduce: A Fast and Robust Framework for Gradient Aggregation in Distributed Learning

We focus on the commonly used synchronous Gradient Descent paradigm for large-scale distributed learning, for which there has been a growing interest to develop efficient and robust gradient aggregation strategies that overcome two key system bottlenecks: communication bandwidth and stragglers' delays. In particular, Ring-AllReduce (RAR) design has been proposed to avoid bandwidth bottleneck at any particular node by allowing each worker to only communicate with its neighbors that are arranged in a logical ring. On the other hand, Gradient Coding (GC) has been recently proposed to mitigate stragglers in a master-worker topology by allowing carefully designed redundant allocation of the data set to the workers. We propose a joint communication topology design and data set allocation strategy, named CodedReduce (CR), that combines the best of both RAR and GC. That is, it parallelizes the communications over a tree topology leading to efficient bandwidth utilization, and carefully designs a redundant data set allocation and coding strategy at the nodes to make the proposed gradient aggregation scheme robust to stragglers. In particular, we quantify the communication parallelization gain and resiliency of the proposed CR scheme, and prove its optimality when the communication topology is a regular tree. Moreover, we characterize the expected run-time of CR and show order-wise speedups compared to the benchmark schemes. Finally, we empirically evaluate the performance of our proposed CR design over Amazon EC2 and demonstrate that it achieves speedups of up to 27.2x and 7.0x, respectively over the benchmarks GC and RAR.

stat.ML↗

arXiv2020

Coded Computing for Distributed Graph Analytics

Performance of distributed graph processing systems significantly suffers from 'communication bottleneck' as a large number of messages are exchanged among servers at each step of the computation. Motivated by graph based MapReduce, we propose a coded computing framework that leverages computation redundancy to alleviate the communication bottleneck in distributed graph processing. We develop a novel 'coding' scheme that systematically injects structured redundancy in computation phase to enable 'coded' multicasting opportunities during message exchange between servers, reducing communication load substantially in large-scale graph processing. For theoretical analysis, we consider random graph models, and prove that our proposed scheme enables an (asymptotically) inverse-linear trade-off between 'computation load' and 'average communication load' for two popular random graph models -- Erdos-Renyi model, and power law model. Particularly, for a given computation load r, (i.e. when each graph vertex is carefully stored at r servers), the proposed scheme slashes the average communication load by (nearly) a multiplicative factor of r. For the Erdos-Renyi model, our proposed scheme is optimal asymptotically as the graph size increases by providing an information-theoretic converse. To illustrate the benefits of our scheme in practice, we implement PageRank over Amazon EC2, using artificial as well as real-world datasets, demonstrating significant gains over conventional PageRank. We also specialize our scheme and extend our theoretical results to two other random graph models -- random bi-partite model, and stochastic block model. They asymptotically enable inverse-linear trade-offs between computation and communication loads in distributed graph processing for these popular random graph models as well. We complement the achievability results with converse bounds for both of these models.

cs.DC↗

arXiv2019

Coded Computation over Heterogeneous Clusters

In large-scale distributed computing clusters, such as Amazon EC2, there are several types of "system noise" that can result in major degradation of performance: bottlenecks due to limited communication bandwidth, latency due to straggler nodes, etc. On the other hand, these systems enjoy abundance of redundancy - a vast number of computing nodes and large storage capacity. There have been recent results that demonstrate the impact of coding for efficient utilization of computation and storage redundancy to alleviate the effect of stragglers and communication bottlenecks in homogeneous clusters. In this paper, we focus on general heterogeneous distributed computing clusters consisting of a variety of computing machines with different capabilities. We propose a coding framework for speeding up distributed computing in heterogeneous clusters by trading redundancy for reducing the latency of computation. In particular, we propose Heterogeneous Coded Matrix Multiplication (HCMM) algorithm for performing distributed matrix multiplication over heterogeneous clusters that is provably asymptotically optimal for a broad class of processing time distributions. Moreover, we show that HCMM is unboundedly faster than any uncoded scheme. To demonstrate practicality of HCMM, we carry out experiments over Amazon EC2 clusters where HCMM is found to be up to $61\%$, $46\%$ and $36\%$ respectively faster than three benchmark load allocation schemes - Uniform Uncoded, Load-balanced Uncoded, and Uniform Coded. Additionally, we provide a generalization to the problem of optimal load allocation in heterogeneous settings, where we take into account the monetary costs associated with the clusters. We argue that HCMM is asymptotically optimal for budget-constrained scenarios as well, and we develop a heuristic algorithm for (HCMM) load allocation for budget-limited computation tasks.

cs.DC↗

arXiv2017

Binary Fading Interference Channel with No CSIT

We study the capacity region of the two-user Binary Fading (or Erasure) Interference Channel where the transmitters have no knowledge of the channel state information. We develop new inner-bounds and outer-bounds for this problem. We identify three regimes based on the channel parameters: weak, moderate, and strong interference regimes. Interestingly, this is similar to the generalized degrees of freedom of the two-user Gaussian interference channel where transmitters have perfect channel knowledge. We show that for the weak interference regime, treating interference as erasure is optimal while for the strong interference regime, decoding interference is optimal. For the moderate interference regime, we provide new inner and outer bounds. The inner-bound is based on a modification of the Han-Kobayashi scheme for the erasure channel, enhanced by time-sharing. We study the gap between our inner-bound and our outer-bounds for the moderate interference regime and compare our results to that of the Gaussian interference channel. Deriving our new outer-bounds has three main steps. We first create a contracted channel that has fewer states compared to the original channel, in order to make the analysis tractable. We then prove the Correlation Lemma that shows an outer-bound on the capacity region of the contracted channel also serves as an outer-bound for the original channel. Finally using the Conditional Entropy Leakage Lemma, we derive our outer-bound on the capacity region of the contracted channel.

cs.IT↗

arXiv2016

Approximate Capacity Region of the Two-User MISO Broadcast Channels with Delayed CSIT

We consider the problem of multiple-input single-output Broadcast Channels with Rayleigh fading where the transmitter has access to delayed knowledge of the channel state information. We first characterize the capacity region of this channel with two users to within constant number of bits for all values of the transmit power. The proposed signaling strategy utilizes the delayed knowledge of the channel state information and the previously transmitted signals, in order to create a signal of common interest for both receivers. This signal would be the quantized version of the summation of the previously transmitted signals. A challenge that arises in deriving the result for finite signal-to-noise ratio regimes is the correlation that exists between the quantization noise and the signal. To guarantee the independence of quantization noise and signal, we extend the framework of lattice quantizers with dither together with an interleaving step. For converse, we use the fact that the capacity region of this problem is upper-bounded by the capacity region of a physically degraded broadcast channel with no channel state information where one receiver has two antennas. Then, we derive an outer-bound on the capacity region of this degraded broadcast channel. Finally, we show how to extend our results to obtain the approximate capacity of the $K$-user multiple-input single-output Broadcast Channel with delayed knowledge of the channel state information at the transmitter to within $2 \log_2 \left( K + 2 \right)$ bits/s/Hz.

cs.IT↗

arXiv2016

Topological Interference Management with Reconfigurable Antennas

We study the symmetric degrees-of-freedom (DoF) of partially connected interference networks under linear coding strategies at transmitters without channel state information beyond topology. We assume that the receivers are equipped with reconfigurable antennas that can switch among their preset modes. In such a network setting, we characterize the class of network topologies in which half linear symmetric DoF is achievable. Moreover, we derive a general upper bound on the linear symmetric DoF for arbitrary network topologies. We also show that this upper bound is tight if the transmitters have at most two co-interferers.

cs.IT↗

arXiv2016

Blind MIMOME Wiretap Channel with Delayed CSIT

We study the Gaussian MIMOME wiretap channel where a transmitter wishes to communicate a confidential message to a legitimate receiver in the presence of eavesdroppers, while the eavesdroppers should not be able to decode the confidential message. Each node in the network is equipped with arbitrary number of antennas. Furthermore, channels are time varying, and there is no channel state information available at the transmitter (CSIT) with respect to eavesdroppers' channels; and transmitter only has access to delayed CSIT of the channel to the legitimate receiver. The secure degrees of freedom (SDoF) for such network has only been characterized for special cases, and is unknown in general. We completely characterize the SDoF of this network for all antenna configurations. In particular, we strictly improve the state-of-the-art achievable scheme for this network by proposing more efficient artificial noise alignment at the receivers. Furthermore, we develop a tight upper bound by utilizing 4 important inequalities that provide lower bounds on the received signal dimensions at receivers which supply delayed CSIT or no CSIT, or at a collection of receivers where some supply no CSIT. These inequalities together allow for analysis of signal dimensions in networks with asymmetric CSIT; and as a result, we present a converse proof that leads to characterization of SDoF for all possible antenna configurations.

cs.IT↗

arXiv2014

Capacity Results for Binary Fading Interference Channels with Delayed CSIT

To study the effect of lack of up-to-date channel state information at the transmitters (CSIT), we consider two-user binary fading interference channels with Delayed-CSIT. We characterize the capacity region for such channels under homogeneous assumption where channel gains have identical and independent distributions across time and space, eliminating the possibility of exploiting time/space correlation. We introduce and discuss several novel coding opportunities created by outdated CSIT that can enlarge the achievable rate region. The capacity-achieving scheme relies on accurate combination, concatenation, and merging of these opportunities, depending on the channel statistics. The outer-bounds are based on an extremal inequality we develop for a binary broadcast channel with Delayed-CSIT. We further extend the results and characterize the capacity region when output feedback links are available from the receivers to the transmitters in addition to the delayed knowledge of the channel state information. We also discuss the extension of our results to the non-homogeneous setting.

cs.IT↗

arXiv2012

Interference Channels with Rate-Limited Feedback

We consider the two-user interference channel with rate-limited feedback. Related prior works focus on the case where feedback links have infinite capacity, while no research has been done for the rate-limited feedback problem. Several new challenges arise due to the capacity limitations of the feedback links, both in deriving inner-bounds and outer-bounds. We study this problem under three different interference models: the El Gamal-Costa deterministic model, the linear deterministic model, and the Gaussian model. For the first two models, we develop an achievable scheme that employs three techniques: Han-Kobayashi message splitting, quantize-and-binning, and decode-and-forward. We also derive new outer-bounds for all three models and we show the optimality of our scheme under the linear deterministic model. In the Gaussian case, we propose a transmission strategy that incorporates lattice codes, inspired by the ideas developed in the first two models. For symmetric channel gains, we prove that the gap between the achievable sum-rate of the proposed scheme and our new outer-bounds is bounded by a constant number of bits, independent of the channel gains.

cs.IT↗

arXiv2010

Bidirectional multi-pair network with a MIMO relay: Beamforming strategies and lack of duality

We address the problem of a multi-user relay network, where multiple single-antenna node pairs want to exchange information by using a multiple antenna relay node. Due to the half-duplex constraint of the relay, the exchange of information takes place in two steps. In the first step, the nodes transmit their data to the relay, while in the second step, the relay is broadcasting the data by using linear and non-linear precoding strategies. We focus on the second step in this paper. We first consider the problem of maximizing the overall rate achievable using linear and dirty-paper type precoding strategies at the relay. Then, we consider minimizing the total power at the relay subject to individual SINR constraints using the same strategies at the relay. We show that the downlink-uplink duality does not hold for the setup considered here, which is a somewhat surprising result. We also show that the beamforming strategy which is optimal in the single-pair case performs very well in the multi-pair case for practically relevant SNR. The results are illustrated by numerical simulations.

cs.IT↗

arXiv2010

Network error correction with unequal link capacities

This paper studies the capacity of single-source single-sink noiseless networks under adversarial or arbitrary errors on no more than z edges. Unlike prior papers, which assume equal capacities on all links, arbitrary link capacities are considered. Results include new upper bounds, network error correction coding strategies, and examples of network families where our bounds are tight. An example is provided of a network where the capacity is 50% greater than the best rate that can be achieved with linear coding. While coding at the source and sink suffices in networks with equal link capacities, in networks with unequal link capacities, it is shown that intermediate nodes may have to do coding, nonlinear error detection, or error correction in order to achieve the network error correction capacity.

cs.IT↗

arXiv2010

On the Capacity of Multi-Hop Wireless Networks with Partial Network Knowledge

In large wireless networks, acquiring full network state information is typically infeasible. Hence, nodes need to flow the information and manage the interference based on partial information about the network. In this paper, we consider multi-hop wireless networks and assume that each source only knows the channel gains that are on the routes from itself to other destinations in the network. We develop several distributed strategies to manage the interference among the users and prove their optimality in maximizing the achievable normalized sum-rate for some classes of networks.

cs.IT↗

arXiv2008

Diversity-Multiplexing Tradeoff of the Half-Duplex Relay Channel

We show that the diversity-multiplexing tradeoff of a half-duplex single-relay channel with identically distributed Rayleigh fading channel gains meets the 2 by 1 MISO bound. We generalize the result to the case when there are N non-interfering relays and show that the diversity-multiplexing tradeoff is equal to the N + 1 by 1 MISO bound.

cs.IT↗

arXiv2008

Approximate capacity of the two-way relay channel: A deterministic approach

We study the capacity of the full-duplex bidirectional (or two-way) relay channel with two nodes and one relay. The channels in the forward direction are assumed to be different (in general) than the channels in the backward direction, i.e. channel reciprocity is not assumed. We use the recently proposed deterministic approach to capture the essence of the problem and to determine a good transmission and relay strategy for the Gaussian channel. Depending on the ratio of the individual channel gains, we propose to use either a simple amplify-and-forward or a particular superposition coding strategy at the relay. We analyze the achievable rate region and show that the scheme achieves to within 3 bits the cut-set bound for all values of channel gains.

cs.IT↗

arXiv2008

Approximate Capacity of Gaussian Relay Networks

We present an achievable rate for general Gaussian relay networks. We show that the achievable rate is within a constant number of bits from the information-theoretic cut-set upper bound on the capacity of these networks. This constant depends on the topology of the network, but not the values of the channel gains. Therefore, we uniformly characterize the capacity of Gaussian relay networks within a constant number of bits, for all channel parameters.

cs.IT↗