arXiv subjects

Stark C. Draper

Publications and source records attributed to Stark C. Draper.

At least 37 records · Page 2Linked to original sources

arXiv2020

Anytime MiniBatch: Exploiting Stragglers in Online Distributed Optimization

Distributed optimization is vital in solving large-scale machine learning problems. A widely-shared feature of distributed optimization techniques is the requirement that all nodes complete their assigned tasks in each computational epoch before the system can proceed to the next epoch. In such settings, slow nodes, called stragglers, can greatly slow progress. To mitigate the impact of stragglers, we propose an online distributed optimization method called Anytime Minibatch. In this approach, all nodes are given a fixed time to compute the gradients of as many data samples as possible. The result is a variable per-node minibatch size. Workers then get a fixed communication time to average their minibatch gradients via several rounds of consensus, which are then used to update primal variables via dual averaging. Anytime Minibatch prevents stragglers from holding up the system without wasting the work that stragglers can complete. We present a convergence analysis and analyze the wall time performance. Our numerical results show that our approach is up to 1.5 times faster in Amazon EC2 and it is up to five times faster when there is greater variability in compute node performance.

cs.LG↗

arXiv2019

Hierarchical Coded Matrix Multiplication

Slow working nodes, known as stragglers, can greatly reduce the speed of distributed computation. Coded matrix multiplication is a recently introduced technique that enables straggler-resistant distributed multiplication of large matrices. A key property is that the finishing time depends only on the work completed by a set of the fastest workers, while the work done by the slowest workers is ignored completely. This paper is motivated by the observation that in real-world commercial cloud computing systems such as Amazon's Elastic Compute Cloud (EC2) the distinction between fast and slow nodes is often a soft one. Thus, if we could also exploit the work completed by stragglers we may realize substantial performance gains. To realize such gains, in this paper we use the idea of hierarchical coding (Ferdinand and Draper, IEEE Int. Symp. Inf. Theory, 2018). We decompose the overall matrix multiplication task into a hierarchy of heterogeneously sized subtasks. The duty to complete each subtask is shared amongst all workers and each subtask is (generally) of a different complexity. The motivation for the hierarchical decomposition is the recognition that more workers will finish the first subtask than the second (or third, forth, etc.). Connecting to error correction coding, earlier subtasks can therefore be designed to be of a higher rate than later subtasks. Through this hierarchical design our scheme exploits the work completed by stragglers, rather than ignoring it, even if that amount is much less than that completed by the fastest workers. We numerically show that our method realizes a 60% improvement in the expected finishing time for a widely studied statistical model of the speed of computation and, on Amazon EC2, the gain is 35%.

cs.IT↗

arXiv2019

Cuboid Partitioning for Hierarchical Coded Matrix Multiplication

Coded matrix multiplication is a technique to enable straggler-resistant multiplication of large matrices in distributed computing systems. In this paper, we first present a conceptual framework to represent the division of work amongst processors in coded matrix multiplication as a cuboid partitioning problem. This framework allows us to unify existing methods and motivates new techniques. Building on this framework, we apply the idea of hierarchical coding (Ferdinand & Draper, 2018) to coded matrix multiplication. The hierarchical scheme we develop is able to exploit the work completed by all processors (fast and slow), rather than ignoring the slow ones, even if the amount of work completed by stragglers is much less than that completed by the fastest workers. On Amazon EC2, we achieve a 37% improvement in average finishing time compared to non-hierarchical schemes.

cs.IT↗

arXiv2018

Efficient learning of neighbor representations for boundary trees and forests

We introduce a semiparametric approach to neighbor-based classification. We build off the recently proposed Boundary Trees algorithm by Mathy et al.(2015) which enables fast neighbor-based classification, regression and retrieval in large datasets. While boundary trees use an Euclidean measure of similarity, the Differentiable Boundary Tree algorithm by Zoran et al.(2017) was introduced to learn low-dimensional representations of complex input data, on which semantic similarity can be calculated to train boundary trees. As is pointed out by its authors, the differentiable boundary tree approach contains a few limitations that prevents it from scaling to large datasets. In this paper, we introduce Differentiable Boundary Sets, an algorithm that overcomes the computational issues of the differentiable boundary tree scheme and also improves its classification accuracy and data representability. Our algorithm is efficiently implementable with existing tools and offers a significant reduction in training time. We test and compare the algorithms on the well known MNIST handwritten digits dataset and the newer Fashion-MNIST dataset by Xiao et al.(2017).

cs.LG↗

arXiv2018

Exploitation of Stragglers in Coded Computation

In cloud computing systems slow processing nodes, often referred to as "stragglers", can significantly extend the computation time. Recent results have shown that error correction coding can be used to reduce the effect of stragglers. In this work we introduce a scheme that, in addition to using error correction to distribute mixed jobs across nodes, is also able to exploit the work completed by all nodes, including stragglers. We first consider vector-matrix multiplication and apply maximum distance separable (MDS) codes to small blocks of sub-matrices. The worker nodes process blocks sequentially, working block-by-block, transmitting partial per-block results to the master as they are completed. Sub-blocking allows a more continuous completion process, which thereby allows us to exploit the work of a much broader spectrum of processors and reduces computation time. We then apply this technique to matrix-matrix multiplication using product code. In this case, we show that the order of computing sub-tasks is a new degree of design freedom that can be exploited to reduce computation time further. We propose a novel approach to analyze the finishing time, which is different from typical order statistics. Simulation results show that the expected computation time decreases by a factor of at least two in compared to previous methods.

cs.IT↗

arXiv2016

Hardware-Based Linear Program Decoding with the Alternating Direction Method of Multipliers

We present a hardware-based implementation of Linear Program (LP) decoding for binary linear codes. LP decoding frames error-correction as an optimization problem. In contrast, variants of Belief Propagation (BP) decoding frame error-correction as a problem of graphical inference. LP decoding has several advantages over BP-based methods, including convergence guarantees and better error-rate performance in high-reliability channels. The latter makes LP decoding attractive for optical transport and storage applications. However, LP decoding, when implemented with general solvers, does not scale to large blocklengths and is not suitable for a parallelized implementation in hardware. It has been recently shown that the Alternating Direction Method of Multipliers (ADMM) can be applied to decompose the LP decoding problem. The result is a message-passing algorithm with a structure very similar to BP. We present new intuition for this decoding algorithm as well as for its major computational primitive: projection onto the parity polytope. Furthermore, we present results for a fixed-point Verilog implementation of ADMM-LP decoding. This implementation targets a Field-Programmable Gate Array (FPGA) platform to evaluate error-rate performance and estimate resource usage. We show that Frame Error Rate (FER) performance well within 0.5dB of double-precision implementations is possible with 10-bit messages. Finally, we outline a number of research opportunities that should be explored en-route to the realization of an Application Specific Integrated Circuit (ASIC) implementation capable of gigabit per second throughput.

cs.IT↗

arXiv2016

Hardware-Based ADMM-LP Decoding

In this paper we present an FPGA-based implementation of linear programming (LP) decoding. LP decoding frames error correction as an optimization problem. This is in contrast to variants of belief propagation (BP) decoding that view error correction as a problem of graphical inference. There are many advantages to taking the optimization perspective: convergence guarantees, improved performance in certain regimes, and a methodology for incorporating the latest developments in optimization techniques. However, LP decoding, when implemented with standard LP solvers, does not easily scale to the blocklengths of modern error-correction codes. In earlier work, we showed that by drawing on decomposition methods from optimization theory, specifically the alternating direction method of multipliers (ADMM), we could build an LP decoding solver that was competitive with BP, both in terms of performance and speed. We also observed empirically that LP decoders have much better high-SNR performance in the "error floor" regime, a trait of particular relevance to optical transport and storage applications. While our previous implementation was in floating point, in this paper we report initial results of a fixed-point, hardware-based realization of our ADMM-LP decoder.

cs.IT↗

arXiv2016

Hardware Based Projection onto The Parity Polytope and Probability Simplex

This paper is concerned with the adaptation to hardware of methods for Euclidean norm projections onto the parity polytope and probability simplex. We first refine recent efforts to develop efficient methods of projection onto the parity polytope. Our resulting algorithm can be configured to have either average computational complexity $\mathcal{O}\left(d\right)$ or worst case complexity $\mathcal{O}\left(d\log{d}\right)$ on a serial processor where $d$ is the dimension of projection space. We show how to adapt our projection routine to hardware. Our projection method uses a sub-routine that involves another Euclidean projection; onto the probability simplex. We therefore explain how to adapt to hardware a well know simplex projection algorithm. The hardware implementations of both projection algorithms achieve area scalings of $\mathcal{O}(d\left(\log{d}\right)^2)$ at a delay of $\mathcal{O}(\left(\log{d}\right)^2)$. Finally, we present numerical results in which we evaluate the fixed-point accuracy and resource scaling of these algorithms when targeting a modern FPGA.

cs.IT↗

arXiv2016

FastCap: An Efficient and Fair Algorithm for Power Capping in Many-Core Systems

Future servers will incorporate many active lowpower modes for different system components, such as cores and memory. Though these modes provide flexibility for power management via Dynamic Voltage and Frequency Scaling (DVFS), they must be operated in a coordinated manner. Such coordinated control creates a combinatorial space of possible power mode configurations. Given the rapid growth of the number of cores, it is becoming increasingly challenging to quickly select the configuration that maximizes the performance under a given power budget. Prior power capping techniques do not scale well to large numbers of cores, and none of those works has considered memory DVFS. In this paper, we present FastCap, our optimization approach for system-wide power capping, using both CPU and memory DVFS. Based on a queuing model, FastCap formulates power capping as a non-linear optimization problem where we seek to maximize the system performance under a power budget, while promoting fairness across applications. Our FastCap algorithm solves the optimization online and efficiently (low complexity on the number of cores), using a small set of performance counters as input. To evaluate FastCap, we simulate it for a many-core server running different types of workloads. Our results show that FastCap caps power draw accurately, while producing better application performance and fairness than many existing CPU power capping methods (even after they are extended to use of memory DVFS as well).

cs.PF↗

arXiv2015

ADMM LP decoding of non-binary LDPC codes in $\mathbb{F}_{2^m}$

In this paper, we develop efficient decoders for non-binary low-density parity-check (LDPC) codes using the alternating direction method of multipliers (ADMM). We apply ADMM to two decoding problems. The first problem is linear programming (LP) decoding. In order to develop an efficient algorithm, we focus on non-binary codes in fields of characteristic two. This allows us to transform each constraint in $\mathbb{F}_{2^m}$ to a set of constraints in $\mathbb{F}_{2}$ that has a factor graph representation. Applying ADMM to the LP decoding problem results in two types of non-trivial sub-routines. The first type requires us to solve an unconstrained quadratic program. We solve this problem efficiently by leveraging new results obtained from studying the above factor graphs. The second type requires Euclidean projection onto polytopes that are studied in the literature, a projection that can be solved efficiently using off-the-shelf techniques, which scale linearly in the dimension of the vector to project. ADMM LP decoding scales linearly with block length, linearly with check degree, and quadratically with field size. The second problem we consider is a penalized LP decoding problem. This problem is obtained by incorporating a penalty term into the LP decoding objective. The purpose of the penalty term is to make non-integer solutions (pseudocodewords) more expensive and hence to improve decoding performance. The ADMM algorithm for the penalized LP problem requires Euclidean projection onto a polytope formed by embedding the constraints specified by the non-binary single parity-check code, which can be solved by applying the ADMM technique to the resulting quadratic program. Empirically, this decoder achieves a much reduced error rate than LP decoding at low signal-to-noise ratios.

cs.IT↗

arXiv2015

LP-decodable multipermutation codes

In this paper, we introduce a new way of constructing and decoding multipermutation codes. Multipermutations are permutations of a multiset that generally consist of duplicate entries. We first introduce a class of binary matrices called multipermutation matrices, each of which corresponds to a unique and distinct multipermutation. By enforcing a set of linear constraints on these matrices, we define a new class of codes that we term LP-decodable multipermutation codes. In order to decode these codes using a linear program (LP), thereby enabling soft decoding, we characterize the convex hull of multipermutation matrices. This characterization allows us to relax the coding constraints to a polytope and to derive two LP decoding problems. These two problems are respectively formulated by relaxing the maximum likelihood decoding problem and the minimum Chebyshev distance decoding problem. Because these codes are non-linear, we also study efficient encoding and decoding algorithms. We first describe an algorithm that maps consecutive integers, one by one, to an ordered list of multipermutations. Based on this algorithm, we develop an encoding algorithm for a code proposed by Shieh and Tsai, a code that falls into our class of LP-decodable multipermutation codes. Regarding decoding algorithms, we propose an efficient distributed decoding algorithm based on the alternating direction method of multipliers (ADMM). Finally, we observe from simulation results that the soft decoding techniques we introduce can significantly outperform hard decoding techniques that are based on quantized channel outputs.

cs.IT↗

arXiv2014

LP-decodable multipermutation codes

In this paper, we introduce a new way of constructing and decoding multipermutation codes. Multipermutations are permutations of a multiset that may consist of duplicate entries. We first introduce a new class of matrices called multipermutation matrices. We characterize the convex hull of multipermutation matrices. Based on this characterization, we propose a new class of codes that we term LP-decodable multipermutation codes. Then, we derive two LP decoding algorithms. We first formulate an LP decoding problem for memoryless channels. We then derive an LP algorithm that minimizes the Chebyshev distance. Finally, we show a numerical example of our algorithm.

cs.IT↗

arXiv2014

The ADMM penalized decoder for LDPC codes

Linear programming (LP) decoding for low-density parity-check (LDPC) codes proposed by Feldman et al. is shown to have theoretical guarantees in several regimes and empirically is not observed to suffer from an error floor. However at low signal-to-noise ratios (SNRs), LP decoding is observed to have worse error performance than belief propagation (BP) decoding. In this paper, we seek to improve LP decoding at low SNRs while still achieving good high SNR performance. We first present a new decoding framework obtained by trying to solve a non-convex optimization problem using the alternating direction method of multipliers (ADMM). This non-convex problem is constructed by adding a penalty term to the LP decoding objective. The goal of the penalty term is to make "pseudocodewords", which are the non-integer vertices of the LP relaxation to which the LP decoder fails, more costly. We name this decoder class the "ADMM penalized decoder". In our simulation results, the ADMM penalized decoder with $\ell_1$ and $\ell_2$ penalties outperforms both BP and LP decoding at all SNRs. For high SNR regimes where it is infeasible to simulate, we use an instanton analysis and show that the ADMM penalized decoder has better high SNR performance than BP decoding. We also develop a reweighted LP decoder using linear approximations to the objective with an $\ell_1$ penalty. We show that this decoder has an improved theoretical recovery threshold compared to LP decoding. In addition, we show that the empirical gain of the reweighted LP decoder is significant at low SNRs.

cs.IT↗

arXiv2014

Unequal Message Protection: Asymptotic and Non-Asymptotic Tradeoffs

We study a form of unequal error protection that we term "unequal message protection" (UMP). The message set of a UMP code is a union of $m$ disjoint message classes. Each class has its own error protection requirement, with some classes needing better error protection than others. We analyze the tradeoff between rates of message classes and the levels of error protection of these codes. We demonstrate that there is a clear performance loss compared to homogeneous (classical) codes with equivalent parameters. This is in sharp contrast to previous literature that considers UMP codes. To obtain our results we generalize finite block length achievability and converse bounds due to Polyanskiy-Poor-Verdú. We evaluate our bounds for the binary symmetric and binary erasure channels, and analyze the asymptotic characteristic of the bounds in the fixed error and moderate deviations regimes. In addition, we consider two questions related to the practical construction of UMP codes. First, we study a "header" construction that prefixes the message class into a header followed by data protection using a standard homogeneous code. We show that, in general, this construction is not optimal at finite block lengths. We further demonstrate that our main UMP achievability bound can be obtained using coset codes, which suggests a path to implementation of tractable UMP codes.

cs.IT↗

arXiv2014

SleepScale: Runtime Joint Speed Scaling and Sleep States Management for Power Efficient Data Centers

Power consumption in data centers has been growing significantly in recent years. To reduce power, servers are being equipped with increasingly sophisticated power management mechanisms. Different mechanisms offer dramatically different trade-offs between power savings and performance penalties. Considering the complexity, variety, and temporally varying nature of the applications hosted in a typical data center, intelligently determining which power management policy to use and when is a complicated task. In this paper we analyze a system model featuring both performance scaling and low-power states. We reveal the interplay between performance scaling and low-power states via intensive simulation and analytic verification. Based on the observations, we present SleepScale, a runtime power management tool designed to efficiently exploit existing power control mechanisms. At run time, SleepScale characterizes power consumption and quality-of-service (QoS) for each low-power state and frequency setting, and selects the best policy for a given QoS constraint. We evaluate SleepScale using workload traces from data centers and achieve significant power savings relative to conventional power management strategies.

cs.PF↗

arXiv2013

The Sender-Excited Secret Key Agreement Model: Capacity, Reliability and Secrecy Exponents

We consider the secret key generation problem when sources are randomly excited by the sender and there is a noiseless public discussion channel. Our setting is thus similar to recent works on channels with action-dependent states where the channel state may be influenced by some of the parties involved. We derive single-letter expressions for the secret key capacity through a type of source emulation analysis. We also derive lower bounds on the achievable reliability and secrecy exponents, i.e., the exponential rates of decay of the probability of decoding error and of the information leakage. These exponents allow us to determine a set of strongly-achievable secret key rates. For degraded eavesdroppers the maximum strongly-achievable rate equals the secret key capacity; our exponents can also be specialized to previously known results. In deriving our strong achievability results we introduce a coding scheme that combines wiretap coding (to excite the channel) and key extraction (to distill keys from residual randomness). The secret key capacity is naturally seen to be a combination of both source- and channel-type randomness. Through examples we illustrate a fundamental interplay between the portion of the secret key rate due to each type of randomness. We also illustrate inherent tradeoffs between the achievable reliability and secrecy exponents. Our new scheme also naturally accommodates rate limits on the public discussion. We show that under rate constraints we are able to achieve larger rates than those that can be attained through a pure source emulation strategy.

cs.IT↗

arXiv2013

Decomposition Methods for Large Scale LP Decoding

When binary linear error-correcting codes are used over symmetric channels, a relaxed version of the maximum likelihood decoding problem can be stated as a linear program (LP). This LP decoder can be used to decode error-correcting codes at bit-error-rates comparable to state-of-the-art belief propagation (BP) decoders, but with significantly stronger theoretical guarantees. However, LP decoding when implemented with standard LP solvers does not easily scale to the block lengths of modern error correcting codes. In this paper we draw on decomposition methods from optimization theory, specifically the Alternating Directions Method of Multipliers (ADMM), to develop efficient distributed algorithms for LP decoding. The key enabling technical result is a "two-slice" characterization of the geometry of the parity polytope, which is the convex hull of all codewords of a single parity check code. This new characterization simplifies the representation of points in the polytope. Using this simplification, we develop an efficient algorithm for Euclidean norm projection onto the parity polytope. This projection is required by ADMM and allows us to use LP decoding, with all its theoretical guarantees, to decode large-scale error correcting codes efficiently. We present numerical results for LDPC codes of lengths more than 1000. The waterfall region of LP decoding is seen to initiate at a slightly higher signal-to-noise ratio than for sum-product BP, however an error floor is not observed for LP decoding, which is not the case for BP. Our implementation of LP decoding using ADMM executes as fast as our baseline sum-product BP decoder, is fully parallelizable, and can be seen to implement a type of message-passing with a particularly simple schedule.

cs.IT↗

arXiv2013

Queuing Theoretic Analysis of Power-performance Tradeoff in Power-efficient Computing

In this paper we study the power-performance relationship of power-efficient computing from a queuing theoretic perspective. We investigate the interplay of several system operations including processing speed, system on/off decisions, and server farm size. We identify that there are oftentimes "sweet spots" in power-efficient operations: there exist optimal combinations of processing speed and system settings that maximize power efficiency. For the single server case, a widely deployed threshold mechanism is studied. We show that there exist optimal processing speed and threshold value pairs that minimize the power consumption. This holds for the threshold mechanism with job batching. For the multi-server case, it is shown that there exist best processing speed and server farm size combinations.

cs.PF↗