arXiv subjects

Kangwook Lee

Publications and source records attributed to Kangwook Lee.

89 records · Page 5Linked to original sources

arXiv2021

FairBatch: Batch Selection for Model Fairness

Training a fair machine learning model is essential to prevent demographic disparity. Existing techniques for improving model fairness require broad changes in either data preprocessing or model training, rendering themselves difficult-to-adopt for potentially already complex machine learning systems. We address this problem via the lens of bilevel optimization. While keeping the standard training algorithm as an inner optimizer, we incorporate an outer optimizer so as to equip the inner problem with an additional functionality: Adaptively selecting minibatch sizes for the purpose of improving model fairness. Our batch selection algorithm, which we call FairBatch, implements this optimization and supports prominent fairness measures: equal opportunity, equalized odds, and demographic parity. FairBatch comes with a significant implementation benefit -- it does not require any modification to data preprocessing or model training. For instance, a single-line change of PyTorch code for replacing batch selection part of model training suffices to employ FairBatch. Our experiments conducted both on synthetic and benchmark real data demonstrate that FairBatch can provide such functionalities while achieving comparable (or even greater) performances against the state of the arts. Furthermore, FairBatch can readily improve fairness of any pre-trained model simply via fine-tuning. It is also compatible with existing batch selection techniques intended for different purposes, such as faster convergence, thus gracefully achieving multiple purposes.

cs.LG↗

arXiv2020

SLM: Learning a Discourse Language Representation with Sentence Unshuffling

We introduce Sentence-level Language Modeling, a new pre-training objective for learning a discourse language representation in a fully self-supervised manner. Recent pre-training methods in NLP focus on learning either bottom or top-level language representations: contextualized word representations derived from language model objectives at one extreme and a whole sequence representation learned by order classification of two given textual segments at the other. However, these models are not directly encouraged to capture representations of intermediate-size structures that exist in natural languages such as sentences and the relationships among them. To that end, we propose a new approach to encourage learning of a contextualized sentence-level representation by shuffling the sequence of input sentences and training a hierarchical transformer model to reconstruct the original ordering. Through experiments on downstream tasks such as GLUE, SQuAD, and DiscoEval, we show that this feature of our model improves the performance of the original BERT by large margins.

cs.CL↗

arXiv2020

Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification

Distributed model training suffers from communication bottlenecks due to frequent model updates transmitted across compute nodes. To alleviate these bottlenecks, practitioners use gradient compression techniques like sparsification, quantization, or low-rank updates. The techniques usually require choosing a static compression ratio, often requiring users to balance the trade-off between model accuracy and per-iteration speedup. In this work, we show that such performance degradation due to choosing a high compression ratio is not fundamental. An adaptive compression strategy can reduce communication while maintaining final test accuracy. Inspired by recent findings on critical learning regimes, in which small gradient errors can have irrecoverable impact on model performance, we propose Accordion a simple yet effective adaptive compression algorithm. While Accordion maintains a high enough compression rate on average, it avoids over-compressing gradients whenever in critical learning regimes, detected by a simple gradient-norm based criterion. Our extensive experimental study over a number of machine learning tasks in distributed environments indicates that Accordion, maintains similar model accuracy to uncompressed training, yet achieves up to 5.5x better compression and up to 4.1x end-to-end speedup over static approaches. We show that Accordion also works for adjusting the batch size, another popular strategy for alleviating communication bottlenecks.

cs.LG↗

arXiv2020

Attack of the Tails: Yes, You Really Can Backdoor Federated Learning

Due to its decentralized nature, Federated Learning (FL) lends itself to adversarial attacks in the form of backdoors during training. The goal of a backdoor is to corrupt the performance of the trained model on specific sub-tasks (e.g., by classifying green cars as frogs). A range of FL backdoor attacks have been introduced in the literature, but also methods to defend against them, and it is currently an open question whether FL systems can be tailored to be robust against backdoors. In this work, we provide evidence to the contrary. We first establish that, in the general case, robustness to backdoors implies model robustness to adversarial examples, a major open problem in itself. Furthermore, detecting the presence of a backdoor in a FL model is unlikely assuming first order oracles or polynomial time. We couple our theoretical results with a new family of backdoor attacks, which we refer to as edge-case backdoors. An edge-case backdoor forces a model to misclassify on seemingly easy inputs that are however unlikely to be part of the training, or test data, i.e., they live on the tail of the input distribution. We explain how these edge-case backdoors can lead to unsavory failures and may have serious repercussions on fairness, and exhibit that with careful tuning at the side of the adversary, one can insert them across a range of machine learning tasks (e.g., image classification, OCR, text prediction, sentiment analysis).

cs.LG↗

arXiv2020

FR-Train: A Mutual Information-Based Approach to Fair and Robust Training

Trustworthy AI is a critical issue in machine learning where, in addition to training a model that is accurate, one must consider both fair and robust training in the presence of data bias and poisoning. However, the existing model fairness techniques mistakenly view poisoned data as an additional bias to be fixed, resulting in severe performance degradation. To address this problem, we propose FR-Train, which holistically performs fair and robust model training. We provide a mutual information-based interpretation of an existing adversarial training-based fairness-only method, and apply this idea to architect an additional discriminator that can identify poisoned data using a clean validation set and reduce its influence. In our experiments, FR-Train shows almost no decrease in fairness and accuracy in the presence of data poisoning by both mitigating the bias and defending against poisoning. We also demonstrate how to construct clean validation sets using crowdsourcing, and release new benchmark datasets.

cs.LG↗

arXiv2018

Hypergraph Spectral Clustering in the Weighted Stochastic Block Model

Spectral clustering is a celebrated algorithm that partitions objects based on pairwise similarity information. While this approach has been successfully applied to a variety of domains, it comes with limitations. The reason is that there are many other applications in which only \emph{multi}-way similarity measures are available. This motivates us to explore the multi-way measurement setting. In this work, we develop two algorithms intended for such setting: Hypergraph Spectral Clustering (HSC) and Hypergraph Spectral Clustering with Local Refinement (HSCLR). Our main contribution lies in performance analysis of the poly-time algorithms under a random hypergraph model, which we name the weighted stochastic block model, in which objects and multi-way measures are modeled as nodes and weights of hyperedges, respectively. Denoting by $n$ the number of nodes, our analysis reveals the following: (1) HSC outputs a partition which is better than a random guess if the sum of edge weights (to be explained later) is $Ω(n)$; (2) HSC outputs a partition which coincides with the hidden partition except for a vanishing fraction of nodes if the sum of edge weights is $ω(n)$; and (3) HSCLR exactly recovers the hidden partition if the sum of edge weights is on the order of $n \log n$. Our results improve upon the state of the arts recently established under the model and they firstly settle the order-wise optimal results for the binary edge weight case. Moreover, we show that our results lead to efficient sketching algorithms for subspace clustering, a computer vision application. Lastly, we show that HSCLR achieves the information-theoretic limits for a special yet practically relevant model, thereby showing no computational barrier for the case.

math.ST↗

arXiv2018

Speeding Up Distributed Machine Learning Using Codes

Codes are widely used in many engineering applications to offer robustness against noise. In large-scale systems there are several types of noise that can affect the performance of distributed machine learning algorithms -- straggler nodes, system failures, or communication bottlenecks -- but there has been little interaction cutting across codes, machine learning, and distributed systems. In this work, we provide theoretical insights on how coded solutions can achieve significant gains compared to uncoded ones. We focus on two of the most basic building blocks of distributed learning algorithms: matrix multiplication and data shuffling. For matrix multiplication, we use codes to alleviate the effect of stragglers, and show that if the number of homogeneous workers is $n$, and the runtime of each subtask has an exponential tail, coded computation can speed up distributed matrix multiplication by a factor of $\log n$. For data shuffling, we use codes to reduce communication bottlenecks, exploiting the excess in storage. We show that when a constant fraction $α$ of the data matrix can be cached at each worker, and $n$ is the number of workers, \emph{coded shuffling} reduces the communication cost by a factor of $(α+ \frac{1}{n})γ(n)$ compared to uncoded shuffling, where $γ(n)$ is the ratio of the cost of unicasting $n$ messages to $n$ users to multicasting a common message (of the same size) to $n$ users. For instance, $γ(n) \simeq n$ if multicasting a message to $n$ users is as cheap as unicasting a message to one user. We also provide experiment results, corroborating our theoretical gains of the coded algorithms.

cs.DC↗

arXiv2018

Hierarchical Coding for Distributed Computing

Coding for distributed computing supports low-latency computation by relieving the burden of straggling workers. While most existing works assume a simple master-worker model, we consider a hierarchical computational structure consisting of groups of workers, motivated by the need to reflect the architectures of real-world distributed computing systems. In this work, we propose a hierarchical coding scheme for this model, as well as analyze its decoding cost and expected computation time. Specifically, we first provide upper and lower bounds on the expected computing time of the proposed scheme. We also show that our scheme enables efficient parallel decoding, thus reducing decoding costs by orders of magnitude over non-hierarchical schemes. When considering both decoding cost and computing time, the proposed hierarchical coding is shown to outperform existing schemes in many practical scenarios.

cs.DC↗

arXiv2017

Simulating outcomes of interventions using a multipurpose simulation program based on the Evolutionary Causal Matrices and Markov Chain

Predicting long-term outcomes of interventions is necessary for educational and social policy-making processes that might widely influence our society for the long-term. However, performing such predictions based on data from large-scale experiments might be challenging due to the lack of time and resources. In order to address this issue, computer simulations based on Evolutionary Causal Matrices and Markov Chain can be used to predict long-term outcomes with relatively small-scale lab data. In this paper, we introduce Python classes implementing a computer simulation model and presented some pilots implementations demonstrating how the model can be utilized for predicting outcomes of diverse interventions. We also introduce the class-structured simulation module both with real experimental data and with hypothetical data formulated based on social psychological theories. Classes developed and tested in the present study provide researchers and practitioners with a feasible and practical method to simulate intervention outcomes prospectively.

stat.AP↗

arXiv2017

Community Recovery in Hypergraphs

Community recovery is a central problem that arises in a wide variety of applications such as network clustering, motion segmentation, face clustering and protein complex detection. The objective of the problem is to cluster data points into distinct communities based on a set of measurements, each of which is associated with the values of a certain number of data points. While most of the prior works focus on a setting in which the number of data points involved in a measurement is two, this work explores a generalized setting in which the number can be more than two. Motivated by applications particularly in machine learning and channel coding, we consider two types of measurements: (1) homogeneity measurement which indicates whether or not the associated data points belong to the same community; (2) parity measurement which denotes the modulo-2 sum of the values of the data points. Such measurements are possibly corrupted by Bernoulli noise. We characterize the fundamental limits on the number of measurements required to reconstruct the communities for the considered models.

cs.IT↗

arXiv2017

PhaseCode: Fast and Efficient Compressive Phase Retrieval based on Sparse-Graph-Codes

We consider the problem of recovering a $K$-sparse complex signal $x$ from $m$ intensity measurements. We propose the PhaseCode algorithm, and show that in the noiseless case, PhaseCode can recover an arbitrarily-close-to-one fraction of the $K$ non-zero signal components using only slightly more than $4K$ measurements when the support of the signal is uniformly random, with order-optimal time and memory complexity of $Θ(K)$. It is known that the fundamental limit for the number of measurements in compressive phase retrieval problem is $4K - o(K)$ to recover the signal exactly and with no assumptions on its support distribution. This shows that under mild relaxation of the conditions, our algorithm is the first constructive \emph{capacity-approaching} compressive phase retrieval algorithm: in fact, our algorithm is also order-optimal in complexity and memory. Next, motivated by some important practical classes of optical systems, we consider a Fourier-friendly constrained measurement setting, and show that its performance matches that of the unconstrained setting. In the Fourier-friendly setting that we consider, the measurement matrix is constrained to be a cascade of Fourier matrices and diagonal matrices. We further demonstrate how PhaseCode can be robustified to noise. Throughout, we provide extensive simulation results that validate the practical power of our proposed algorithms for the sparse unconstrained and Fourier-friendly measurement settings, for noiseless and noisy scenarios. A key contribution of our work is the novel use of coding-theoretic tools like density evolution methods for the design and analysis of fast and efficient algorithms for compressive phase-retrieval problems.

cs.IT↗

arXiv2016

Predicting Long-term Outcomes of Educational Interventions Using the Evolutionary Causal Matrices and Markov Chain Based on Educational Neuroscience

We developed a prediction model based on the evolutionary causal matrices (ECM) and the Markov Chain to predict long-term influences of educational interventions on adolescents development. Particularly, we created a computational model predicting longitudinal influences of different types of stories of moral exemplars on adolescents voluntary service participation. We tested whether the developed prediction model can properly predict a long-term longitudinal trend of change in voluntary service participation rate by comparing prediction results and surveyed data. Furthermore, we examined which type of intervention would most effectively promote service engagement and what is the minimum required frequency of intervention to produce a large effect. We discussed the implications of the developed prediction model in educational interventions based on educational neuroscience.

stat.AP↗

arXiv2016

Fast and Robust Compressive Phase Retrieval with Sparse-Graph Codes

In this paper, we tackle the compressive phase retrieval problem in the presence of noise. The noisy compressive phase retrieval problem is to recover a $K$-sparse complex signal $s \in \mathbb{C}^n$, from a set of $m$ noisy quadratic measurements: $ y_i=| a_i^H s |^2+w_i$, where $a_i^H\in\mathbb{C}^n$ is the $i$th row of the measurement matrix $A\in\mathbb{C}^{m\times n}$, and $w_i$ is the additive noise to the $i$th measurement. We consider the regime where $K=βn^δ$, with constants $β>0$ and $δ\in(0,1)$. We use the architecture of PhaseCode algorithm, and robustify it using two schemes: the almost-linear scheme and the sublinear scheme. We prove that with high probability, the almost-linear scheme recovers $s$ with sample complexity $Θ(K \log(n))$ and computational complexity $Θ(n \log(n))$, and the sublinear scheme recovers $s$ with sample complexity $Θ(K\log^3(n))$ and computational complexity $Θ(K\log^3(n))$. To the best of our knowledge, this is the first scheme that achieves sublinear computational complexity for compressive phase retrieval problem. Finally, we provide simulation results that support our theoretical contributions.

cs.IT↗

arXiv2015

SAFFRON: A Fast, Efficient, and Robust Framework for Group Testing based on Sparse-Graph Codes

Group testing tackles the problem of identifying a population of $K$ defective items from a set of $n$ items by pooling groups of items efficiently in order to cut down the number of tests needed. The result of a test for a group of items is positive if any of the items in the group is defective and negative otherwise. The goal is to judiciously group subsets of items such that defective items can be reliably recovered using the minimum number of tests, while also having a low-complexity decoding procedure. We describe SAFFRON (Sparse-grAph codes Framework For gROup testiNg), a non-adaptive group testing paradigm that recovers at least a $(1-ε)$-fraction (for any arbitrarily small $ε> 0$) of $K$ defective items with high probability with $m=6C(ε)K\log_2{n}$ tests, where $C(ε)$ is a precisely characterized constant that depends only on $ε$. For instance, it can provably recover at least $(1-10^{-6})K$ defective items with $m \simeq 68 K \log_2{n}$ tests. The computational complexity of the decoding algorithm of SAFFRON is $\mathcal{O}(K\log n)$, which is order-optimal. Further, we robustify SAFFRON such that it can reliably recover the set of $K$ defective items even in the presence of erroneous or noisy test results. We also propose Singleton-Only-SAFFRON, a variant of SAFFRON, that recovers all the $K$ defective items with $m=2e(1+α)K\log K \log_2 n$ tests with probability $1-\mathcal{O}{\left(\frac{1}{K^α}\right)}$, where $α>0$ is a constant. By leveraging powerful design and analysis tools from modern sparse-graph coding theory, SAFFRON is the first approach to reliable, large-scale probabilistic group testing that offers both precisely characterizable number of tests needed (down to the constants) together with order-optimal decoding complexity.

cs.IT↗

arXiv2015

Capacity-Approaching PhaseCode for Low-Complexity Compressive Phase Retrieval

In this paper, we tackle the general compressive phase retrieval problem. The problem is to recover a K-sparse complex vector of length n, $x\in \mathbb{C}^n$, from the magnitudes of m linear measurements, $y=|Ax|$, where $A \in \mathbb{C}^{m \times n}$ can be designed, and the magnitudes are taken component-wise for vector $Ax\in \mathbb{C}^m$. We propose a variant of the PhaseCode algorithm, and show that, using an irregular left-degree sparse-graph code construction, the algorithm can recover almost all the K non-zero signal components using only slightly more than 4K measurements under some mild assumptions, with optimal time and memory complexity of ${\cal O}(K)$. It is known that the fundamental limit for the number of measurements in compressive phase retrieval problem is $4K - o(K)$. To the best of our knowledge, this is the first constructive capacity-approaching compressive phase retrieval algorithm. As a second contribution, we propose another variant of the PhaseCode algorithm that is based on a Compressive Sensing framework involving sparse-graph codes. Our proposed algorithm is an instance of a more powerful "separation" architecture that can be used to address the compressive phase-retrieval problem in general. This modular design features a compressive sensing outer layer, and a trigonometric-based phase-retrieval inner layer. The compressive-sensing layer operates as a linear phase-aware compressive measurement subsystem, while the trig-based phase-retrieval layer provides the desired abstraction between the actually targeted nonlinear phase-retrieval problem and the induced linear compressive-sensing problem. Invoking this architecture based on the use of sparse-graph codes for the compressive sensing layer, we show that we can exactly recover a signal from only the magnitudes of its linear measurements using only slightly more than 6K measurements.

cs.IT↗

arXiv2013

The MDS Queue: Analysing the Latency Performance of Erasure Codes

In order to scale economically, data centers are increasingly evolving their data storage methods from the use of simple data replication to the use of more powerful erasure codes, which provide the same level of reliability as replication but at a significantly lower storage cost. In particular, it is well known that Maximum-Distance-Separable (MDS) codes, such as Reed-Solomon codes, provide the maximum storage efficiency. While the use of codes for providing improved reliability in archival storage systems, where the data is less frequently accessed (or so-called "cold data"), is well understood, the role of codes in the storage of more frequently accessed and active "hot data", where latency is the key metric, is less clear. In this paper, we study data storage systems based on MDS codes through the lens of queueing theory, and term this the "MDS queue." We analytically characterize the (average) latency performance of MDS queues, for which we present insightful scheduling policies that form upper and lower bounds to performance, and are observed to be quite tight. Extensive simulations are also provided and used to validate our theoretical analysis. We also employ the framework of the MDS queue to analyse different methods of performing so-called degraded reads (reading of partial data) in distributed data storage.

cs.IT↗

arXiv2013

When Do Redundant Requests Reduce Latency ?

Several systems possess the flexibility to serve requests in more than one way. For instance, a distributed storage system storing multiple replicas of the data can serve a request from any of the multiple servers that store the requested data, or a computational task may be performed in a compute-cluster by any one of multiple processors. In such systems, the latency of serving the requests may potentially be reduced by sending "redundant requests": a request may be sent to more servers than needed, and it is deemed served when the requisite number of servers complete service. Such a mechanism trades off the possibility of faster execution of at least one copy of the request with the increase in the delay due to an increased load on the system. Due to this tradeoff, it is unclear when redundant requests may actually help. Several recent works empirically evaluate the latency performance of redundant requests in diverse settings. This work aims at an analytical study of the latency performance of redundant requests, with the primary goals of characterizing under what scenarios sending redundant requests will help (and under what scenarios they will not help), as well as designing optimal redundant-requesting policies. We first present a model that captures the key features of such systems. We show that when service times are i.i.d. memoryless or "heavier", and when the additional copies of already-completed jobs can be removed instantly, redundant requests reduce the average latency. On the other hand, when service times are "lighter" or when service times are memoryless and removal of jobs is not instantaneous, then not having any redundancy in the requests is optimal under high loads. Our results hold for arbitrary arrival processes.

cs.NI↗