arXiv subjects

Seyoung Kim

Publications and source records attributed to Seyoung Kim.

At least 19 recordsLinked to original sources

arXiv2022

Neural Network Training with Asymmetric Crosspoint Elements

Analog crossbar arrays comprising programmable nonvolatile resistors are under intense investigation for acceleration of deep neural network training. However, the ubiquitous asymmetric conductance modulation of practical resistive devices critically degrades the classification performance of networks trained with conventional algorithms. Here, we describe and experimentally demonstrate an alternative fully-parallel training algorithm: Stochastic Hamiltonian Descent. Instead of conventionally tuning weights in the direction of the error function gradient, this method programs the network parameters to successfully minimize the total energy (Hamiltonian) of the system that incorporates the effects of device asymmetry. We provide critical intuition on why device asymmetry is fundamentally incompatible with conventional training algorithms and how the new approach exploits it as a useful feature instead. Our technique enables immediate realization of analog deep learning accelerators based on readily available device technologies.

cs.LG↗

arXiv2021

EiGLasso for Scalable Sparse Kronecker-Sum Inverse Covariance Estimation

In many real-world problems, complex dependencies are present both among samples and among features. The Kronecker sum or the Cartesian product of two graphs, each modeling dependencies across features and across samples, has been used as an inverse covariance matrix for a matrix-variate Gaussian distribution, as an alternative to a Kronecker-product inverse covariance matrix, due to its more intuitive sparse structure. However, the existing methods for sparse Kronecker-sum inverse covariance estimation are limited in that they do not scale to more than a few hundred features and samples and that the unidentifiable parameters pose challenges in estimation. In this paper, we introduce EiGLasso, a highly scalable method for sparse Kronecker-sum inverse covariance estimation, based on Newton's method combined with eigendecomposition of the two graphs for exploiting the structure of Kronecker sum. EiGLasso further reduces computation time by approximating the Hessian based on the eigendecomposition of the sample and feature graphs. EiGLasso achieves quadratic convergence with the exact Hessian and linear convergence with the approximate Hessian. We describe a simple new approach to estimating the unidentifiable parameters that generalizes the existing methods. On simulated and real-world data, we demonstrate that EiGLasso achieves two to three orders-of-magnitude speed-up compared to the existing methods.

stat.ML↗

arXiv2021

SEMULATOR: Emulating the Dynamics of Crossbar Array-based Analog Neural System with Regression Neural Networks

As deep neural networks require tremendous amount of computation and memory, analog computing with emerging memory devices is a promising alternative to digital computing for edge devices. However, because of the increasing simulation time for analog computing system, it has not been explored. To overcome this issue, analytically approximated simulators are developed, but these models are inaccurate and narrow down the options for peripheral circuits for multiply-accumulate operation (MAC). In this sense, we propose a methodology, SEMULATOR (SiMULATOR by Emulating the analog computing block) which uses a deep neural network to emulate the behavior of crossbar-based analog computing system. With the proposed neural architecture, we experimentally and theoretically shows that it emulates a MAC unit for neural computation. In addition, the simulation time is incomparably reduced when it compared to the circuit simulators such as SPICE.

cs.LG↗

arXiv2020

Hardware and software co-optimization for the initialization failure of the ReRAM based cross-bar array

Recent advances in deep neural network demand more than millions of parameters to handle and mandate the high-performance computing resources with improved efficiency. The cross-bar array architecture has been considered as one of the promising deep learning architectures that shows a significant computing gain over the conventional processors. To investigate the feasibility of the architecture, we examine non-idealities and their impact on the performance. Specifically, we study the impact of failed cells due to the initialization process of the resistive memory based cross-bar array. Unlike the conventional memory array, individual memory elements cannot be rerouted and, thus, may have a critical impact on model accuracy. We categorize the possible failures and propose hardware implementation that minimizes catastrophic failures. Such hardware optimization bounds the possible logical value of the failed cells and gives us opportunities to compensate for the loss of accuracy via off-line training. By introducing the random weight defects during the training, we show that the model becomes more resilient on the device initialization failures, therefore, less prone to degrade the inference performance due to the failed devices. Our study sheds light on the hardware and software co-optimization procedure to cope with potentially catastrophic failures in the cross-bar array.

cs.ET↗

arXiv2019

Zero-shifting Technique for Deep Neural Network Training on Resistive Cross-point Arrays

A resistive memory device-based computing architecture is one of the promising platforms for energy-efficient Deep Neural Network (DNN) training accelerators. The key technical challenge in realizing such accelerators is to accumulate the gradient information without a bias. Unlike the digital numbers in software which can be assigned and accessed with desired accuracy, numbers stored in resistive memory devices can only be manipulated following the physics of the device, which can significantly limit the training performance. Therefore, additional techniques and algorithm-level remedies are required to achieve the best possible performance in resistive memory device-based accelerators. In this paper, we analyze asymmetric conductance modulation characteristics in RRAM by Soft-bound synapse model and present an in-depth analysis on the relationship between device characteristics and DNN model accuracy using a 3-layer DNN trained on the MNIST dataset. We show that the imbalance between up and down update leads to a poor network performance. We introduce a concept of symmetry point and propose a zero-shifting technique which can compensate imbalance by programming the reference device and changing the zero value point of the weight. By using this zero-shifting method, we show that network performance dramatically improves for imbalanced synapse devices.

cs.ET↗

arXiv2017

Analog CMOS-based Resistive Processing Unit for Deep Neural Network Training

Recently we have shown that an architecture based on resistive processing unit (RPU) devices has potential to achieve significant acceleration in deep neural network (DNN) training compared to today's software-based DNN implementations running on CPU/GPU. However, currently available device candidates based on non-volatile memory technologies do not satisfy all the requirements to realize the RPU concept. Here, we propose an analog CMOS-based RPU design (CMOS RPU) which can store and process data locally and can be operated in a massively parallel manner. We analyze various properties of the CMOS RPU to evaluate the functionality and feasibility for acceleration of DNN training.

cs.ET↗

arXiv2015

Large-Scale Optimization Algorithms for Sparse Conditional Gaussian Graphical Models

This paper addresses the problem of scalable optimization for L1-regularized conditional Gaussian graphical models. Conditional Gaussian graphical models generalize the well-known Gaussian graphical models to conditional distributions to model the output network influenced by conditioning input variables. While highly scalable optimization methods exist for sparse Gaussian graphical model estimation, state-of-the-art methods for conditional Gaussian graphical models are not efficient enough and more importantly, fail due to memory constraints for very large problems. In this paper, we propose a new optimization procedure based on a Newton method that efficiently iterates over two sub-problems, leading to drastic improvement in computation time compared to the previous methods. We then extend our method to scale to large problems under memory constraints, using block coordinate descent to limit memory usage while achieving fast convergence. Using synthetic and genomic data, we show that our methods can solve one million dimensional problems to high accuracy in a little over a day on a single machine.

stat.ML↗

arXiv2012

Tree-guided group lasso for multi-response regression with structured sparsity, with an application to eQTL mapping

We consider the problem of estimating a sparse multi-response regression function, with an application to expression quantitative trait locus (eQTL) mapping, where the goal is to discover genetic variations that influence gene-expression levels. In particular, we investigate a shrinkage technique capable of capturing a given hierarchical structure over the responses, such as a hierarchical clustering tree with leaf nodes for responses and internal nodes for clusters of related responses at multiple granularity, and we seek to leverage this structure to recover covariates relevant to each hierarchically-defined cluster of responses. We propose a tree-guided group lasso, or tree lasso, for estimating such structured sparsity under multi-response regression by employing a novel penalty function constructed from the tree. We describe a systematic weighting scheme for the overlapping groups in the tree-penalty such that each regression coefficient is penalized in a balanced manner despite the inhomogeneous multiplicity of group memberships of the regression coefficients due to overlaps among groups. For efficient optimization, we employ a smoothing proximal gradient method that was originally developed for a general class of structured-sparsity-inducing penalties. Using simulated and yeast data sets, we demonstrate that our method shows a superior performance in terms of both prediction errors and recovery of true sparsity patterns, compared to other methods for learning a multivariate-response regression.

stat.ML↗

arXiv2012

Modeling Waveform Shapes with Random Eects Segmental Hidden Markov Models

In this paper we describe a general probabilistic framework for modeling waveforms such as heartbeats from ECG data. The model is based on segmental hidden Markov models (as used in speech recognition) with the addition of random effects to the generative model. The random effects component of the model handles shape variability across different waveforms within a general class of waveforms of similar shape. We show that this probabilistic model provides a unified framework for learning these models from sets of waveform data as well as parsing, classification, and prediction of new waveforms. We derive a computationally efficient EM algorithm to fit the model on multiple waveforms, and introduce a scoring method that evaluates a test waveform based on its shape. Results on two real-world data sets demonstrate that the random effects methodology leads to improved accuracy (compared to alternative approaches) on classification and segmentation of real-world waveforms.

stat.AP↗

arXiv2012

Smoothing proximal gradient method for general structured sparse regression

We study the problem of estimating high-dimensional regression models regularized by a structured sparsity-inducing penalty that encodes prior structural information on either the input or output variables. We consider two widely adopted types of penalties of this kind as motivating examples: (1) the general overlapping-group-lasso penalty, generalized from the group-lasso penalty; and (2) the graph-guided-fused-lasso penalty, generalized from the fused-lasso penalty. For both types of penalties, due to their nonseparability and nonsmoothness, developing an efficient optimization method remains a challenging problem. In this paper we propose a general optimization approach, the smoothing proximal gradient (SPG) method, which can solve structured sparse regression problems with any smooth convex loss under a wide spectrum of structured sparsity-inducing penalties. Our approach combines a smoothing technique with an effective proximal gradient method. It achieves a convergence rate significantly faster than the standard first-order methods, subgradient methods, and is much more scalable than the most widely used interior-point methods. The efficiency and scalability of our method are demonstrated on both simulation experiments and real genetic data sets.

stat.ML↗

arXiv2012

Feature Selection via Block-Regularized Regression

Identifying co-varying causal elements in very high dimensional feature space with internal structures, e.g., a space with as many as millions of linearly ordered features, as one typically encounters in problems such as whole genome association (WGA) mapping, remains an open problem in statistical learning. We propose a block-regularized regression model for sparse variable selection in a high-dimensional space where the covariates are linearly ordered, and are possibly subject to local statistical linkages (e.g., block structures) due to spacial or temporal proximity of the features. Our goal is to identify a small subset of relevant covariates that are not merely from random positions in the ordering, but grouped as contiguous blocks from large number of ordered covariates. Following a typical linear regression framework between the features and the response, our proposed model employs a sparsity-enforcing Laplacian prior for the regression coefficients, augmented by a 1st-order Markovian process along the feature sequence that "activates" the regression coefficients in a coupled fashion. We describe a sampling-based learning algorithm and demonstrate the performance of our method on simulated and biological data for marker identification under WGA.

stat.ME↗

arXiv2012

Coulomb Drag and Magnetotransport in Graphene Double Layers

We review the fabrication and key transport properties of graphene double layers, consisting of two graphene monolayers placed in close proximity, independently contacted, and separated by an ultra-thin dielectric. We outline a simple band structure model relating the layer densities to the applied gate and inter-layer biases, and show that calculations and experimental results are in excellent agreement both at zero and in high magnetic fields. Coulomb drag measurements, which probe the electron-electron scattering between the two layers reveal two distinct regime: (i) diffusive drag at elevated temperatures, and (ii) mesoscopic fluctuation-dominated drag at low temperatures. We discuss the Coulomb drag results within the framework of existing theories.

cond-mat.mes-hall↗

arXiv2012

Quantum Hall Effect in Bernal Stacked and Twisted Bilayer Graphene Grown on Cu by Chemical Vapor Deposition

We examine the quantum Hall effect in bilayer graphene grown on Cu substrates by chemical vapor deposition. Spatially resolved Raman spectroscopy suggests a mixture of Bernal (A-B) stacked and rotationally faulted (twisted) domains. Magnetotransport measurements performed on bilayer domains with a wide 2D band reveal quantum Hall states (QHSs) at filling factors $ν=4, 8, 12$ consistent with a Bernal stacked bilayer, while magnetotransport measurements in bilayer domains defined by a narrow 2D band show a superposition of QHSs of two independent monolayers. The analysis of the Shubnikov-de Haas oscillations measured in twisted graphene bilayers provides the carrier density in each layer as a function of the gate bias and the inter-layer capacitance.

cond-mat.mes-hall↗

arXiv2012

Smoothing Proximal Gradient Method for General Structured Sparse Learning

We study the problem of learning high dimensional regression models regularized by a structured-sparsity-inducing penalty that encodes prior structural information on either input or output sides. We consider two widely adopted types of such penalties as our motivating examples: 1) overlapping group lasso penalty, based on the l1/l2 mixed-norm penalty, and 2) graph-guided fusion penalty. For both types of penalties, due to their non-separability, developing an efficient optimization method has remained a challenging problem. In this paper, we propose a general optimization approach, called smoothing proximal gradient method, which can solve the structured sparse regression problems with a smooth convex loss and a wide spectrum of structured-sparsity-inducing penalties. Our approach is based on a general smoothing technique of Nesterov. It achieves a convergence rate faster than the standard first-order method, subgradient method, and is much more scalable than the most widely used interior-point method. Numerical results are reported to demonstrate the efficiency and scalability of the proposed method.

cs.LG↗

arXiv2011

Direct Measurement of the Fermi Energy in Graphene Using a Double Layer Structure

We describe a technique which allows a direct measurement of the relative Fermi energy in an electron system using a double layer structure, where graphene is one of the two layers. We illustrate this method by probing the Fermi energy as a function of density in a graphene monolayer, at zero and in high magnetic fields. This technique allows us to determine the Fermi velocity, Landau level spacing, and Landau level broadening in graphene. We find that the N=0 Landau level broadening is larger by comparison to the broadening of upper and lower Landau levels.

cond-mat.mes-hall↗

arXiv2011

Magnetotransport Properties of Quasi-Free Standing Epitaxial Graphene Bilayer on SiC: Evidence for Bernal Stacking

We investigate the magnetotransport properties of quasi-free standing epitaxial graphene bilayer on SiC, grown by atmospheric pressure graphitization in Ar, followed by H$_2$ intercalation. At the charge neutrality point the longitudinal resistance shows an insulating behavior, which follows a temperature dependence consistent with variable range hopping transport in a gapped state. In a perpendicular magnetic field, we observe quantum Hall states (QHSs) both at filling factors ($ν$) multiple of four ($ν=4, 8, 12$), as well as broken valley symmetry QHSs at $ν=0$ and $ν=6$. These results unambiguously show that the quasi-free standing graphene bilayer grown on the Si-face of SiC exhibits Bernal stacking.

cond-mat.mes-hall↗

arXiv2011

Spin-Polarized to Valley-Polarized Transition in Graphene Bilayers at $ν=0$ in High Magnetic Fields

We investigate the transverse electric field ($E$) dependence of the $ν$=0 quantum Hall state (QHS) in dual-gated graphene bilayers in high magnetic fields. The longitudinal resistivity ($ρ_{xx}$) measured at $ν$=0 shows an insulating behavior which is strongest in the vicinity of $E$=0, and at large $E$-fields. At a fixed perpendicular magnetic field ($B$), the $ν$=0 QHS undergoes a transition as a function of $E$, marked by a minimum, temperature-independent $ρ_{xx}$. This observation is explained by a transition from a spin polarized $ν$=0 QHS at small $E$-fields, to a valley (layer) polarized $ν$=0 QHS at large $E$-fields. The $E$-field value at which the transition occurs has a linear dependence on $B$

cond-mat.mes-hall↗

arXiv2011

Coulomb Drag of Massless Fermions in Graphene

Using a novel structure, consisting of two, independently contacted graphene single layers separated by an ultra-thin dielectric, we experimentally measure the Coulomb drag of massless fermions in graphene. At temperatures higher than 50 K, the Coulomb drag follows a temperature and carrier density dependence consistent with the Fermi liquid regime. As the temperature is reduced, the Coulomb drag exhibits giant fluctuations with an increasing amplitude, thanks to the interplay between coherent transport in the graphene layer and interaction between the two layers.

cond-mat.mes-hall↗