arXiv subjects

Jinwon Lee

Publications and source records attributed to Jinwon Lee.

25 records · Page 2Linked to original sources

arXiv2019

QKD: Quantization-aware Knowledge Distillation

Quantization and Knowledge distillation (KD) methods are widely used to reduce memory and power consumption of deep neural networks (DNNs), especially for resource-constrained edge devices. Although their combination is quite promising to meet these requirements, it may not work as desired. It is mainly because the regularization effect of KD further diminishes the already reduced representation power of a quantized model. To address this short-coming, we propose Quantization-aware Knowledge Distillation (QKD) wherein quantization and KD are care-fully coordinated in three phases. First, Self-studying (SS) phase fine-tunes a quantized low-precision student network without KD to obtain a good initialization. Second, Co-studying (CS) phase tries to train a teacher to make it more quantizaion-friendly and powerful than a fixed teacher. Finally, Tutoring (TU) phase transfers knowledge from the trained teacher to the student. We extensively evaluate our method on ImageNet and CIFAR-10/100 datasets and show an ablation study on networks with both standard and depthwise-separable convolutions. The proposed QKD outperformed existing state-of-the-art methods (e.g., 1.3% improvement on ResNet-18 with W4A4, 2.6% on MobileNetV2 with W4A4). Additionally, QKD could recover the full-precision accuracy at as low as W3A3 quantization on ResNet and W6A6 quantization on MobilenetV2.

cs.CV↗

arXiv2019

Emergent Honeycomb Network of Topological Excitations in Correlated Charge Density Wave

When two periodic potentials compete in materials, one may adopt the other, which straightforwardly generates topological defects. Of particular interest are domain walls in charge-, dipole-, and spin-ordered systems, which govern macroscopic properties and important functionality. However, detailed atomic and electronic structures of domain walls have often been uncertain and the microscopic mechanism of their functionality has been elusive. Here, we clarify the complete atomic and electronic structures of the domain wall network, a honeycomb network connected by Z$_{3}$ vortices, in the nearly commensurate Mott charge-density wave (CDW) phase of 1T-TaS$_{2}$. Scanning tunneling microscopy resolves characteristic charge orders within domain walls and their vortices. Density functional theory calculations disclose their unique atomic relaxations and the metallic in-gap states confined tightly therein. A generic theory is constructed, which connects this emergent honeycomb network of conducting electrons to the enhanced superconductivity.

cond-mat.str-el↗

arXiv2019

Accelerated Training for CNN Distributed Deep Learning through Automatic Resource-Aware Layer Placement

The Convolutional Neural Network (CNN) model, often used for image classification, requires significant training time to obtain high accuracy. To this end, distributed training is performed with the parameter server (PS) architecture using multiple servers. Unfortunately, scalability has been found to be poor in existing architectures. We find that the PS network is the bottleneck as it communicates a large number of gradients and parameters with the many workers. This is because synchronization with the many workers has to occur at every step of training. Depending on the model, communication can be in the several hundred MBs per synchronization. In this paper, we propose a scheme to reduce network traffic through layer placement that considers the resources that each layer uses. Through analysis of the characteristics of CNN, we find that placement of layers can be done in an effective manner. We then incorporate this observation within the TensorFlow framework such that layers can be automatically placed for more efficient training. Our evaluation making use of this placement scheme show that training time can be significantly reduced without loss of accuracy for many CNN models.

cs.DC↗

arXiv2018

Automatic Grammar Augmentation for Robust Voice Command Recognition

This paper proposes a novel pipeline for automatic grammar augmentation that provides a significant improvement in the voice command recognition accuracy for systems with small footprint acoustic model (AM). The improvement is achieved by augmenting the user-defined voice command set, also called grammar set, with alternate grammar expressions. For a given grammar set, a set of potential grammar expressions (candidate set) for augmentation is constructed from an AM-specific statistical pronunciation dictionary that captures the consistent patterns and errors in the decoding of AM induced by variations in pronunciation, pitch, tempo, accent, ambiguous spellings, and noise conditions. Using this candidate set, greedy optimization based and cross-entropy-method (CEM) based algorithms are considered to search for an augmented grammar set with improved recognition accuracy utilizing a command-specific dataset. Our experiments show that the proposed pipeline along with algorithms considered in this paper significantly reduce the mis-detection and mis-classification rate without increasing the false-alarm rate. Experiments also demonstrate the consistent superior performance of CEM method over greedy-based algorithms.

cs.CL↗

arXiv2018

Strong interband interaction in the excitonic insulator phase of Ta$_2$NiSe$_5$

Excitonic insulator (EI) was proposed in 60's as a distinct insulating state originating from pure electronic interaction, but its material realization has been elusive with extremely few material candidates and with only limited evidence such as anomalies in transport properties, band dispersions, or optical transitions. We investigate the real-space electronic states of the low temperature phase in Ta$_2$NiSe$_5$ with an atomic resolution to clearly identify the quasiparticle energy gap together with the strong electron-hole band renormalization using scanning tunneling microscopy (STM) and spectroscopy (STS). These results are in good agreement with the EI transition scenario in Ta$_2$NiSe$_5$. Our spatially-resolved STS data and theoretical calculations reveal further the orbital inversion at band edges, which indicates the exciton condensation close to the Bardeen-Cooper-Schrieffer regime.

cond-mat.mtrl-sci↗

arXiv2017

Correlated electronic states at domain walls of a Mott-charge-density-wave insulator 1T-TaS2

Domain walls in interacting electronic systems can have distinct localized states, which often govern physical properties and may lead to unprecedented functionalities and novel devices. However, electronic states within domain walls themselves have not been clearly identified and understood for strongly correlated electron systems. Here, we resolve the electronic states localized on domain walls in a Mott-charge-density-wave(CDW) insulator 1T-TaS2 using scanning tunneling spectroscopy. We establish that the domain wall state decomposes into two nonconducting states located at the center of domain walls and edges of domains. Theoretical calculations reveal their atomistic origin as the local reconstruction of domain walls under the strong influence of electron correlation. Our results introduce a concept for the domain wall electronic property, the wall's own internal degrees of freedom, which is potentially related to the controllability of domain wall electronic properties.

cond-mat.str-el↗

arXiv2016

A Fully Convolutional Neural Network for Speech Enhancement

In hearing aids, the presence of babble noise degrades hearing intelligibility of human speech greatly. However, removing the babble without creating artifacts in human speech is a challenging task in a low SNR environment. Here, we sought to solve the problem by finding a `mapping' between noisy speech spectra and clean speech spectra via supervised learning. Specifically, we propose using fully Convolutional Neural Networks, which consist of lesser number of parameters than fully connected networks. The proposed network, Redundant Convolutional Encoder Decoder (R-CED), demonstrates that a convolutional network can be 12 times smaller than a recurrent network and yet achieves better performance, which shows its applicability for an embedded system: the hearing aids.

cs.LG↗