arXiv subjects

Jinho Lee

Publications and source records attributed to Jinho Lee.

At least 55 records · Page 3Linked to original sources

arXiv2022

It's All In the Teacher: Zero-Shot Quantization Brought Closer to the Teacher

Model quantization is considered as a promising method to greatly reduce the resource requirements of deep neural networks. To deal with the performance drop induced by quantization errors, a popular method is to use training data to fine-tune quantized networks. In real-world environments, however, such a method is frequently infeasible because training data is unavailable due to security, privacy, or confidentiality concerns. Zero-shot quantization addresses such problems, usually by taking information from the weights of a full-precision teacher network to compensate the performance drop of the quantized networks. In this paper, we first analyze the loss surface of state-of-the-art zero-shot quantization techniques and provide several findings. In contrast to usual knowledge distillation problems, zero-shot quantization often suffers from 1) the difficulty of optimizing multiple loss terms together, and 2) the poor generalization capability due to the use of synthetic samples. Furthermore, we observe that many weights fail to cross the rounding threshold during training the quantized networks even when it is necessary to do so for better performance. Based on the observations, we propose AIT, a simple yet powerful technique for zero-shot quantization, which addresses the aforementioned two problems in the following way: AIT i) uses a KL distance loss only without a cross-entropy loss, and ii) manipulates gradients to guarantee that a certain portion of weights are properly updated after crossing the rounding thresholds. Experiments show that AIT outperforms the performance of many existing methods by a great margin, taking over the overall state-of-the-art position in the field.

cs.CV↗

arXiv2021

Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples

Model quantization is known as a promising method to compress deep neural networks, especially for inferences on lightweight mobile or edge devices. However, model quantization usually requires access to the original training data to maintain the accuracy of the full-precision models, which is often infeasible in real-world scenarios for security and privacy issues. A popular approach to perform quantization without access to the original data is to use synthetically generated samples, based on batch-normalization statistics or adversarial learning. However, the drawback of such approaches is that they primarily rely on random noise input to the generator to attain diversity of the synthetic samples. We find that this is often insufficient to capture the distribution of the original data, especially around the decision boundaries. To this end, we propose Qimera, a method that uses superposed latent embeddings to generate synthetic boundary supporting samples. For the superposed embeddings to better reflect the original distribution, we also propose using an additional disentanglement mapping layer and extracting information from the full-precision model. The experimental results show that Qimera achieves state-of-the-art performances for various settings on data-free quantization. Code is available at https://github.com/iamkanghyunchoi/qimera.

cs.LG↗

arXiv2021

Visualizing the Energy-gap Modulations of the Cuprate Pair Density Wave State

When Cooper pairs are formed with finite center-of-mass momentum, the defining characteristic is a spatially modulating superconducting energy gap $Δ(r)$. Recently, this concept has been generalized to the pair density wave (PDW) state predicted to exist in a variety of strongly correlated electronic materials such as the cuprates. Although the signature of a cuprate PDW has been detected in Cooper-pair tunnelling, the distinctive signature in single-electron tunneling of a periodic $Δ(r)$ modulation has never been observed. Here, using a new approach, we discover strong $Δ(r)$ modulations in Bi$_2$Sr$_2$CaCu$_2$O$_8$+$δ$ that have eight-unit-cell periodicity or wavevectors $Q=2π/a_0(1/8,0)$; $2π/a_0(0,1/8)$. This constitutes the first energy-resolved spectroscopic evidence for the cuprate PDW state. An analysis of spatial arrangements of $Δ(r)$ modulations then reveals that this PDW is predominantly unidirectional, but with an arrangement of nanoscale domains indicative of a vestigial PDW. Simultaneous imaging of the local-density-of-states $N(r,E)$ reveals electronic modulations with wavevectors $Q$ and $2Q$, as anticipated when the PDW coexists with superconductivity. Finally, by visualizing the topological defects in these $N(r,E)$ density waves at $2Q$, we discover them to be concentrated in areas where the PDW spatial phase changes by $π$, as predicted by the theory of half-vortices in a PDW state. Overall, this is a compelling demonstration, from multiple single-electron signatures, of a PDW state coexisting with superconductivity at zero magnetic field, in the canonical cuprate Bi$_2$Sr$_2$CaCu$_2$O$_8$+$δ$.

cond-mat.supr-con↗

arXiv2021

An Attention Module for Convolutional Neural Networks

Attention mechanism has been regarded as an advanced technique to capture long-range feature interactions and to boost the representation capability for convolutional neural networks. However, we found two ignored problems in current attentional activations-based models: the approximation problem and the insufficient capacity problem of the attention maps. To solve the two problems together, we initially propose an attention module for convolutional neural networks by developing an AW-convolution, where the shape of attention maps matches that of the weights rather than the activations. Our proposed attention module is a complementary method to previous attention-based schemes, such as those that apply the attention mechanism to explore the relationship between channel-wise and spatial features. Experiments on several datasets for image classification and object detection tasks show the effectiveness of our proposed attention module. In particular, our proposed attention module achieves 1.00% Top-1 accuracy improvement on ImageNet classification over a ResNet101 baseline and 0.63 COCO-style Average Precision improvement on the COCO object detection on top of a Faster R-CNN baseline with the backbone of ResNet101-FPN. When integrating with the previous attentional activations-based models, our proposed attention module can further increase their Top-1 accuracy on ImageNet classification by up to 0.57% and COCO-style Average Precision on the COCO object detection by up to 0.45. Code and pre-trained models will be publicly available.

cs.CV↗

arXiv2021

AutoReCon: Neural Architecture Search-based Reconstruction for Data-free Compression

Data-free compression raises a new challenge because the original training dataset for a pre-trained model to be compressed is not available due to privacy or transmission issues. Thus, a common approach is to compute a reconstructed training dataset before compression. The current reconstruction methods compute the reconstructed training dataset with a generator by exploiting information from the pre-trained model. However, current reconstruction methods focus on extracting more information from the pre-trained model but do not leverage network engineering. This work is the first to consider network engineering as an approach to design the reconstruction method. Specifically, we propose the AutoReCon method, which is a neural architecture search-based reconstruction method. In the proposed AutoReCon method, the generator architecture is designed automatically given the pre-trained model for reconstruction. Experimental results show that using generators discovered by the AutoRecon method always improve the performance of data-free compression.

cs.CV↗

arXiv2021

DANCE: Differentiable Accelerator/Network Co-Exploration

To cope with the ever-increasing computational demand of the DNN execution, recent neural architecture search (NAS) algorithms consider hardware cost metrics into account, such as GPU latency. To further pursue a fast, efficient execution, DNN-specialized hardware accelerators are being designed for multiple purposes, which far-exceeds the efficiency of the GPUs. However, those hardware-related metrics have been proven to exhibit non-linear relationships with the network architectures. Therefore it became a chicken-and-egg problem to optimize the network against the accelerator, or to optimize the accelerator against the network. In such circumstances, this work presents DANCE, a differentiable approach towards the co-exploration of the hardware accelerator and network architecture design. At the heart of DANCE is a differentiable evaluator network. By modeling the hardware evaluation software with a neural network, the relation between the accelerator architecture and the hardware metrics becomes differentiable, allowing the search to be performed with backpropagation. Compared to the naive existing approaches, our method performs co-exploration in a significantly shorter time, while achieving superior accuracy and hardware cost metrics.

cs.LG↗

arXiv2021

GradPIM: A Practical Processing-in-DRAM Architecture for Gradient Descent

In this paper, we present GradPIM, a processing-in-memory architecture which accelerates parameter updates of deep neural networks training. As one of processing-in-memory techniques that could be realized in the near future, we propose an incremental, simple architectural design that does not invade the existing memory protocol. Extending DDR4 SDRAM to utilize bank-group parallelism makes our operation designs in processing-in-memory (PIM) module efficient in terms of hardware cost and performance. Our experimental results show that the proposed architecture can improve the performance of DNN training and greatly reduce memory bandwidth requirement while posing only a minimal amount of overhead to the protocol and DRAM area.

cs.LG↗

arXiv2021

Homogeneous superconducting gap in DBCO synthesized by oxide molecular beam epitaxy

Much of what is known about high-temperature cuprate superconductors stems from studies based on two surface analytical tools, angle-resolved photoemission spectroscopy (ARPES) and spectroscopic imaging scanning tunneling microscopy (SI-STM). A question of general interest is whether and when the surface properties probed by ARPES and SI-STM are representative of the intrinsic properties of bulk materials. We find this question is prominent in thin films of a rarely studied cuprate DBCO. We synthesize DBCO films by oxide molecular beam epitaxy and study them by in situ ARPES and SI-STM. Both ARPES and SI-STM show that the surface DBCO layer is different from the bulk of the film. It is heavily underdoped, while the doping level in the bulk is close to optimal doping evidenced by bulk-sensitive mutual inductance measurements. ARPES shows the typical electronic structure of a heavily underdoped CuO2 plane and two sets of one-dimensional bands originating from the CuO chains with one of them gapped. SI-STM reveals two different energy scales in the local density of states, with one corresponding to the superconductivity and the other one to the pseudogap. While the pseudogap shows large variations over the length scale of a few nanometers, the superconducting gap is very homogeneous. This indicates that the pseudogap and superconductivity are of different origins.

cond-mat.supr-con↗

arXiv2020

Deep Composer Classification Using Symbolic Representation

In this study, we train deep neural networks to classify composer on a symbolic domain. The model takes a two-channel two-dimensional input, i.e., onset and note activations of time-pitch representation, which is converted from MIDI recordings and performs a single-label classification. On the experiments conducted on MAESTRO dataset, we report an F1 value of 0.8333 for the classification of 13~classical composers.

cs.SD↗

arXiv2020

SoFAr: Shortcut-based Fractal Architectures for Binary Convolutional Neural Networks

Binary Convolutional Neural Networks (BCNNs) can significantly improve the efficiency of Deep Convolutional Neural Networks (DCNNs) for their deployment on resource-constrained platforms, such as mobile and embedded systems. However, the accuracy degradation of BCNNs is still considerable compared with their full precision counterpart, impeding their practical deployment. Because of the inevitable binarization error in the forward propagation and gradient mismatch problem in the backward propagation, it is nontrivial to train BCNNs to achieve satisfactory accuracy. To ease the difficulty of training, the shortcut-based BCNNs, such as residual connection-based Bi-real ResNet and dense connection-based BinaryDenseNet, introduce additional shortcuts in addition to the shortcuts already present in their full precision counterparts. Furthermore, fractal architectures have been also been used to improve the training process of full-precision DCNNs since the fractal structure triggers effects akin to deep supervision and lateral student-teacher information flow. Inspired by the shortcuts and fractal architectures, we propose two Shortcut-based Fractal Architectures (SoFAr) specifically designed for BCNNs: 1. residual connection-based fractal architectures for binary ResNet, and 2. dense connection-based fractal architectures for binary DenseNet. Our proposed SoFAr combines the adoption of shortcuts and the fractal architectures in one unified model, which is helpful in the training of BCNNs. Results show that our proposed SoFAr achieves better accuracy compared with shortcut-based BCNNs. Specifically, the Top-1 accuracy of our proposed RF-c4d8 ResNet37(41) and DRF-c2d2 DenseNet51(53) on ImageNet outperforms Bi-real ResNet18(64) and BinaryDenseNet51(32) by 3.29% and 1.41%, respectively, with the same computational complexity overhead.

cs.CV↗

arXiv2020

MAPS: Multi-agent Reinforcement Learning-based Portfolio Management System

Generating an investment strategy using advanced deep learning methods in stock markets has recently been a topic of interest. Most existing deep learning methods focus on proposing an optimal model or network architecture by maximizing return. However, these models often fail to consider and adapt to the continuously changing market conditions. In this paper, we propose the Multi-Agent reinforcement learning-based Portfolio management System (MAPS). MAPS is a cooperative system in which each agent is an independent "investor" creating its own portfolio. In the training procedure, each agent is guided to act as diversely as possible while maximizing its own return with a carefully designed loss function. As a result, MAPS as a system ends up with a diversified portfolio. Experiment results with 12 years of US market data show that MAPS outperforms most of the baselines in terms of Sharpe ratio. Furthermore, our results show that adding more agents to our system would allow us to get a higher Sharpe ratio by lowering risk with a more diversified portfolio.

cs.AI↗

arXiv2020

Atomic-scale Electronic Structure of the Cuprate Pair Density Wave State Coexisting with Superconductivity

The defining characteristic of hole-doped cuprates is $d$-wave high temperature superconductivity. However, intense theoretical interest is now focused on whether a pair density wave state (PDW) could coexist with cuprate superconductivity (D. F. Agterberg et al., Annual Review of Condensed Matter Physics 11, 231 (2020)). Here, we use a strong-coupling mean-field theory of cuprates, to model the atomic-scale electronic structure of an eight-unit-cell periodic, $d$-symmetry form factor, pair density wave (PDW) state coexisting with $d$-wave superconductivity (DSC). From this PDW+DSC model, the atomically-resolved density of Bogoliubov quasiparticle states N(r,E) is predicted at the terminal BiO surface of Bi$_2$Sr$_2$CaCu$_2$O$_8$ and compared with high-precision electronic visualization experiments using spectroscopic imaging STM. The PDW+DSC model predictions include the intra-unit-cell structure and periodic modulations of N(r,E), the modulations of the coherence peak energy $Δ_p$ (r), and the characteristics of Bogoliubov quasiparticle interference in scattering-wavevector space (q-space). Consistency between all these predictions and the corresponding experiments indicates that lightly hole-doped Bi$_2$Sr$_2$CaCu$_2$O$_8$ does contain a PDW+DSC state. Moreover, in the model the PDW+DSC state becomes unstable to a pure DSC state at a critical hole density p*, with empirically equivalent phenomena occurring in the experiments. All these results are consistent with a picture in which the cuprate translational symmetry breaking state is a PDW, the observed charge modulations are its consequence, the antinodal pseudogap is that of the PDW state, and the cuprate critical point at p* ~ 19% occurs due to disappearance of this PDW.

cond-mat.supr-con↗

arXiv2020

SimEx: Express Prediction of Inter-dataset Similarity by a Fleet of Autoencoders

Knowing the similarity between sets of data has a number of positive implications in training an effective model, such as assisting an informed selection out of known datasets favorable to model transfer or data augmentation problems with an unknown dataset. Common practices to estimate the similarity between data include comparing in the original sample space, comparing in the embedding space from a model performing a certain task, or fine-tuning a pretrained model with different datasets and evaluating the performance changes therefrom. However, these practices would suffer from shallow comparisons, task-specific biases, or extensive time and computations required to perform comparisons. We present SimEx, a new method for early prediction of inter-dataset similarity using a set of pretrained autoencoders each of which is dedicated to reconstructing a specific part of known data. Specifically, our method takes unknown data samples as input to those pretrained autoencoders, and evaluate the difference between the reconstructed output samples against their original input samples. Our intuition is that, the more similarity exists between the unknown data samples and the part of known data that an autoencoder was trained with, the better chances there could be that this autoencoder makes use of its trained knowledge, reconstructing output samples closer to the originals. We demonstrate that our method achieves more than 10x speed-up in predicting inter-dataset similarity compared to common similarity-estimating practices. We also demonstrate that the inter-dataset similarity estimated by our method is well-correlated with common practices and outperforms the baselines approaches of comparing at sample- or embedding-spaces, without newly training anything at the comparison time.

cs.LG↗

arXiv2019

MUTE: Data-Similarity Driven Multi-hot Target Encoding for Neural Network Design

Target encoding is an effective technique to deliver better performance for conventional machine learning methods, and recently, for deep neural networks as well. However, the existing target encoding approaches require significant increase in the learning capacity, thus demand higher computation power and more training data. In this paper, we present a novel and efficient target encoding scheme, MUTE to improve both generalizability and robustness of a target model by understanding the inter-class characteristics of a target dataset. By extracting the confusion level between the target classes in a dataset, MUTE strategically optimizes the Hamming distances among target encoding. Such optimized target encoding offers higher classification strength for neural network models with negligible computation overhead and without increasing the model size. When MUTE is applied to the popular image classification networks and datasets, our experimental results show that MUTE offers better generalization and defense against the noises and adversarial attacks over the existing solutions.

cs.LG↗

arXiv2019

Facile and time-resolved chemical growth of nanoporous CaxCoO2 thin films for flexible and thermoelectric applications

CaxCoO2 thin films can be promising for widespread flexible thermoelectric applications in a wide temperature range from room-temperature self-powered wearable applications (by harvesting power from body heat) to energy harvesting from hot surfaces (e.g., hot pipes) if a cost-effective and facile growth technique is developed. Here, we demonstrate a time resolved, facile and ligand-free soft chemical method for the growth of nanoporous Ca0.35CoO2 thin films on sapphire and mica substrates from a water-based precursor ink, composed of in-situ prepared Ca2+-DMF and Co2+-DMF complexes. Mica serves as flexible substrate as well as sacrificial layer for film transfer. The grown films are oriented and can sustain bending stress until a bending radius of 15 mm. Despite the presence of nanopores, the power factor of Ca0.35CoO2 film is found to be as high as 0.50 x 10-4 Wm-1K-2 near room temperature. The present technique, being simple and fast to be potentially suitable for cost-effective industrial upscaling.

cond-mat.mtrl-sci↗

arXiv2019

Global Stock Market Prediction Based on Stock Chart Images Using Deep Q-Network

We applied Deep Q-Network with a Convolutional Neural Network function approximator, which takes stock chart images as input, for making global stock market predictions. Our model not only yields profit in the stock market of the country where it was trained but generally yields profit in global stock markets. We trained our model only in the US market and tested it in 31 different countries over 12 years. The portfolios constructed based on our model's output generally yield about 0.1 to 1.0 percent return per transaction prior to transaction costs in 31 countries. The results show that there are some patterns on stock chart image, that tend to predict the same future stock price movements across global stock markets. Moreover, the results show that future stock prices can be predicted even if the training and testing procedures are done in different countries. Training procedure could be done in relatively large and liquid markets (e.g., USA) and tested in small markets. This result demonstrates that artificial intelligence based stock price forecasting models can be used in relatively small markets (emerging countries) even though they do not have a sufficient amount of data for training.

q-fin.GN↗

arXiv2018

System G Distributed Graph Database

Motivated by the need to extract knowledge and value from interconnected data, graph analytics on big data is a very active area of research in both industry and academia. To support graph analytics efficiently a large number of in memory graph libraries, graph processing systems and graph databases have emerged. Projects in each of these categories focus on particular aspects such as static versus dynamic graphs, off line versus on line processing, small versus large graphs, etc. While there has been much advance in graph processing in the past decades, there is still a need for a fast graph processing, using a cluster of machines with distributed storage. In this paper, we discuss a novel distributed graph database called System G designed for efficient graph data storage and processing on modern computing architectures. In particular we describe a single node graph database and a runtime and communication layer that allows us to compose a distributed graph database from multiple single node instances. From various industry requirements, we find that fast insertions and large volume concurrent queries are critical parts of the graph databases and we optimize our database for such features. We experimentally show the efficiency of System G for storing data and processing graph queries on state-of-the-art platforms.

cs.DB↗

arXiv2016

Globally Optimal Object Tracking with Fully Convolutional Networks

Tracking is one of the most important but still difficult tasks in computer vision and pattern recognition. The main difficulties in the tracking field are appearance variation and occlusion. Most traditional tracking methods set the parameters or templates to track target objects in advance and should be modified accordingly. Thus, we propose a new and robust tracking method using a Fully Convolutional Network (FCN) to obtain an object probability map and Dynamic Programming (DP) to seek the globally optimal path through all frames of video. Our proposed method solves the object appearance variation problem with the use of a FCN and deals with occlusion by DP. We show that our method is effective in tracking various single objects through video frames.

cs.CV↗