arXiv subjects

Juho Lee

Publications and source records attributed to Juho Lee.

At least 127 records · Page 7Linked to original sources

arXiv2020

Cost-effective Interactive Attention Learning with Neural Attention Processes

We propose a novel interactive learning framework which we refer to as Interactive Attention Learning (IAL), in which the human supervisors interactively manipulate the allocated attentions, to correct the model's behavior by updating the attention-generating network. However, such a model is prone to overfitting due to scarcity of human annotations, and requires costly retraining. Moreover, it is almost infeasible for the human annotators to examine attentions on tons of instances and features. We tackle these challenges by proposing a sample-efficient attention mechanism and a cost-effective reranking algorithm for instances and features. First, we propose Neural Attention Process (NAP), which is an attention generator that can update its behavior by incorporating new attention-level supervisions without any retraining. Secondly, we propose an algorithm which prioritizes the instances and the features by their negative impacts, such that the model can yield large improvements with minimal human feedback. We validate IAL on various time-series datasets from multiple domains (healthcare, real-estate, and computer vision) on which it significantly outperforms baselines with conventional attention mechanisms, or without cost-effective reranking, with substantially less retraining and human-model interaction cost.

cs.LG↗

arXiv2020

Multi-space excitation as an alternative to the Landauer picture for non-equilibrium quantum transport

While the Landauer viewpoint constitutes a modern basis to understand nanoscale electronic transport and to realize first-principles implementations of the non-equilibrium Green's function (NEGF) formalism, seeking an alternative picture could be beneficial for the fundamental understanding and practical calculations of quantum transport processes. Herein, introducing a micro-canonical picture that maps the finite-bias quantum transport process to a drain-to-source or multi-electrode optical excitation, the multi-space constrained-search density functional theory (MS-DFT) formalism for first-principles electronic structure and quantum transport calculations is developed. Performing MS-DFT calculations for the benzenedithiolate single-molecule junction, it is shown that MS-DFT and standard DFT-NEGF calculations produce practically equivalent electronic and transmission data. Importantly, the variational convergence of "non-equilibrium total energy" within MS-DFT is demonstrated, which should have significant implications for in operando studies of nanoscale devices. Establishing a viable alternative to the Landauer viewpoint, the developed formalism should provide valuable atomistic information in the development of next-generation nanodevices.

cond-mat.mes-hall↗

arXiv2020

Quasi-Fermi level splitting in nanoscale junctions from $\textit{ab initio}$

The splitting of quasi-Fermi levels (QFLs) represents a key concept utilized to describe finite-bias operations of semiconductor devices, but its atomic-scale characterization remains a significant challenge. Herein, the non-equilibrium QFL or electrochemical potential profiles within single-molecule junctions obtained from the newly developed first-principles multi-space constrained-search density functional formalism are presented. Benchmarking the standard non-equilibrium Green's function calculation results, it is first established that algorithmically the notion of separate electrode-originated nonlocal QFLs should be maintained within the channel region during self-consistent finite-bias electronic structure calculations. For the insulating hexandithiolate junction, the QFL profiles exhibit discontinuities at the left and right electrode interfaces and across the molecule the accompanying electrostatic potential drops linearly and Landauer residual-resistivity dipoles are uniformly distributed. For the conducting hexatrienedithiolate junction, on the other hand, the electrode QFLs penetrate into the channel region and produce split QFLs. With the highest occupied molecular orbital entering the bias window and becoming a good transport channel, the split QFLs are also accompanied by the nonlinear electrostatic potential drop and asymmetric Landauer residua-resistivity dipole formation. Our findings underscore the importance of the first-principles extraction of QFLs in nanoscale junctions and point to a new direction for the computational design of next-generation electronic, optoelectronic, and electrochemical devices.

cond-mat.mes-hall↗

arXiv2020

Sparse Vector Transmission: An Idea Whose Time Has Come

In recent years, we are witnessing bewildering variety of automated services and applications of vehicles, robots, sensors, and machines powered by the artificial intelligence technologies. Communication mechanism associated with these services is dearly distinct from human-centric communications. One important feature for the machine-centric communications is that the amount of information to be transmitted is tiny. In view of the short packet transmission, relying on today's transmission mechanism would not be efficient due to the waste of resources, large decoding latency, and expensive operational cost. In this article, we present an overview of the sparse vector transmission (SVT), a scheme to transmit a short-sized information after the sparse transformation. We discuss basics of SVT, two distinct SVT strategies, viz., frequency-domain sparse transmission and sparse vector coding with detailed operations, and also demonstrate the effectiveness in realistic wireless environments.

eess.SP↗

arXiv2019

Deep Mixed Effect Model using Gaussian Processes: A Personalized and Reliable Prediction for Healthcare

We present a personalized and reliable prediction model for healthcare, which can provide individually tailored medical services such as diagnosis, disease treatment, and prevention. Our proposed framework targets at making personalized and reliable predictions from time-series data, such as Electronic Health Records (EHR), by modeling two complementary components: i) a shared component that captures global trend across diverse patients and ii) a patient-specific component that models idiosyncratic variability for each patient. To this end, we propose a composite model of a deep neural network to learn complex global trends from the large number of patients, and Gaussian Processes (GP) to probabilistically model individual time-series given relatively small number of visits per patient. We evaluate our model on diverse and heterogeneous tasks from EHR datasets and show practical advantages over standard time-series deep models such as pure Recurrent Neural Network (RNN).

stat.ML↗

arXiv2019

Graph Embedding VAE: A Permutation Invariant Model of Graph Structure

Generative models of graph structure have applications in biology and social sciences. The state of the art is GraphRNN, which decomposes the graph generation process into a series of sequential steps. While effective for modest sizes, it loses its permutation invariance for larger graphs. Instead, we present a permutation invariant latent-variable generative model relying on graph embeddings to encode structure. Using tools from the random graph literature, our model is highly scalable to large graphs with likelihood evaluation and generation in $O(|V | + |E|)$.

cs.LG↗

arXiv2019

Deep Amortized Clustering

We propose a deep amortized clustering (DAC), a neural architecture which learns to cluster datasets efficiently using a few forward passes. DAC implicitly learns what makes a cluster, how to group data points into clusters, and how to count the number of clusters in datasets. DAC is meta-learned using labelled datasets for training, a process distinct from traditional clustering algorithms which usually require hand-specified prior knowledge about cluster shapes/structures. We empirically show, on both synthetic and image data, that DAC can efficiently and accurately cluster new datasets coming from the same distribution used to generate training datasets.

cs.LG↗

arXiv2019

Beyond the Chinese Restaurant and Pitman-Yor processes: Statistical Models with Double Power-law Behavior

Bayesian nonparametric approaches, in particular the Pitman-Yor process and the associated two-parameter Chinese Restaurant process, have been successfully used in applications where the data exhibit a power-law behavior. Examples include natural language processing, natural images or networks. There is also growing empirical evidence that some datasets exhibit a two-regime power-law behavior: one regime for small frequencies, and a second regime, with a different exponent, for high frequencies. In this paper, we introduce a class of completely random measures which are doubly regularly-varying. Contrary to the Pitman-Yor process, we show that when completely random measures in this class are normalized to obtain random probability measures and associated random partitions, such partitions exhibit a double power-law behavior. We discuss in particular three models within this class: the beta prime process (Broderick et al. (2015, 2018), a novel process called generalized BFRY process, and a mixture construction. We derive efficient Markov chain Monte Carlo algorithms to estimate the parameters of these models. Finally, we show that the proposed models provide a better fit than the Pitman-Yor process on various datasets.

stat.ML↗

arXiv2019

Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks

Many machine learning tasks such as multiple instance learning, 3D shape recognition, and few-shot image classification are defined on sets of instances. Since solutions to such problems do not depend on the order of elements of the set, models used to address them should be permutation invariant. We present an attention-based neural network module, the Set Transformer, specifically designed to model interactions among elements in the input set. The model consists of an encoder and a decoder, both of which rely on attention mechanisms. In an effort to reduce computational complexity, we introduce an attention scheme inspired by inducing point methods from sparse Gaussian process literature. It reduces the computation time of self-attention from quadratic to linear in the number of elements in the set. We show that our model is theoretically attractive and we evaluate it on a range of tasks, demonstrating the state-of-the-art performance compared to recent methods for set-structured data.

cs.LG↗

arXiv2019

A unified construction for series representations and finite approximations of completely random measures

Infinite-activity completely random measures (CRMs) have become important building blocks of complex Bayesian nonparametric models. They have been successfully used in various applications such as clustering, density estimation, latent feature models, survival analysis or network science. Popular infinite-activity CRMs include the (generalized) gamma process and the (stable) beta process. However, except in some specific cases, exact simulation or scalable inference with these models is challenging and finite-dimensional approximations are often considered. In this work, we propose a general and unified framework to derive both series representations and finite-dimensional approximations of CRMs. Our framework can be seen as an extension of constructions based on size-biased sampling of Poisson point process [Perman1992]. It includes as special cases several known series representations as well as novel ones. In particular, we show that one can get novel series representations for the generalized gamma process and the stable beta process. We also provide some analysis of the truncation error.

math.ST↗

arXiv2019

Adaptive Network Sparsification with Dependent Variational Beta-Bernoulli Dropout

While variational dropout approaches have been shown to be effective for network sparsification, they are still suboptimal in the sense that they set the dropout rate for each neuron without consideration of the input data. With such input-independent dropout, each neuron is evolved to be generic across inputs, which makes it difficult to sparsify networks without accuracy loss. To overcome this limitation, we propose adaptive variational dropout whose probabilities are drawn from sparsity-inducing beta Bernoulli prior. It allows each neuron to be evolved either to be generic or specific for certain inputs, or dropped altogether. Such input-adaptive sparsity-inducing dropout allows the resulting network to tolerate larger degree of sparsity without losing its expressive power by removing redundancies among features. We validate our dependent variational beta-Bernoulli dropout on multiple public datasets, on which it obtains significantly more compact networks than baseline methods, with consistent accuracy improvements over the base networks.

stat.ML↗

arXiv2019

Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learning

The goal of few-shot learning is to learn a classifier that generalizes well even when trained with a limited number of training instances per class. The recently introduced meta-learning approaches tackle this problem by learning a generic classifier across a large number of multiclass classification tasks and generalizing the model to a new task. Yet, even with such meta-learning, the low-data problem in the novel classification task still remains. In this paper, we propose Transductive Propagation Network (TPN), a novel meta-learning framework for transductive inference that classifies the entire test set at once to alleviate the low-data problem. Specifically, we propose to learn to propagate labels from labeled instances to unlabeled test instances, by learning a graph construction module that exploits the manifold structure in the data. TPN jointly learns both the parameters of feature embedding and the graph construction in an end-to-end manner. We validate TPN on multiple benchmark datasets, on which it largely outperforms existing few-shot learning approaches and achieves the state-of-the-art results.

cs.LG↗

arXiv2018

Semimetallicity and Negative Differential Resistance from Hybrid Halide Perovskite Nanowires

In the rapidly progressing field of organometal halide perovskites, the dimensional reduction could open up new opportunities for device applications. Herein, taking the recently synthesized trimethylsulfonium lead triiodide (CH$_3$)$_3$SPbI$_3$ perovskite as a representative example, we carry out first-principles calculations and study the nanostructuring and device application of halide perovskite nanowires. We find that the one-dimensional (1D) (CH$_3$)$_3$SPbI$_3$ structure is structurally stable, and the electronic structures of higher-dimensional forms are robustly determined at the 1D level. Remarkably, due to the face-sharing [PbI$_6$] octahedral atomic structure, the organic ligand-removed 1D PbI$_3$ frameworks are also found to be stable. Moreover, the PbI$_3$ columns avoid the Peierls distortion and assume a semimetallic character, contradicting the conventional assumption of semiconducting metal-halogen inorganic frameworks. Adopting the bundled nanowire junctions consisting of (CH$_3$)$_3$SPbI$_3$ channels with sub-5 nm dimensions sandwiched between PbI$_3$ electrodes, we finally obtain high current densities and large room-temperature negative differential resistance (NDR). It will be emphasized that the NDR originates from the combination of the near-Ohmic character of (CH$_3$)$_3$SPbI$_3$-PbI$_3$ contacts and a novel NDR mechanism that involves the quantum-mechanical hybridization between channel and electrode states. Our work demonstrates the great potential of low-dimensional hybrid perovskites toward advanced electronic devices beyond actively-pursued photonic applications.

cond-mat.mes-hall↗

arXiv2018

DropMax: Adaptive Variational Softmax

We propose DropMax, a stochastic version of softmax classifier which at each iteration drops non-target classes according to dropout probabilities adaptively decided for each instance. Specifically, we overlay binary masking variables over class output probabilities, which are input-adaptively learned via variational inference. This stochastic regularization has an effect of building an ensemble classifier out of exponentially many classifiers with different decision boundaries. Moreover, the learning of dropout rates for non-target classes on each instance allows the classifier to focus more on classification against the most confusing classes. We validate our model on multiple public datasets for classification, on which it obtains significantly improved accuracy over the regular softmax classifier and other baselines. Further analysis of the learned dropout probabilities shows that our model indeed selects confusing classes more often when it performs classification.

cs.LG↗

arXiv2018

A Bayesian model for sparse graphs with flexible degree distribution and overlapping community structure

We consider a non-projective class of inhomogeneous random graph models with interpretable parameters and a number of interesting asymptotic properties. Using the results of Bollobás et al. [2007], we show that i) the class of models is sparse and ii) depending on the choice of the parameters, the model is either scale-free, with power-law exponent greater than 2, or with an asymptotic degree distribution which is power-law with exponential cut-off. We propose an extension of the model that can accommodate an overlapping community structure. Scalable posterior inference can be performed due to the specific choice of the link probability. We present experiments on five different real-world networks with up to 100,000 nodes and edges, showing that the model can provide a good fit to the degree distribution and recovers well the latent community structure.

stat.ML↗

arXiv2018

Ultra-Reliable and Low-Latency Communications in 5G Downlink: Physical Layer Aspects

Ultra reliable and low latency communications (URLLC) is a new service category in 5G to accommodate emerging services and applications having stringent latency and reliability requirements. In order to support URLLC, there should be both evolutionary and revolutionary changes in the air interface named 5G new radio (NR). In this article, we provide an up-to-date overview of URLLC with an emphasis on the physical layer challenges and solutions in 5G NR downlink. We highlight key requirements of URLLC and then elaborate the physical layer issues and enabling technologies including packet and frame structure, scheduling schemes, and reliability improvement techniques, which have been discussed in the 3GPP Release 15 standardization.

cs.IT↗

arXiv2018

Uncertainty-Aware Attention for Reliable Interpretation and Prediction

Attention mechanism is effective in both focusing the deep learning models on relevant features and interpreting them. However, attentions may be unreliable since the networks that generate them are often trained in a weakly-supervised manner. To overcome this limitation, we introduce the notion of input-dependent uncertainty to the attention mechanism, such that it generates attention for each feature with varying degrees of noise based on the given input, to learn larger variance on instances it is uncertain about. We learn this Uncertainty-aware Attention (UA) mechanism using variational inference, and validate it on various risk prediction tasks from electronic health records on which our model significantly outperforms existing attention models. The analysis of the learned attentions shows that our model generates attentions that comply with clinicians' interpretation, and provide richer interpretation via learned variance. Further evaluation of both the accuracy of the uncertainty calibration and the prediction performance with "I don't know" decision show that UA yields networks with high reliability as well.

stat.ML↗

arXiv2018

Origin and Control of Polyacrylonitrile Alignments on Carbon Nanotube and Graphene Nanoribbon

While one of the most promising applications of carbon nanotubes (CNTs) is to enhance polymer orientation and crystallization to achieve advanced carbon fibers, the successful realization of this goal has been hindered by the insufficient atomistic understanding of polymer-CNT interfaces. We herein theoretically study polyacrylonitrile (PAN)-CNT hybrid structures as a representative example of polymer-CNT composites. Based on density-functional theory calculations, we first find that the relative orientation of polar PAN nitrile groups with respect to the CNT surface is the key factor that determines the PAN-CNT interface energetics and the lying-down PAN configurations are much more preferable than their standing-up counterparts. The CNT curvature is identified as another important factor, giving the largest binding energy in the zero-curvature graphene limit. Charge transfer analysis explains the unique tendency of linear PAN alignments on the CNT surface and the possibility of ordered PAN-PAN assembly. Next, performing large-scale molecular dynamics simulations, we show that the desirable linear PAN-CNT alignment can be achieved even for relatively large initial misorientations and further demonstrate that graphene nanoribbons are a promising carbon nano-reinforcement candidate. The microscopic understanding accumulated in this study will provide design guidelines for the development of next-generation carbon nanofibers.

cond-mat.mtrl-sci↗