arXiv subjects

Wonjae Kim

Publications and source records attributed to Wonjae Kim.

25 records · Page 2Linked to original sources

arXiv2021

Conditional Generation of Periodic Signals with Fourier-Based Decoder

Periodic signals play an important role in daily lives. Although conventional sequential models have shown remarkable success in various fields, they still come short in modeling periodicity; they either collapse, diverge or ignore details. In this paper, we introduce a novel framework inspired by Fourier series to generate periodic signals. We first decompose the given signals into multiple sines and cosines and then conditionally generate periodic signals with the output components. We have shown our model efficacy on three tasks: reconstruction, imputation and conditional generation. Our model outperforms baselines in all tasks and shows more stable and refined results.

cs.NE↗

arXiv2021

ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision

Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks. Current approaches to VLP heavily rely on image feature extraction processes, most of which involve region supervision (e.g., object detection) and the convolutional architecture (e.g., ResNet). Although disregarded in the literature, we find it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more computation than the multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive power of the visual embedder and its predefined visual vocabulary. In this paper, we present a minimal VLP model, Vision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically simplified to just the same convolution-free manner that we process textual inputs. We show that ViLT is up to tens of times faster than previous VLP models, yet with competitive or better downstream task performance. Our code and pre-trained weights are available at https://github.com/dandelin/vilt.

stat.ML↗

arXiv2020

Diversified Mutual Learning for Deep Metric Learning

Mutual learning is an ensemble training strategy to improve generalization by transferring individual knowledge to each other while simultaneously training multiple models. In this work, we propose an effective mutual learning method for deep metric learning, called Diversified Mutual Metric Learning, which enhances embedding models with diversified mutual learning. We transfer relational knowledge for deep metric learning by leveraging three kinds of diversities in mutual learning: (1) model diversity from different initializations of models, (2) temporal diversity from different frequencies of parameter update, and (3) view diversity from different augmentations of inputs. Our method is particularly adequate for inductive transfer learning at the lack of large-scale data, where the embedding model is initialized with a pretrained model and then fine-tuned on a target dataset. Extensive experiments show that our method significantly improves individual models as well as their ensemble. Finally, the proposed method with a conventional triplet loss achieves the state-of-the-art performance of Recall@1 on standard datasets: 69.9 on CUB-200-2011 and 89.1 on CARS-196.

cs.CV↗

arXiv2020

Discrete Infomax Codes for Supervised Representation Learning

Learning compact discrete representations of data is a key task on its own or for facilitating subsequent processing of data. In this paper we present a model that produces Discrete InfoMax Codes (DIMCO); we learn a probabilistic encoder that yields k-way d-dimensional codes associated with input data. Our model's learning objective is to maximize the mutual information between codes and labels with a regularization, which enforces entries of a codeword to be as independent as possible. We show that the infomax principle also justifies previous loss functions (e.g., cross-entropy) as its special cases. Our analysis also shows that using shorter codes, as DIMCO does, reduces overfitting in the context of few-shot classification. Through experiments in various domains, we observe this implicit meta-regularization effect of DIMCO. Furthermore, we show that the codes learned by DIMCO are efficient in terms of both memory and retrieval time compared to previous methods.

stat.ML↗

arXiv2019

Learning Dynamics of Attention: Human Prior for Interpretable Machine Reasoning

Without relevant human priors, neural networks may learn uninterpretable features. We propose Dynamics of Attention for Focus Transition (DAFT) as a human prior for machine reasoning. DAFT is a novel method that regularizes attention-based reasoning by modelling it as a continuous dynamical system using neural ordinary differential equations. As a proof of concept, we augment a state-of-the-art visual reasoning model with DAFT. Our experiments reveal that applying DAFT yields similar performance to the original model while using fewer reasoning steps, showing that it implicitly learns to skip unnecessary steps. We also propose a new metric, Total Length of Transition (TLT), which represents the effective reasoning step size by quantifying how much a given model's focus drifts while reasoning about a question. We show that adding DAFT results in lower TLT, demonstrating that our method indeed obeys the human prior towards shorter reasoning paths in addition to producing more interpretable attention maps. Our code is available at https://github.com/kakao/DAFT.

stat.ML↗

arXiv2017

Scaling of graphene field-effect transistors supported on hexagonal boron nitride: radio-frequency stability as a limiting factor

The quality of graphene in nanodevices has increased hugely thanks to the use of hexagonal boron nitride as a supporting layer. This paper studies to which extent hBN together with channel length scaling can be exploited in graphene field effect transistors (GFETs) to get a competitive radio frequency (RF) performance. Carrier mobility and saturation velocity were obtained from an ensemble Monte Carlo simulator that accounted for the relevant scattering mechanisms (intrinsic phonons, scattering with impurities and defects, etc.). This information is fed into a self consistent simulator, which solves the drift diffusion equation coupled with the two dimensional Poisson's equation to take full account of short channel effects. Simulated GFET characteristics were benchmarked against experimental data from our fabricated devices. Our simulations show that scalability is supposed to bring to RF performance an improvement that is, however, highly limited by instability. Despite the possibility of a lower performance, a careful choice of the bias point can avoid instability. Nevertheless, maximum oscillation frequencies are still achievable in the THz region for channel lengths of a few hundreds of nanometers.

physics.app-ph↗

arXiv2015

A physics based model of gate tunable metal-graphene contact resistance benchmarked against experimental data

The metal-graphene contact resistance is a technological bottleneck for the realization of viable graphene based electronics. We report a useful model to find the gate tunable components of this resistance determined by the sequential tunneling of carriers between the 3D-metal and 2D-graphene underneath followed by Klein tunneling to the graphene in the channel. This model quantifies the intrinsic factors that control that resistance, including the effect of unintended chemical doping. Our results agree with experimental results for several metals.

cond-mat.mes-hall↗