arXiv subjects

Sae-Young Chung

Publications and source records attributed to Sae-Young Chung.

At least 19 recordsLinked to original sources

arXiv2023

Test-Time Adaptation via Self-Training with Nearest Neighbor Information

Test-time adaptation (TTA) aims to adapt a trained classifier using online unlabeled test data only, without any information related to the training procedure. Most existing TTA methods adapt the trained classifier using the classifier's prediction on the test data as pseudo-label. However, under test-time domain shift, accuracy of the pseudo labels cannot be guaranteed, and thus the TTA methods often encounter performance degradation at the adapted classifier. To overcome this limitation, we propose a novel test-time adaptation method, called Test-time Adaptation via Self-Training with nearest neighbor information (TAST), which is composed of the following procedures: (1) adds trainable adaptation modules on top of the trained feature extractor; (2) newly defines a pseudo-label distribution for the test data by using the nearest neighbor information; (3) trains these modules only a few times during test time to match the nearest neighbor-based pseudo label distribution and a prototype-based class distribution for the test data; and (4) predicts the label of test data using the average predicted class distribution from these modules. The pseudo-label generation is based on the basic intuition that a test data and its nearest neighbor in the embedding space are likely to share the same label under the domain shift. By utilizing multiple randomly initialized adaptation modules, TAST extracts useful information for the classification of the test data under the domain shift, using the nearest neighbor information. TAST showed better performance than the state-of-the-art TTA methods on two standard benchmark tasks, domain generalization, namely VLCS, PACS, OfficeHome, and TerraIncognita, and image corruption, particularly CIFAR-10/100C.

cs.CV↗

arXiv2022

Unsupervised Visual Representation Learning via Mutual Information Regularized Assignment

This paper proposes Mutual Information Regularized Assignment (MIRA), a pseudo-labeling algorithm for unsupervised representation learning inspired by information maximization. We formulate online pseudo-labeling as an optimization problem to find pseudo-labels that maximize the mutual information between the label and data while being close to a given model probability. We derive a fixed-point iteration method and prove its convergence to the optimal solution. In contrast to baselines, MIRA combined with pseudo-label prediction enables a simple yet effective clustering-based representation learning without incorporating extra training techniques or artificial constraints such as sampling strategy, equipartition constraints, etc. With relatively small training epochs, representation learned by MIRA achieves state-of-the-art performance on various downstream tasks, including the linear/k-NN evaluation and transfer learning. Especially, with only 400 epochs, our method applied to ImageNet dataset with ResNet-50 architecture achieves 75.6% linear evaluation accuracy.

cs.CV↗

arXiv2022

Few-Example Clustering via Contrastive Learning

We propose Few-Example Clustering (FEC), a novel algorithm that performs contrastive learning to cluster few examples. Our method is composed of the following three steps: (1) generation of candidate cluster assignments, (2) contrastive learning for each cluster assignment, and (3) selection of the best candidate. Based on the hypothesis that the contrastive learner with the ground-truth cluster assignment is trained faster than the others, we choose the candidate with the smallest training loss in the early stage of learning in step (3). Extensive experiments on the \textit{mini}-ImageNet and CUB-200-2011 datasets show that FEC outperforms other baselines by about 3.2% on average under various scenarios. FEC also exhibits an interesting learning curve where clustering performance gradually increases and then sharply drops.

cs.LG↗

arXiv2022

Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture

The generalization ability of most meta-reinforcement learning (meta-RL) methods is largely limited to test tasks that are sampled from the same distribution used to sample training tasks. To overcome the limitation, we propose Latent Dynamics Mixture (LDM) that trains a reinforcement learning agent with imaginary tasks generated from mixtures of learned latent dynamics. By training a policy on mixture tasks along with original training tasks, LDM allows the agent to prepare for unseen test tasks during training and prevents the agent from overfitting the training tasks. LDM significantly outperforms standard meta-RL methods in test returns on the gridworld navigation and MuJoCo tasks where we strictly separate the training task distribution and the test task distribution.

cs.LG↗

arXiv2021

Unsupervised Embedding Adaptation via Early-Stage Feature Reconstruction for Few-Shot Classification

We propose unsupervised embedding adaptation for the downstream few-shot classification task. Based on findings that deep neural networks learn to generalize before memorizing, we develop Early-Stage Feature Reconstruction (ESFR) -- a novel adaptation scheme with feature reconstruction and dimensionality-driven early stopping that finds generalizable features. Incorporating ESFR consistently improves the performance of baseline methods on all standard settings, including the recently proposed transductive method. ESFR used in conjunction with the transductive method further achieves state-of-the-art performance on mini-ImageNet, tiered-ImageNet, and CUB; especially with 1.2%~2.0% improvements in accuracy over the previous best performing method on 1-shot setting.

cs.CV↗

arXiv2020

Robust Training with Ensemble Consensus

Since deep neural networks are over-parameterized, they can memorize noisy examples. We address such a memorization issue in the presence of label noise. From the fact that deep neural networks cannot generalize to neighborhoods of memorized features, we hypothesize that noisy examples do not consistently incur small losses on the network under a certain perturbation. Based on this, we propose a novel training method called Learning with Ensemble Consensus (LEC) that prevents overfitting to noisy examples by removing them based on the consensus of an ensemble of perturbed networks. One of the proposed LECs, LTEC outperforms the current state-of-the-art methods on noisy MNIST, CIFAR-10, and CIFAR-100 in an efficient manner.

cs.LG↗

arXiv2020

Novelty Detection Via Blurring

Conventional out-of-distribution (OOD) detection schemes based on variational autoencoder or Random Network Distillation (RND) have been observed to assign lower uncertainty to the OOD than the target distribution. In this work, we discover that such conventional novelty detection schemes are also vulnerable to the blurred images. Based on the observation, we construct a novel RND-based OOD detector, SVD-RND, that utilizes blurred images during training. Our detector is simple, efficient at test time, and outperforms baseline OOD detectors in various domains. Further results show that SVD-RND learns better target distribution representation than the baseline RND algorithm. Finally, SVD-RND combined with geometric transform achieves near-perfect detection accuracy on the CelebA dataset.

cs.LG↗

arXiv2019

Sample-Efficient Deep Reinforcement Learning via Episodic Backward Update

We propose Episodic Backward Update (EBU) - a novel deep reinforcement learning algorithm with a direct value propagation. In contrast to the conventional use of the experience replay with uniform random sampling, our agent samples a whole episode and successively propagates the value of a state to its previous states. Our computationally efficient recursive algorithm allows sparse and delayed rewards to propagate directly through all transitions of the sampled episode. We theoretically prove the convergence of the EBU method and experimentally demonstrate its performance in both deterministic and stochastic environments. Especially in 49 games of Atari 2600 domain, EBU achieves the same mean and median human normalized performance of DQN by using only 5% and 10% of samples, respectively.

cs.LG↗

arXiv2019

Fourier Phase Retrieval with Extended Support Estimation via Deep Neural Network

We consider the problem of sparse phase retrieval from Fourier transform magnitudes to recover the $k$-sparse signal vector and its support $\mathcal{T}$. We exploit extended support estimate $\mathcal{E}$ with size larger than $k$ satisfying $\mathcal{E} \supseteq \mathcal{T}$ and obtained by a trained deep neural network (DNN). To make the DNN learnable, it provides $\mathcal{E}$ as the union of equivalent solutions of $\mathcal{T}$ by utilizing modulo Fourier invariances. Set $\mathcal{E}$ can be estimated with short running time via the DNN, and support $\mathcal{T}$ can be determined from the DNN output rather than from the full index set by applying hard thresholding to $\mathcal{E}$. Thus, the DNN-based extended support estimation improves the reconstruction performance of the signal with a low complexity burden dependent on $k$. Numerical results verify that the proposed scheme has a superior performance with lower complexity compared to local search-based greedy sparse phase retrieval and a state-of-the-art variant of the Fienup method.

stat.ML↗

arXiv2019

Bayesian Approach with Extended Support Estimation for Sparse Regression

A greedy algorithm called Bayesian multiple matching pursuit (BMMP) is proposed to estimate a sparse signal vector and its support given $m$ linear measurements. Unlike the maximum a posteriori (MAP) support detection, which was proposed by Lee to estimate the support by selecting an index with the maximum likelihood ratio of the correlation given by a normalized version of the orthogonal matching pursuit (OMP), the proposed method uses the correlation given by the matching pursuit proposed by Davies and Eldar. BMMP exploits the diversity gain to estimate the support by considering multiple support candidates, each of which is obtained by iteratively selecting an index set with a size different for each candidate. In particular, BMMP considers an extended support estimate whose maximal size is $m$ in the process to obtain each of the support candidates. It is observed that BMMP outperforms other state-of-the-art methods and approaches the ideal limit of the signal sparsity in our simulation setting.

cs.IT↗

arXiv2019

Tree Search Network for Sparse Regression

We consider the classical sparse regression problem of recovering a sparse signal $x_0$ given a measurement vector $y = Φx_0+w$. We propose a tree search algorithm driven by the deep neural network for sparse regression (TSN). TSN improves the signal reconstruction performance of the deep neural network designed for sparse regression by performing a tree search with pruning. It is observed in both noiseless and noisy cases, TSN recovers synthetic and real signals with lower complexity than a conventional tree search and is superior to existing algorithms by a large margin for various types of the sensing matrix $Φ$, widely used in sparse regression.

cs.LG↗

arXiv2017

Community Detection with Colored Edges

In this paper, we prove a sharp limit on the community detection problem with colored edges. We assume two equal-sized communities and there are $m$ different types of edges. If two vertices are in the same community, the distribution of edges follows $p_i=α_i\log{n}/n$ for $1\leq i \leq m$, otherwise the distribution of edges is $q_i=β_i\log{n}/n$ for $1\leq i \leq m$, where $α_i$ and $β_i$ are positive constants and $n$ is the total number of vertices. Under these assumptions, a fundamental limit on community detection is characterized using the Hellinger distance between the two distributions. If $\sum_{i=1}^{m} {(\sqrt{α_i} - \sqrt{β_i})}^2 >2$, then the community detection via maximum likelihood (ML) estimator is possible with high probability. If $\sum_{i=1}^m {(\sqrt{α_i} - \sqrt{β_i})}^2 < 2$, the probability that the ML estimator fails to detect the communities does not go to zero.

cs.SI↗

arXiv2016

Capacity of Continuous-Space Electromagnetic Channels with Lossy Transceiver

In this paper, the capacity of continuous-space electromagnetic channels, where transceivers are confined in given lossy regions, is analyzed. First of all, the regions confining the transceivers are assumed to be filled with dielectric, which is either lossy or lossless. Then, for capacity analysis, we use the exact power consumption that takes into account the electromagnetic interaction between the field and the source. In addition, the exact noise model followed from the fluctuation-dissipation theorem in thermodynamics is used at the receive side. The contribution of our work is summarized as follows. First, we characterize the channel capacity as a function of the size and the physical property of the regions confining the transceivers and analytically show how the radiation efficiency affects the capacity. We also show that the outgoing channel at the transmit side and the incoming channel at the receive side are information-theoretically equivalent, and thus, the capacities of both channels are the same. Additionally, the quality factor, which is inversely proportional to the bandwidth, is theoretically derived, and the relationship between the spatial degrees of freedom of the channel and the quality factor is analyzed. Besides, we consider how the power consumption is affected by the backscattered waves and compare the recent experimental demonstration with our work by solving the gain-optimization problem with the constraint on the quality factor.

cs.IT↗

arXiv2016

Greedy Subspace Pursuit for Joint Sparse Recovery

In this paper, we address the sparse multiple measurement vector (MMV) problem where the objective is to recover a set of sparse nonzero row vectors or indices of a signal matrix from incomplete measurements. Ideally, regardless of the number of columns in the signal matrix, the sparsity (k) plus one measurements is sufficient for the uniform recovery of signal vectors for almost all signals, i.e., excluding a set of Lebesgue measure zero. To approach the "k+1" lower bound with computational efficiency even when the rank of signal matrix is smaller than k, we propose a greedy algorithm called Two-stage orthogonal Subspace Matching Pursuit (TSMP) whose theoretical results approach the lower bound with less restriction than the Orthogonal Subspace Matching Pursuit (OSMP) and Subspace-Augmented MUltiple SIgnal Classification (SA-MUSIC) algorithms. We provide non-asymptotical performance guarantees of OSMP and TSMP by covering both noiseless and noisy cases. Variants of restricted isometry property and mutual coherence are used to improve the performance guarantees. Numerical simulations demonstrate that the proposed scheme has low complexity and outperforms most existing greedy methods. This shows that the minimum number of measurements for the success of TSMP converges more rapidly to the lower bound than the existing methods as the number of columns of the signal matrix increases.

cs.IT↗

arXiv2015

On the DoF region of the two-user Interference Channel with an Instantaneous Relay

This paper studies the Degrees of Freedom (DoF) of the two-user multi-antenna Gaussian interference channel with an {\em instantaneous relay}, or relay without delay, where the relay transmitted signal in channel use $t$ can depend on all received signals up to and including that at channel use $t$. It is assumed that the two transmitters and the two receivers have $M$ antennas, while the relay receives through $N$ antennas and transmits through $L$ antennas. An achievable DoF region is derived, for all possible values of $(M,N,L)$, based on a memoryless linear transmission strategy at the relay that aims to {\it neutralize} as much interference as possible at the receivers. The proposed scheme is shown to attain the largest sum DoF among all memoryless linear transmission strategies at the relay and to actually be optimal for certain values of $(M,N,L)$.

cs.IT↗

arXiv2015

A Unified Approach for Network Information Theory

In this paper, we take a unified approach for network information theory and prove a coding theorem, which can recover most of the achievability results in network information theory that are based on random coding. The final single-letter expression has a very simple form, which was made possible by many novel elements such as a unified framework that represents various network problems in a simple and unified way, a unified coding strategy that consists of a few basic ingredients but can emulate many known coding techniques if needed, and new proof techniques beyond the use of standard covering and packing lemmas. For example, in our framework, sources, channels, states and side information are treated in a unified way and various constraints such as cost and distortion constraints are unified as a single joint-typicality constraint. Our theorem can be useful in proving many new achievability results easily and in some cases gives simpler rate expressions than those obtained using conventional approaches. Furthermore, our unified coding can strictly outperform existing schemes. For example, we obtain a generalized decode-compress-amplify-and-forward bound as a simple corollary of our main theorem and show it strictly outperforms previously known coding schemes. Using our unified framework, we formally define and characterize three types of network duality based on channel input-output reversal and network flow reversal combined with packing-covering duality.

cs.IT↗

arXiv2015

Noisy Network Coding with Partial DF

In this paper, we propose a noisy network coding integrated with partial decode-and-forward relaying for single-source multicast discrete memoryless networks (DMN's). Our coding scheme generalizes the partial-decode-compress-and-forward scheme (Theorem 7) by Cover and El Gamal. This is the first time the theorem is generalized for DMN's such that each relay performs both partial decode-and-forward and compress-and-forward simultaneously. Our coding scheme simultaneously generalizes both noisy network coding by Lim, Kim, El Gamal, and Chung and distributed decode-and-forward by Lim, Kim, and Kim. It is not trivial to combine the two schemes because of inherent incompatibility in their encoding and decoding strategies. We solve this problem by sending the same long message over multiple blocks at the source and at the same time by letting the source find the auxiliary covering indices that carry information about the message simultaneously over all blocks.

cs.IT↗

arXiv2013

Degrees of Freedom of the Rank-deficient Interference Channel with Feedback

We investigate the total degrees of freedom (DoF) of the K-user rank-deficient interference channel with feedback. For the two-user case, we characterize the total DoF by developing an achievable scheme and deriving a matching upper bound. For the three-user case, we develop a new achievable scheme which employs interference alignment to efficiently utilize the dimension of the received signal space. In addition, we derive an upper bound for the general K-user case and show the tightness of the bound when the number of antennas at each node is sufficiently large. As a consequence of these results, we show that feedback can increase the DoF when the number of antennas at each node is large enough as compared to the ranks of channel matrices. This finding is in contrast to the full-rank interference channel where feedback provides no DoF gain. The gain comes from using feedback to provide alternative signal paths, thereby effectively increasing the ranks of desired channel matrices.

cs.IT↗