arXiv subjects

Jun Zhu

Publications and source records attributed to Jun Zhu.

At least 415 records · Page 23Linked to original sources

arXiv2020

Learning Implicit Generative Models by Teaching Explicit Ones

Implicit generative models are difficult to train as no explicit density functions are defined. Generative adversarial nets (GANs) present a minimax framework to train such models, which however can suffer from mode collapse due to the nature of the JS-divergence. This paper presents a learning by teaching (LBT) approach to learning implicit models, which intrinsically avoids the mode collapse problem by optimizing a KL-divergence rather than the JS-divergence in GANs. In LBT, an auxiliary density estimator is introduced to fit the implicit model's distribution while the implicit model teaches the density estimator to match the data distribution. LBT is formulated as a bilevel optimization problem, whose optimal generator matches the true data distribution. LBT can be naturally integrated with GANs to derive a hybrid LBT-GAN that enjoys complimentary benefits. Finally, we present a stochastic gradient ascent algorithm with unrolling to solve the challenging learning problems. Experimental results demonstrate the effectiveness of our method.

cs.LG↗

arXiv2020

Absolutely convergent fixed-point fast sweeping WENO methods for steady state of hyperbolic conservation laws

Fixed-point iterative sweeping methods were developed in the literature to efficiently solve steady state solutions of Hamilton-Jacobi equations and hyperbolic conservation laws. Similar as other fast sweeping schemes, the key components of this class of methods are the Gauss-Seidel iterations and alternating sweeping strategy to achieve fast convergence rate. Furthermore, good properties of fixed-point iterative sweeping methods include that they have explicit forms and do not involve inverse operation of nonlinear local systems, and they can be applied to general hyperbolic equations using any monotone numerical fluxes and high order approximations easily. In [L. Wu, Y.-T. Zhang, S. Zhang and C.-W. Shu, Commun. Comput. Phys., 20 (2016)], a fifth order fixed-point sweeping WENO scheme was designed and it was shown that the scheme converges much faster than the total variation diminishing (TVD) Runge-Kutta approach by stability improvement of high order schemes with a forward Euler time-marching. An open problem is that for some benchmark numerical examples, the iteration residue of the fixed-point sweeping WENO scheme hangs at a truncation error level instead of settling down to machine zero. This issue makes it difficult to determine the convergence criterion for the iteration and challenging to apply the method to complex problems. To solve this issue, in this paper we apply the multi-resolution WENO scheme developed in [J. Zhu and C.-W. Shu, J. Comput. Phys., 375 (2018)] to the fifth order fixed-point sweeping WENO scheme and obtain an absolutely convergent fixed-point fast sweeping method for steady state of hyperbolic conservation laws, i.e., the residue of the fast sweeping iterations converges to machine zero / round off errors for all benchmark problems tested.

math.NA↗

arXiv2020

Dynamic Window-level Granger Causality of Multi-channel Time Series

Granger causality method analyzes the time series causalities without building a complex causality graph. However, the traditional Granger causality method assumes that the causalities lie between time series channels and remain constant, which cannot model the real-world time series data with dynamic causalities along the time series channels. In this paper, we present the dynamic window-level Granger causality method (DWGC) for multi-channel time series data. We build the causality model on the window-level by doing the F-test with the forecasting errors on the sliding windows. We propose the causality indexing trick in our DWGC method to reweight the original time series data. Essentially, the causality indexing is to decrease the auto-correlation and increase the cross-correlation causal effects, which improves the DWGC method. Theoretical analysis and experimental results on two synthetic and one real-world datasets show that the improved DWGC method with causality indexing better detects the window-level causalities.

stat.ME↗

arXiv2020

Superconducting proximity effect in a transparent van der Waals superconductor-metal junction

We report on Andreev reflections at clean NbSe2-bilayer graphene junctions. The high transparency of the junction, which manifests as a large conductance enhancement of up to 1.8, enables us to see clear evidence of a proximity-induced superconducting gap in bilayer graphene and two Andreev reflections through a vertical NbSe2-graphene and a lateral graphene-graphene junction respectively. Quantum transport simulations capture the complexity of the experimental data and illuminate the impact of various microscopic parameters on the transmission of the junction. Our work establishes the practice and understanding of an all-van-der-Waals, high-performance superconducting junction. The realization of a highly transparent proximized graphene-graphene junction opens up possibilities to engineer emergent quantum phenomena.

cond-mat.mes-hall↗

arXiv2020

Triple Memory Networks: a Brain-Inspired Method for Continual Learning

Continual acquisition of novel experience without interfering previously learned knowledge, i.e. continual learning, is critical for artificial neural networks, but limited by catastrophic forgetting. A neural network adjusts its parameters when learning a new task, but then fails to conduct the old tasks well. By contrast, the brain has a powerful ability to continually learn new experience without catastrophic interference. The underlying neural mechanisms possibly attribute to the interplay of hippocampus-dependent memory system and neocortex-dependent memory system, mediated by prefrontal cortex. Specifically, the two memory systems develop specialized mechanisms to consolidate information as more specific forms and more generalized forms, respectively, and complement the two forms of information in the interplay. Inspired by such brain strategy, we propose a novel approach named triple memory networks (TMNs) for continual learning. TMNs model the interplay of hippocampus, prefrontal cortex and sensory cortex (a neocortex region) as a triple-network architecture of generative adversarial networks (GAN). The input information is encoded as specific representation of the data distributions in a generator, or generalized knowledge of solving tasks in a discriminator and a classifier, with implementing appropriate brain-inspired algorithms to alleviate catastrophic forgetting in each module. Particularly, the generator replays generated data of the learned tasks to the discriminator and the classifier, both of which are implemented with a weight consolidation regularizer to complement the lost information in generation process. TMNs achieve new state-of-the-art performance on a variety of class-incremental learning benchmarks on MNIST, SVHN, CIFAR-10 and ImageNet-50, comparing with strong baseline methods.

cs.LG↗

arXiv2020

Rethinking Softmax Cross-Entropy Loss for Adversarial Robustness

Previous work shows that adversarially robust generalization requires larger sample complexity, and the same dataset, e.g., CIFAR-10, which enables good standard accuracy may not suffice to train robust models. Since collecting new training data could be costly, we focus on better utilizing the given data by inducing the regions with high sample density in the feature space, which could lead to locally sufficient samples for robust learning. We first formally show that the softmax cross-entropy (SCE) loss and its variants convey inappropriate supervisory signals, which encourage the learned feature points to spread over the space sparsely in training. This inspires us to propose the Max-Mahalanobis center (MMC) loss to explicitly induce dense feature regions in order to benefit robustness. Namely, the MMC loss encourages the model to concentrate on learning ordered and compact representations, which gather around the preset optimal centers for different classes. We empirically demonstrate that applying the MMC loss can significantly improve robustness even under strong adaptive attacks, while keeping state-of-the-art accuracy on clean inputs with little extra computation compared to the SCE loss.

cs.LG↗

arXiv2020

Mixup Inference: Better Exploiting Mixup to Defend Adversarial Attacks

It has been widely recognized that adversarial examples can be easily crafted to fool deep networks, which mainly root from the locally non-linear behavior nearby input examples. Applying mixup in training provides an effective mechanism to improve generalization performance and model robustness against adversarial perturbations, which introduces the globally linear behavior in-between training examples. However, in previous work, the mixup-trained models only passively defend adversarial attacks in inference by directly classifying the inputs, where the induced global linearity is not well exploited. Namely, since the locality of the adversarial perturbations, it would be more efficient to actively break the locality via the globality of the model predictions. Inspired by simple geometric intuition, we develop an inference principle, named mixup inference (MI), for mixup-trained models. MI mixups the input with other random clean samples, which can shrink and transfer the equivalent perturbation if the input is adversarial. Our experiments on CIFAR-10 and CIFAR-100 demonstrate that MI can further improve the adversarial robustness for the models trained by mixup and its variants.

cs.LG↗

arXiv2020

A Wasserstein Minimum Velocity Approach to Learning Unnormalized Models

Score matching provides an effective approach to learning flexible unnormalized models, but its scalability is limited by the need to evaluate a second-order derivative. In this paper, we present a scalable approximation to a general family of learning objectives including score matching, by observing a new connection between these objectives and Wasserstein gradient flows. We present applications with promise in learning neural density estimators on manifolds, and training implicit variational and Wasserstein auto-encoders with a manifold-valued prior.

stat.ML↗

arXiv2020

To Relieve Your Headache of Training an MRF, Take AdVIL

We propose a black-box algorithm called {\it Adversarial Variational Inference and Learning} (AdVIL) to perform inference and learning on a general Markov random field (MRF). AdVIL employs two variational distributions to approximately infer the latent variables and estimate the partition function of an MRF, respectively. The two variational distributions provide an estimate of the negative log-likelihood of the MRF as a minimax optimization problem, which is solved by stochastic gradient descent. AdVIL is proven convergent under certain conditions. On one hand, compared with contrastive divergence, AdVIL requires a minimal assumption about the model structure and can deal with a broader family of MRFs. On the other hand, compared with existing black-box methods, AdVIL provides a tighter estimate of the log partition function and achieves much better empirical results.

cs.LG↗

arXiv2020

Analyzing the Noise Robustness of Deep Neural Networks

Adversarial examples, generated by adding small but intentionally imperceptible perturbations to normal examples, can mislead deep neural networks (DNNs) to make incorrect predictions. Although much work has been done on both adversarial attack and defense, a fine-grained understanding of adversarial examples is still lacking. To address this issue, we present a visual analysis method to explain why adversarial examples are misclassified. The key is to compare and analyze the datapaths of both the adversarial and normal examples. A datapath is a group of critical neurons along with their connections. We formulate the datapath extraction as a subset selection problem and solve it by constructing and training a neural network. A multi-level visualization consisting of a network-level visualization of data flows, a layer-level visualization of feature maps, and a neuron-level visualization of learned features, has been designed to help investigate how datapaths of adversarial and normal examples diverge and merge in the prediction process. A quantitative evaluation and a case study were conducted to demonstrate the promise of our method to explain the misclassification of adversarial examples.

cs.HC↗

arXiv2019

Benchmarking Adversarial Robustness

Deep neural networks are vulnerable to adversarial examples, which becomes one of the most important research problems in the development of deep learning. While a lot of efforts have been made in recent years, it is of great significance to perform correct and complete evaluations of the adversarial attack and defense algorithms. In this paper, we establish a comprehensive, rigorous, and coherent benchmark to evaluate adversarial robustness on image classification tasks. After briefly reviewing plenty of representative attack and defense methods, we perform large-scale experiments with two robustness curves as the fair-minded evaluation criteria to fully understand the performance of these methods. Based on the evaluation results, we draw several important findings and provide insights for future research.

cs.CV↗

arXiv2019

Control variates and Rao-Blackwellization for deterministic sweep Markov chains

We study control variate methods for Markov chain Monte Carlo (MCMC) in the setting of deterministic sweep sampling using $K\geq 2$ transition kernels. New variance reduction results are provided for MCMC averages based on sweeps over general transition kernels, leading to a particularly simple control variate estimator in the setting of deterministic sweep Gibbs sampling. Theoretical comparisons of our proposed control variate estimators with existing literature are made, and a simulation study is performed to examine the amount of variance reduction in some example cases. We also relate control variate approaches to approaches based on conditioning (or Rao-Blackwellization), and show that the latter can be viewed as an approximation of the former. Our theoretical results hold for Markov chains under standard geometric drift assumptions.

math.ST↗

arXiv2019

The Search for Sparse, Robust Neural Networks

Recent work on deep neural network pruning has shown there exist sparse subnetworks that achieve equal or improved accuracy, training time, and loss using fewer network parameters when compared to their dense counterparts. Orthogonal to pruning literature, deep neural networks are known to be susceptible to adversarial examples, which may pose risks in security- or safety-critical applications. Intuition suggests that there is an inherent trade-off between sparsity and robustness such that these characteristics could not co-exist. We perform an extensive empirical evaluation and analysis testing the Lottery Ticket Hypothesis with adversarial training and show this approach enables us to find sparse, robust neural networks. Code for reproducing experiments is available here: https://github.com/justincosentino/robust-sparse-networks.

cs.LG↗

arXiv2019

Landau Levels of Bilayer Graphene in a WSe$_2$/Bilayer Graphene van der Waals Heterostructure

Heterostructures formed between two different van der Waals materials enable interactions and functionalities absent in each component. In this work we show that vicinity to an atomically thin WSe$_2$ sheet dramatically impacts the energies of the symmetry-broken low Landau levels of bilayer graphene, possibly due to screening. We present a systematic study of the magnetic field and electrical displacement field dependences of the Landau level gaps at filling factor $ν$ = 1, 2, 3, and compare to BN encapsulated pristine bilayer graphene. The exchange-dominated energy splitting between the N = 0 and 1 orbital wave functions is significantly enhanced, which leads to a modified phase diagram at filling factor $ν$ = 0 and larger energy gaps at $ν$ = 1 and 3 in WSe$_2$/bilayer graphene heterostructures. The exchange-enhanced spin gap at $ν$ = 2, on the other hand, is reduced by approximately two-fold. Our results demonstrate a possible way to engineer quantum Hall phenomena via van der Waals heterostructures.

cond-mat.mes-hall↗

arXiv2019

Ferromagnetic van der Waals compound MnSb$_{1.8}$Bi$_{0.2}$Te$_4$

The intersection of topology and magnetism represents a new playground to discover novel quantum phenomena and device concepts. In this work, we show that a van der Waals compound MnSb$_{1.8}$Bi$_{0.2}$Te$_4$ exhibits a ferromagnetic ground state with a Curie temperature of 26 K, in contrast to the antiferromagnetic order previously found for other members of the Mn(Sb, Bi)$_2$Te$_4$ family. We employ magneto-transport, bulk magnetization and neutron scattering studies to illustrate the magnetic and electrical properties of MnSb$_{1.8}$Bi$_{0.2}$Te$_4$ and report on the observation of an unusual anomalous Hall effect. Our results are an important step in the synthesis and understanding of ferromagnetic topological insulators.

cond-mat.mes-hall↗

arXiv2019

Generative Well-intentioned Networks

We propose Generative Well-intentioned Networks (GWINs), a novel framework for increasing the accuracy of certainty-based, closed-world classifiers. A conditional generative network recovers the distribution of observations that the classifier labels correctly with high certainty. We introduce a reject option to the classifier during inference, allowing the classifier to reject an observation instance rather than predict an uncertain label. These rejected observations are translated by the generative network to high-certainty representations, which are then relabeled by the classifier. This architecture allows for any certainty-based classifier or rejection function and is not limited to multilayer perceptrons. The capability of this framework is assessed using benchmark classification datasets and shows that GWINs significantly improve the accuracy of uncertain observations.

cs.LG↗

arXiv2019

Cluster Alignment with a Teacher for Unsupervised Domain Adaptation

Deep learning methods have shown promise in unsupervised domain adaptation, which aims to leverage a labeled source domain to learn a classifier for the unlabeled target domain with a different distribution. However, such methods typically learn a domain-invariant representation space to match the marginal distributions of the source and target domains, while ignoring their fine-level structures. In this paper, we propose Cluster Alignment with a Teacher (CAT) for unsupervised domain adaptation, which can effectively incorporate the discriminative clustering structures in both domains for better adaptation. Technically, CAT leverages an implicit ensembling teacher model to reliably discover the class-conditional structure in the feature space for the unlabeled target domain. Then CAT forces the features of both the source and the target domains to form discriminative class-conditional clusters and aligns the corresponding clusters across domains. Empirical results demonstrate that CAT achieves state-of-the-art results in several unsupervised domain adaptation scenarios.

cs.CV↗

arXiv2019

A Simple yet Effective Baseline for Robust Deep Learning with Noisy Labels

Recently deep neural networks have shown their capacity to memorize training data, even with noisy labels, which hurts generalization performance. To mitigate this issue, we provide a simple but effective baseline method that is robust to noisy labels, even with severe noise. Our objective involves a variance regularization term that implicitly penalizes the Jacobian norm of the neural network on the whole training set (including the noisy-labeled data), which encourages generalization and prevents overfitting to the corrupted labels. Experiments on both synthetically generated incorrect labels and realistic large-scale noisy datasets demonstrate that our approach achieves state-of-the-art performance with a high tolerance to severe noise.

cs.LG↗