arXiv subjects

Lei Yu

Publications and source records attributed to Lei Yu.

At least 253 records · Page 14Linked to original sources

arXiv2019

CRNet: Image Super-Resolution Using A Convolutional Sparse Coding Inspired Network

Convolutional Sparse Coding (CSC) has been attracting more and more attention in recent years, for making full use of image global correlation to improve performance on various computer vision applications. However, very few studies focus on solving CSC based image Super-Resolution (SR) problem. As a consequence, there is no significant progress in this area over a period of time. In this paper, we exploit the natural connection between CSC and Convolutional Neural Networks (CNN) to address CSC based image SR. Specifically, Convolutional Iterative Soft Thresholding Algorithm (CISTA) is introduced to solve CSC problem and it can be implemented using CNN architectures. Then we develop a novel CSC based SR framework analogy to the traditional SC based SR methods. Two models inspired by this framework are proposed for pre-/post-upsampling SR, respectively. Compared with recent state-of-the-art SR methods, both of our proposed models show superior performance in terms of both quantitative and qualitative measurements.

eess.IV↗

arXiv2019

On Conditional Correlations

The Pearson correlation, correlation ratio, and maximal correlation have been well-studied in the literature. In this paper, we study the conditional versions of these quantities. We extend the most important properties of the unconditional versions to the conditional versions, and also derive some new properties. Based on the conditional maximal correlation, we define an information-correlation function of two arbitrary random variables, and use it to derive an impossibility result for the problem of the non-interactive simulation of random variables.

math.PR↗

arXiv2019

Facial Feature Embedded CycleGAN for VIS-NIR Translation

VIS-NIR face recognition remains a challenging task due to the distinction between spectral components of two modalities and insufficient paired training data. Inspired by the CycleGAN, this paper presents a method aiming to translate VIS face images into fake NIR images whose distributions are intended to approximate those of true NIR images, which is achieved by proposing a new facial feature embedded CycleGAN. Firstly, to learn the particular feature of NIR domain while preserving common facial representation between VIS and NIR domains, we employ a general facial feature extractor (FFE) to replace the encoder in the original generator of CycleGAN. For implementing the facial feature extractor, herein the MobileFaceNet is pretrained on a VIS face database, and is able to extract effective features. Secondly, the domain-invariant feature learning is enhanced by considering a new pixel consistency loss. Lastly, we establish a new WHU VIS-NIR database which varies in face rotation and expressions to enrich the training data. Experimental results on the Oulu-CASIA NIR-VIS database and the WHU VIS-NIR database show that the proposed FFE-based CycleGAN (FFE-CycleGAN) outperforms state-of-the-art VIS-NIR face recognition methods and achieves 96.5\% accuracy.

cs.CV↗

arXiv2019

Diverse Exploration via Conjugate Policies for Policy Gradient Methods

We address the challenge of effective exploration while maintaining good performance in policy gradient methods. As a solution, we propose diverse exploration (DE) via conjugate policies. DE learns and deploys a set of conjugate policies which can be conveniently generated as a byproduct of conjugate gradient descent. We provide both theoretical and empirical results showing the effectiveness of DE at achieving exploration, improving policy performance, and the advantage of DE over exploration by random policy perturbations.

cs.LG↗

arXiv2019

Towards Demystifying Membership Inference Attacks

Membership inference attacks seek to infer membership of individual training instances of a model to which an adversary has black-box access through a machine learning-as-a-service API. In providing an in-depth characterization of membership privacy risks against machine learning models, this paper presents a comprehensive study towards demystifying membership inference attacks from two complimentary perspectives. First, we provide a generalized formulation of the development of a black-box membership inference attack model. Second, we characterize the importance of model choice on model vulnerability through a systematic evaluation of a variety of machine learning models and model combinations using multiple datasets. Through formal analysis and empirical evidence from extensive experimentation, we characterize under what conditions a model may be vulnerable to such black-box membership inference attacks. We show that membership inference vulnerability is data-driven and corresponding attack models are largely transferable. Though different model types display different vulnerabilities to membership inference, so do different datasets. Our empirical results additionally show that (1) using the type of target model under attack within the attack model may not increase attack effectiveness and (2) collaborative learning exposes vulnerabilities to membership inference risks when the adversary is a participant. We also discuss countermeasure and mitigation strategies.

cs.CR↗

arXiv2019

Learning and Evaluating General Linguistic Intelligence

We define general linguistic intelligence as the ability to reuse previously acquired knowledge about a language's lexicon, syntax, semantics, and pragmatic conventions to adapt to new tasks quickly. Using this definition, we analyze state-of-the-art natural language understanding models and conduct an extensive empirical investigation to evaluate them against these criteria through a series of experiments that assess the task-independence of the knowledge being acquired by the learning process. In addition to task performance, we propose a new evaluation metric based on an online encoding of the test data that quantifies how quickly an existing agent (model) learns a new task. Our results show that while the field has made impressive progress in terms of model architectures that generalize to many tasks, these models still require a lot of in-domain training examples (e.g., for fine tuning, training task-specific modules), and are prone to catastrophic forgetting. Moreover, we find that far from solving general tasks (e.g., document question answering), our models are overfitting to the quirks of particular datasets (e.g., SQuAD). We discuss missing components and conjecture on how to make progress toward general linguistic intelligence.

cs.LG↗

arXiv2019

Variational Smoothing in Recurrent Neural Network Language Models

We present a new theoretical perspective of data noising in recurrent neural network language models (Xie et al., 2017). We show that each variant of data noising is an instance of Bayesian recurrent neural networks with a particular variational distribution (i.e., a mixture of Gaussians whose weights depend on statistics derived from the corpus such as the unigram distribution). We use this insight to propose a more principled method to apply at prediction time and propose natural extensions to data noising under the variational framework. In particular, we propose variational smoothing with tied input and output embedding matrices and an element-wise variational smoothing method. We empirically verify our analysis on two benchmark language modeling datasets and demonstrate performance improvements over existing data noising methods.

cs.CL↗

arXiv2018

Image Super-Resolution via RL-CSC: When Residual Learning Meets Convolutional Sparse Coding

We propose a simple yet effective model for Single Image Super-Resolution (SISR), by combining the merits of Residual Learning and Convolutional Sparse Coding (RL-CSC). Our model is inspired by the Learned Iterative Shrinkage-Threshold Algorithm (LISTA). We extend LISTA to its convolutional version and build the main part of our model by strictly following the convolutional form, which improves the network's interpretability. Specifically, the convolutional sparse codings of input feature maps are learned in a recursive manner, and high-frequency information can be recovered from these CSCs. More importantly, residual learning is applied to alleviate the training difficulty when the network goes deeper. Extensive experiments on benchmark datasets demonstrate the effectiveness of our method. RL-CSC (30 layers) outperforms several recent state-of-the-arts, e.g., DRRN (52 layers) and MemNet (80 layers) in both accuracy and visual qualities. Codes and more results are available at https://github.com/axzml/RL-CSC.

cs.CV↗

arXiv2018

Adversarial Examples in Deep Learning: Characterization and Divergence

The burgeoning success of deep learning has raised the security and privacy concerns as more and more tasks are accompanied with sensitive data. Adversarial attacks in deep learning have emerged as one of the dominating security threat to a range of mission-critical deep learning systems and applications. This paper takes a holistic and principled approach to perform statistical characterization of adversarial examples in deep learning. We provide a general formulation of adversarial examples and elaborate on the basic principle for adversarial attack algorithm design. We introduce easy and hard categorization of adversarial attacks to analyze the effectiveness of adversarial examples in terms of attack success rate, degree of change in adversarial perturbation, average entropy of prediction qualities, and fraction of adversarial examples that lead to successful attacks. We conduct extensive experimental study on adversarial behavior in easy and hard attacks under deep learning models with different hyperparameters and different deep learning frameworks. We show that the same adversarial attack behaves differently under different hyperparameters and across different frameworks due to the different features learned under different deep learning model training process. Our statistical characterization with strong empirical evidence provides a transformative enlightenment on mitigation strategies towards effective countermeasures against present and future adversarial attacks.

cs.LG↗

arXiv2018

Simulation of Random Variables under Rényi Divergence Measures of All Orders

The random variable simulation problem consists in using a $k$-dimensional i.i.d. random vector $X^{k}$ with distribution $P_{X}^{k}$ to simulate an $n$-dimensional i.i.d. random vector $Y^{n}$ so that its distribution is approximately $Q_{Y}^{n}$. In contrast to previous works, in this paper we consider the standard Rényi divergence and two variants of all orders to measure the level of approximation. These two variants are the max-Rényi divergence $D_α^{\mathsf{max}}(P,Q)$ and the sum-Rényi divergence $D_α^{+}(P,Q)$. When $α=\infty$, these two measures are strong because for any $ε>0$, $D_{\infty}^{\mathsf{max}}(P,Q)\leqε$ or $D_{\infty}^{+}(P,Q)\leqε$ implies $e^{-ε}\leq\frac{P(x)}{Q(x)}\leq e^ε$ for all $x$. Under these Rényi divergence measures, we characterize the asymptotics of normalized divergences as well as the Rényi conversion rates. The latter is defined as the supremum of $\frac{n}{k}$ such that the Rényi divergences vanish asymptotically. In addition, when the Rényi parameter is in the interval $(0,1)$, the Rényi conversion rates equal the ratio of the Shannon entropies $\frac{H\left(P_{X}\right)}{H\left(Q_{Y}\right)}$, which is consistent with traditional results in which the total variation measure was adopted. When the Rényi parameter is in the interval $(1,\infty]$, the Rényi conversion rates are, in general, smaller than $\frac{H\left(P_{X}\right)}{H\left(Q_{Y}\right)}$. When specialized to the case in which either $P_{X}$ or $Q_{Y}$ is uniform, the simulation problem reduces to the source resolvability and intrinsic randomness problems. The preceding results are used to characterize the asymptotics of Rényi divergences and the Rényi conversion rates for these two cases.

cs.IT↗

arXiv2018

Beyond the Central Limit Theorem: Universal and Non-universal Simulations of Random Variables by General Mappings

Motivated by the Central Limit Theorem, in this paper, we study both universal and non-universal simulations of random variables with an arbitrary target distribution $Q_{Y}$ by general mappings, not limited to linear ones (as in the Central Limit Theorem). We derive the fastest convergence rate of the approximation errors for such problems. Interestingly, we show that for discontinuous or absolutely continuous $P_{X}$, the approximation error for the universal simulation is almost as small as that for the non-universal one; and moreover, for both universal and non-universal simulations, the approximation errors by general mappings are strictly smaller than those by linear mappings. Furthermore, we also generalize these results to simulation from Markov processes, and simulation of random elements (or general random variables).

math.PR↗

arXiv2018

Rényi Resolvability and Its Applications to the Wiretap Channel

The conventional channel resolvability problem refers to the determination of the minimum rate required for an input process so that the output distribution approximates a target distribution in either the total variation distance or the relative entropy. In contrast to previous works, in this paper, we use the (normalized or unnormalized) Rényi divergence (with the Rényi parameter in $[0,2]\cup\{\infty\}$) to measure the level of approximation. We also provide asymptotic expressions for normalized Rényi divergence when the Rényi parameter is larger than or equal to $1$ as well as (lower and upper) bounds for the case when the same parameter is smaller than $1$. We characterize the Rényi resolvability, which is defined as the minimum rate required to ensure that the Rényi divergence vanishes asymptotically. The Rényi resolvabilities are the same for both the normalized and unnormalized divergence cases. In addition, when the Rényi parameter smaller than~$1$, consistent with the traditional case where the Rényi parameter is equal to~$1$, the Rényi resolvability equals the minimum mutual information over all input distributions that induce the target output distribution. When the Rényi parameter is larger than $1$ the Rényi resolvability is, in general, larger than the mutual information. The optimal Rényi divergence is proven to vanish at least exponentially fast for both of these two cases, as long as the code rate is larger than the Rényi resolvability. The optimal exponential rate of decay for i.i.d.\ random codes is also characterized exactly. We apply these results to the wiretap channel, and completely characterize the optimal tradeoff between the rates of the secret and non-secret messages when the leakage measure is given by the (unnormalized) Rényi divergence.

cs.IT↗

arXiv2018

Sentence Encoding with Tree-constrained Relation Networks

The meaning of a sentence is a function of the relations that hold between its words. We instantiate this relational view of semantics in a series of neural models based on variants of relation networks (RNs) which represent a set of objects (for us, words forming a sentence) in terms of representations of pairs of objects. We propose two extensions to the basic RN model for natural language. First, building on the intuition that not all word pairs are equally informative about the meaning of a sentence, we use constraints based on both supervised and unsupervised dependency syntax to control which relations influence the representation. Second, since higher-order relations are poorly captured by a sum of pairwise relations, we use a recurrent extension of RNs to propagate information so as to form representations of higher order relations. Experiments on sentence classification, sentence pair classification, and machine translation reveal that, while basic RNs are only modestly effective for sentence representation, recurrent RNs with latent syntax are a reliably powerful representational device.

cs.CL↗

arXiv2018

Tackling Sequence to Sequence Mapping Problems with Neural Networks

In Natural Language Processing (NLP), it is important to detect the relationship between two sequences or to generate a sequence of tokens given another observed sequence. We call the type of problems on modelling sequence pairs as sequence to sequence (seq2seq) mapping problems. A lot of research has been devoted to finding ways of tackling these problems, with traditional approaches relying on a combination of hand-crafted features, alignment models, segmentation heuristics, and external linguistic resources. Although great progress has been made, these traditional approaches suffer from various drawbacks, such as complicated pipeline, laborious feature engineering, and the difficulty for domain adaptation. Recently, neural networks emerged as a promising solution to many problems in NLP, speech recognition, and computer vision. Neural models are powerful because they can be trained end to end, generalise well to unseen examples, and the same framework can be easily adapted to a new domain. The aim of this thesis is to advance the state-of-the-art in seq2seq mapping problems with neural networks. We explore solutions from three major aspects: investigating neural models for representing sequences, modelling interactions between sequences, and using unpaired data to boost the performance of neural models. For each aspect, we propose novel models and evaluate their efficacy on various tasks of seq2seq mapping.

cs.CL↗

arXiv2018

Analog-to-digital conversion revolutionized by deep learning

As the bridge between the analog world and digital computers, analog-to-digital converters are generally used in modern information systems such as radar, surveillance, and communications. For the configuration of analog-to-digital converters in future high-frequency broadband systems, we introduce a revolutionary architecture that adopts deep learning technology to overcome tradeoffs between bandwidth, sampling rate, and accuracy. A photonic front-end provides broadband capability for direct sampling and speed multiplication. Trained deep neural networks learn the patterns of system defects, maintaining high accuracy of quantized data in a succinct and adaptive manner. Based on numerical and experimental demonstrations, we show that the proposed architecture outperforms state-of-the-art analog-to-digital converters, confirming the potential of our approach in future analog-to-digital converter design and performance enhancement of future information systems.

eess.SP↗

arXiv2018

Codegree threshold for tiling $k$-graphs with two edges sharing exactly $\ell$ vertices

Given integer $k$ and a $k$-graph $F$, let $t_{k-1}(n,F)$ be the minimum integer $t$ such that every $k$-graph $H$ on $n$ vertices with codegree at least $t$ contains an $F$-factor. For integers $k\geq3$ and $0\leq\ell\leq k-1$, let $\mathcal{Y}_{k,\ell}$ be a $k$-graph with two edges that shares exactly $\ell$ vertices. Han and Zhao (JCTA, 2015) asked the following question: For all $k\ge 3$, $0\le \ell\le k-1$ and sufficiently large $n$ divisible by $2k-\ell$, determine the exact value of $t_{k-1}(n,\mathcal{Y}_{k,\ell})$. In this paper, we show that $t_{k-1}(n,\mathcal{Y}_{k,\ell})=\frac{n}{2k-\ell}$ for $k\geq3$ and $1\leq\ell\leq k-2$, combining with two previously known results of Rödl, Ruciński and Szemerédi {(JCTA, 2009)} and Gao, Han and Zhao (arXiv, 2016), the question of Han and Zhao is solved completely.

math.CO↗

arXiv2018

Distortion Bounds for Source Broadcast Problems

This paper investigates the joint source-channel coding problem of sending a memoryless source over a memoryless broadcast channel. An inner bound and several outer bounds on the admissible distortion region are derived, which respectively generalize and unify several existing bounds. As a consequence, we also obtain an inner bound and an outer bound for the degraded broadcast channel case. When specialized to the Gaussian or binary source broadcast, the inner bound and outer bound not only recover the best known inner bound and outer bound in the literature, but also generate some new results. Besides, we also extend the inner bound and outer bounds to the Wyner-Ziv source broadcast problem, i.e., source broadcast with side information available at decoders. Some new bounds are obtained when specialized to the Wyner-Ziv Gaussian and Wyner-Ziv binary cases.

cs.IT↗

arXiv2018

Diverse Exploration for Fast and Safe Policy Improvement

We study an important yet under-addressed problem of quickly and safely improving policies in online reinforcement learning domains. As its solution, we propose a novel exploration strategy - diverse exploration (DE), which learns and deploys a diverse set of safe policies to explore the environment. We provide DE theory explaining why diversity in behavior policies enables effective exploration without sacrificing exploitation. Our empirical study shows that an online policy improvement algorithm framework implementing the DE strategy can achieve both fast policy improvement and safe online performance.

cs.LG↗