arXiv subjects

Yanjun Han

Publications and source records attributed to Yanjun Han.

At least 73 records · Page 4Linked to original sources

arXiv2020

Bias Correction with Jackknife, Bootstrap, and Taylor Series

We analyze bias correction methods using jackknife, bootstrap, and Taylor series. We focus on the binomial model, and consider the problem of bias correction for estimating $f(p)$, where $f \in C[0,1]$ is arbitrary. We characterize the supremum norm of the bias of general jackknife and bootstrap estimators for any continuous functions, and demonstrate the in delete-$d$ jackknife, different values of $d$ may lead to drastically different behaviors in jackknife. We show that in the binomial model, iterating the bootstrap bias correction infinitely many times may lead to divergence of bias and variance, and demonstrate that the bias properties of the bootstrap bias corrected estimator after $r-1$ rounds are of the same order as that of the $r$-jackknife estimator if a bounded coefficients condition is satisfied.

math.ST↗

arXiv2019

Optimal rates of entropy estimation over Lipschitz balls

We consider the problem of minimax estimation of the entropy of a density over Lipschitz balls. Dropping the usual assumption that the density is bounded away from zero, we obtain the minimax rates $(n\ln n)^{-s/(s+d)} + n^{-1/2}$ for $0<s\leq 2$ for densities supported on $[0,1]^d$, where $s$ is the smoothness parameter and $n$ is the number of independent samples. We generalize the results to densities with unbounded support: given an Orlicz functions $Ψ$ of rapid growth (such as the sub-exponential and sub-Gaussian classes), the minimax rates for densities with bounded $Ψ$-Orlicz norm increase to $(n\ln n)^{-s/(s+d)} (Ψ^{-1}(n))^{d(1-d/p(s+d))} + n^{-1/2}$, where $p$ is the norm parameter in the Lipschitz ball. We also show that the integral-form plug-in estimators with kernel density estimates fail to achieve the minimax rates, and characterize their worst case performances over the Lipschitz ball. One of the key steps in analyzing the bias relies on a novel application of the Hardy-Littlewood maximal inequality, which also leads to a new inequality on the Fisher information that may be of independent interest.

math.ST↗

arXiv2019

Batched Multi-armed Bandits Problem

In this paper, we study the multi-armed bandit problem in the batched setting where the employed policy must split data into a small number of batches. While the minimax regret for the two-armed stochastic bandits has been completely characterized in \cite{perchet2016batched}, the effect of the number of arms on the regret for the multi-armed case is still open. Moreover, the question whether adaptively chosen batch sizes will help to reduce the regret also remains underexplored. In this paper, we propose the BaSE (batched successive elimination) policy to achieve the rate-optimal regrets (within logarithmic factors) for batched multi-armed bandits, with matching lower bounds even if the batch sizes are determined in an adaptive manner.

stat.ML↗

arXiv2019

Domain Compression and its Application to Randomness-Optimal Distributed Goodness-of-Fit

We study goodness-of-fit of discrete distributions in the distributed setting, where samples are divided between multiple users who can only release a limited amount of information about their samples due to various information constraints. Recently, a subset of the authors showed that having access to a common random seed (i.e., shared randomness) leads to a significant reduction in the sample complexity of this problem. In this work, we provide a complete understanding of the interplay between the amount of shared randomness available, the stringency of information constraints, and the sample complexity of the testing problem by characterizing a tight trade-off between these three parameters. We provide a general distributed goodness-of-fit protocol that as a function of the amount of shared randomness interpolates smoothly between the private- and public-coin sample complexities. We complement our upper bound with a general framework to prove lower bounds on the sample complexity of this testing problems under limited shared randomness. Finally, we instantiate our bounds for the two archetypal information constraints of communication and local privacy, and show that our sample complexity bounds are optimal as a function of all the parameters of the problem, including the amount of shared randomness. A key component of our upper bounds is a new primitive of domain compression, a tool that allows us to map distributions to a much smaller domain size while preserving their pairwise distances, using a limited amount of randomness.

cs.DS↗

arXiv2019

Lower Bounds for Learning Distributions under Communication Constraints via Fisher Information

We consider the problem of learning high-dimensional, nonparametric and structured (e.g. Gaussian) distributions in distributed networks, where each node in the network observes an independent sample from the underlying distribution and can use $k$ bits to communicate its sample to a central processor. We consider three different models for communication. Under the independent model, each node communicates its sample to a central processor by independently encoding it into $k$ bits. Under the more general sequential or blackboard communication models, nodes can share information interactively but each node is restricted to write at most $k$ bits on the final transcript. We characterize the impact of the communication constraint $k$ on the minimax risk of estimating the underlying distribution under $\ell^2$ loss. We develop minimax lower bounds that apply in a unified way to many common statistical models and reveal that the impact of the communication constraint can be qualitatively different depending on the tail behavior of the score function associated with each model. A key ingredient in our proofs is a geometric characterization of Fisher information from quantized samples.

cs.IT↗

arXiv2019

Expectation of the Largest bet size in Labouchere System

For Labouchere system with winning probability $p$ at each coup, we prove that the expectation of the largest bet size under any initial list is finite if $p>\frac{1}{2}$, and is infinite if $p\le \frac{1}{2}$, solving the open conjecture in Grimmett and Stirzaker (2001). The same result holds for a general family of betting systems, and the proof builds upon a recursive representation of the optimal betting system in the larger family.

math.PR↗

arXiv2019

Adaptive Estimation of Shannon Entropy

We consider estimating the Shannon entropy of a discrete distribution $P$ from $n$ i.i.d. samples. Recently, Jiao, Venkat, Han, and Weissman, and Wu and Yang constructed approximation theoretic estimators that achieve the minimax $L_2$ rates in estimating entropy. Their estimators are consistent given $n \gg \frac{S}{\ln S}$ samples, where $S$ is the alphabet size, and it is the best possible sample complexity. In contrast, the Maximum Likelihood Estimator (MLE), which is the empirical entropy, requires $n\gg S$ samples. In the present paper we significantly refine the minimax results of existing work. To alleviate the pessimism of minimaxity, we adopt the adaptive estimation framework, and show that the minimax rate-optimal estimator in Jiao, Venkat, Han, and Weissman achieves the minimax rates simultaneously over a nested sequence of subsets of distributions $P$, without knowing the alphabet size $S$ or which subset $P$ lies in. In other words, their estimator is adaptive with respect to this nested sequence of the parameter space, which is characterized by the entropy of the distribution. We also characterize the maximum risk of the MLE over this nested sequence, and show, for every subset in the sequence, that the performance of the minimax rate-optimal estimator with $n$ samples is essentially that of the MLE with $n\ln n$ samples, thereby further substantiating the generality of the phenomenon identified by Jiao, Venkat, Han, and Weissman.

cs.IT↗

arXiv2018

Entropy Rate Estimation for Markov Chains with Large State Space

Estimating the entropy based on data is one of the prototypical problems in distribution property testing and estimation. For estimating the Shannon entropy of a distribution on $S$ elements with independent samples, [Paninski2004] showed that the sample complexity is sublinear in $S$, and [Valiant--Valiant2011] showed that consistent estimation of Shannon entropy is possible if and only if the sample size $n$ far exceeds $\frac{S}{\log S}$. In this paper we consider the problem of estimating the entropy rate of a stationary reversible Markov chain with $S$ states from a sample path of $n$ observations. We show that: (1) As long as the Markov chain mixes not too slowly, i.e., the relaxation time is at most $O(\frac{S}{\ln^3 S})$, consistent estimation is achievable when $n \gg \frac{S^2}{\log S}$. (2) As long as the Markov chain has some slight dependency, i.e., the relaxation time is at least $1+Ω(\frac{\ln^2 S}{\sqrt{S}})$, consistent estimation is impossible when $n \lesssim \frac{S^2}{\log S}$. Under both assumptions, the optimal estimation accuracy is shown to be $Θ(\frac{S^2}{n \log S})$. In comparison, the empirical entropy rate requires at least $Ω(S^2)$ samples to be consistent, even when the Markov chain is memoryless. In addition to synthetic experiments, we also apply the estimators that achieve the optimal sample complexity to estimate the entropy rate of the English language in the Penn Treebank and the Google One Billion Words corpora, which provides a natural benchmark for language modeling and relates it directly to the widely used perplexity measure.

cs.LG↗

arXiv2018

The Nearest Neighbor Information Estimator is Adaptively Near Minimax Rate-Optimal

We analyze the Kozachenko--Leonenko (KL) nearest neighbor estimator for the differential entropy. We obtain the first uniform upper bound on its performance over Hölder balls on a torus without assuming any conditions on how close the density could be from zero. Accompanying a new minimax lower bound over the Hölder ball, we show that the KL estimator is achieving the minimax rates up to logarithmic factors without cognizance of the smoothness parameter $s$ of the Hölder ball for $s\in (0,2]$ and arbitrary dimension $d$, rendering it the first estimator that provably satisfies this property.

stat.ML↗

arXiv2018

Local moment matching: A unified methodology for symmetric functional estimation and distribution estimation under Wasserstein distance

We present \emph{Local Moment Matching (LMM)}, a unified methodology for symmetric functional estimation and distribution estimation under Wasserstein distance. We construct an efficiently computable estimator that achieves the minimax rates in estimating the distribution up to permutation, and show that the plug-in approach of our unlabeled distribution estimator is "universal" in estimating symmetric functionals of discrete distributions. Instead of doing best polynomial approximation explicitly as in existing literature of functional estimation, the plug-in approach conducts polynomial approximation implicitly and attains the optimal sample complexity for the entropy, power sum and support size functionals.

stat.ME↗

arXiv2018

Minimax Estimation of the $L_1$ Distance

We consider the problem of estimating the $L_1$ distance between two discrete probability measures $P$ and $Q$ from empirical data in a nonasymptotic and large alphabet setting. When $Q$ is known and one obtains $n$ samples from $P$, we show that for every $Q$, the minimax rate-optimal estimator with $n$ samples achieves performance comparable to that of the maximum likelihood estimator (MLE) with $n\ln n$ samples. When both $P$ and $Q$ are unknown, we construct minimax rate-optimal estimators whose worst case performance is essentially that of the known $Q$ case with $Q$ being uniform, implying that $Q$ being uniform is essentially the most difficult case. The \emph{effective sample size enlargement} phenomenon, identified in Jiao \emph{et al.} (2015), holds both in the known $Q$ case for every $Q$ and the $Q$ unknown case. However, the construction of optimal estimators for $\|P-Q\|_1$ requires new techniques and insights beyond the approximation-based method of functional estimation in Jiao \emph{et al.} (2015).

math.ST↗

arXiv2017

Electrically Pumped Orbital Angular Momentum (OAM) Laser at Telecom Wavelengths

Semiconductor lasers capable of generating a vortex beam with a specific orbital angular momentum (OAM) order are highly attractive for applications ranging from nanoparticle manipulation, imaging and microscopy to fibre and quantum communications. In this work, an electrically pumped OAM laser operating at telecom wavelengths is fabricated by monolithically integrating an optical vortex emitter with a distributed feedback (DFB) laser on the same InGaAsP/InP epitaxial wafer. A single-step dry etching process is adopted to complete the OAM emitter, equipped with specially designed top gratings. The vortex beam emitted by the integrated laser is captured, and its OAM mode purity characterized. The electrically pumped OAM laser eliminates the external laser required by silicon- or silicon-on-insulator (SOI)-based OAM emitters, thus demonstrating great potential for applications in communication systems and quantum domain.

physics.optics↗

arXiv2017

Estimating the Fundamental Limits is Easier than Achieving the Fundamental Limits

We show through case studies that it is easier to estimate the fundamental limits of data processing than to construct explicit algorithms to achieve those limits. Focusing on binary classification, data compression, and prediction under logarithmic loss, we show that in the finite space setting, when it is possible to construct an estimator of the limits with vanishing error with $n$ samples, it may require at least $n\ln n$ samples to construct an explicit algorithm to achieve the limits.

cs.IT↗

arXiv2017

Generalizations of Maximal Inequalities to Arbitrary Selection Rules

We present a generalization of the maximal inequalities that upper bound the expectation of the maximum of $n$ jointly distributed random variables. We control the expectation of a randomly selected random variable from $n$ jointly distributed random variables, and present bounds that are at least as tight as the classical maximal inequalities, and much tighter when the distribution of selection index is near deterministic. A new family of information theoretic measures were introduced in the process, which may be of independent interest.

math.PR↗

arXiv2017

Maximum Likelihood Estimation of Functionals of Discrete Distributions

We consider the problem of estimating functionals of discrete distributions, and focus on tight nonasymptotic analysis of the worst case squared error risk of widely used estimators. We apply concentration inequalities to analyze the random fluctuation of these estimators around their expectations, and the theory of approximation using positive linear operators to analyze the deviation of their expectations from the true functional, namely their \emph{bias}. We characterize the worst case squared error risk incurred by the Maximum Likelihood Estimator (MLE) in estimating the Shannon entropy $H(P) = \sum_{i = 1}^S -p_i \ln p_i$, and $F_α(P) = \sum_{i = 1}^S p_i^α,α>0$, up to multiplicative constants, for any alphabet size $S\leq \infty$ and sample size $n$ for which the risk may vanish. As a corollary, for Shannon entropy estimation, we show that it is necessary and sufficient to have $n \gg S$ observations for the MLE to be consistent. In addition, we establish that it is necessary and sufficient to consider $n \gg S^{1/α}$ samples for the MLE to consistently estimate $F_α(P), 0<α<1$. The minimax rate-optimal estimators for both problems require $S/\ln S$ and $S^{1/α}/\ln S$ samples, which implies that the MLE has a strictly sub-optimal sample complexity. When $1<α<3/2$, we show that the worst-case squared error rate of convergence for the MLE is $n^{-2(α-1)}$ for infinite alphabet size, while the minimax squared error rate is $(n\ln n)^{-2(α-1)}$. When $α\geq 3/2$, the MLE achieves the minimax optimal rate $n^{-1}$ regardless of the alphabet size. As an application of the general theory, we analyze the Dirichlet prior smoothing techniques for Shannon entropy estimation. We show that no matter how we tune the parameters in the Dirichlet prior, this technique cannot achieve the minimax rates in entropy estimation.

cs.IT↗

arXiv2017

Dependence Measures Bounding the Exploration Bias for General Measurements

We propose a framework to analyze and quantify the bias in adaptive data analysis. It generalizes that proposed by Russo and Zou'15, applying to measurements whose moment generating function exists, measurements with a finite $p$-norm, and measurements in general Orlicz spaces. We introduce a new class of dependence measures which retain key properties of mutual information while more effectively quantifying the exploration bias for heavy tailed distributions. We provide examples of cases where our bounds are nearly tight in situations where the original framework of Russo and Zou'15 does not apply.

cs.IT↗

arXiv2017

Demystifying ResNet

The Residual Network (ResNet), proposed in He et al. (2015), utilized shortcut connections to significantly reduce the difficulty of training, which resulted in great performance boosts in terms of both training and generalization error. It was empirically observed in He et al. (2015) that stacking more layers of residual blocks with shortcut 2 results in smaller training error, while it is not true for shortcut of length 1 or 3. We provide a theoretical explanation for the uniqueness of shortcut 2. We show that with or without nonlinearities, by adding shortcuts that have depth two, the condition number of the Hessian of the loss function at the zero initial point is depth-invariant, which makes training very deep models no more difficult than shallow ones. Shortcuts of higher depth result in an extremely flat (high-order) stationary point initially, from which the optimization algorithm is hard to escape. The shortcut 1, however, is essentially equivalent to no shortcuts, which has a condition number exploding to infinity as the number of layers grows. We further argue that as the number of layers tends to infinity, it suffices to only look at the loss function at the zero initial point. Extensive experiments are provided accompanying our theoretical results. We show that initializing the network to small weights with shortcut 2 achieves significantly better results than random Gaussian (Xavier) initialization, orthogonal initialization, and shortcuts of deeper depth, from various perspectives ranging from final loss, learning dynamics and stability, to the behavior of the Hessian along the learning process.

cs.NE↗

arXiv2016

Broadband frequency comb generation in aluminum nitride-on-sapphire microresonators

Development of chip-scale optical frequency comb with the coverage from ultra-violet (UV) to mid-infrared (MIR) wavelength is of great significance. To expand the comb spectrum into the challenging UV region, a material platform with high UV transparency is crucial. In this paper, crystalline aluminum nitride (AlN)-onsapphire film is demonstrated for efficient Kerr frequency comb generation. Near-infrared (NIR) comb with nearly octave-spanning coverage and low parametric threshold is achieved in continuous-wave pumped high-quality-factor AlN microring resonators. The competition between stimulated Raman scattering (SRS) and hyperparametric oscillation is investigated, along with broadband comb generation via Raman-assisted four-wave mixing (FWM). Thanks to its wide bandgap, excellent crystalline quality as well as intrinsic quadratic and cubic susceptibilities, AlN-on-sapphire platform should be appealing for integrated nonlinear optics from MIR to UV region.

physics.optics↗