Searcharxiv⌕ Search

arXiv subjects

Iosif Pinelis

Publications and source records attributed to Iosif Pinelis.

At least 55 records · Page 3Linked to original sources
arXiv2017

Exact Lower Bounds for the Agnostic Probably-Approximately-Correct (PAC) Machine Learning Model

We provide an exact non-asymptotic lower bound on the minimax expected excess risk (EER) in the agnostic probably-ap\-proximately-correct (PAC) machine learning classification model and identify minimax learning algorithms as certain maximally symmetric and minimally randomized "voting" procedures. Based on this result, an exact asymptotic lower bound on the minimax EER is provided. This bound is of the simple form $c_\infty/\sqrtν$ as $ν\to\infty$, where $c_\infty=0.16997\dots$ is a universal constant, $ν=m/d$, $m$ is the size of the training sample, and $d$ is the Vapnik--Chervonenkis dimension of the hypothesis class. It is shown that the differences between these asymptotic and non-asymptotic bounds, as well as the differences between these two bounds and the maximum EER of any learning algorithms that minimize the empirical risk, are asymptotically negligible, and all these differences are due to ties in the mentioned "voting" procedures. A few easy to compute non-asymptotic lower bounds on the minimax EER are also obtained, which are shown to be close to the exact asymptotic lower bound $c_\infty/\sqrtν$ even for rather small values of the ratio $ν=m/d$. As an application of these results, we substantially improve existing lower bounds on the tail probability of the excess risk. Among the tools used are Bayes estimation and apparently new identities and inequalities for binomial distributions.

cs.LG↗
arXiv2017

Approximating sums by integrals only: multiple sums and sums over lattice polytopes

The Euler--Maclaurin (EM) summation formula is used in many theoretical studies and numerical calculations. It approximates the sum $\sum_{k=0}^{n-1} f(k)$ of values of a function $f$ by a linear combination of a corresponding integral of $f$ and values of its higher-order derivatives $f^{(j)}$. An alternative (Alt) summation formula was recently presented by the author, which approximates the sum by a linear combination of integrals only, without using high-order derivatives of $f$. It was shown that the Alt formula will in most cases outperform, or greatly outperform, the EM formula in terms of the execution time and memory use. In the present paper, a multiple-sum/multi-index-sum extension of the Alt formula is given, with applications to summing possibly divergent multi-index series and to sums over the integral points of integral lattice polytopes.

math.CA↗
arXiv2017

An alternative to the Euler--Maclaurin formula: Approximating sums by integrals only

The Euler--Maclaurin (EM) summation formula is used in many theoretical studies and numerical calculations. It approximates the sum $\sum_{k=0}^{n-1} f(k)$ of values of a function $f$ by a linear combination of a corresponding integral of $f$ and values of its higher-order derivatives $f^{(j)}$. An alternative (Alt) summation formula is proposed, which approximates the sum by a linear combination of integrals only, without using high-order derivatives of $f$. Explicit and rather easy to use bounds on the remainder are given. Possible extensions to multi-index summation are suggested. Applications to summing possibly divergent series are presented. It is shown that the Alt formula will in most cases outperform, or greatly outperform, the EM formula in terms of the execution time and memory use. One of the advantages of the Alt calculations is that, in contrast with the EM ones, they can be almost completely parallelized. Illustrative examples are given. In one of the examples, where an array of values of the Hurwitz generalized zeta function is computed with high accuracy, it is shown that both our implementation of the EM formula and, especially, the Alt formula perform much faster than the built-in Mathematica command HurwitzZeta[].

math.CA↗
arXiv2017

Measure extension by local approximation

Measurable sets are defined as those locally approximable, in a certain sense, by sets in the given algebra (or ring). A corresponding measure extension theorem is proved. It is also shown that a set is locally approximable in the mentioned sense if and only if it is Carathéodory-measurable.

math.CA↗
arXiv2016

Optimal-order bounds on the rate of convergence to normality for maximum likelihood estimators

It is well known that under general regularity conditions the distribution of the maximum likelihood estimator (MLE) is asymptotically normal. Very recently, bounds of the optimal order $O(1/\sqrt n)$ on the closeness of the distribution of the MLE to normality in the so-called bounded Wasserstein distance were obtained, where $n$ is the sample size. However, the corresponding bounds on the Kolmogorov distance were only of the order $O(1/n^{1/4})$. In this note, bounds of the optimal order $O(1/\sqrt n)$ on the closeness of the distribution of the MLE to normality in the Kolmogorov distance are given, as well as their nonuniform counterparts, which work better for large deviations of the MLE. These results are based on previously obtained general optimal-order bounds on the rate of convergence to normality in the multivariate delta method. The crucial observation is that, under natural conditions, the MLE can be tightly enough bracketed between two smooth enough functions of the sum of independent random vectors, which makes the delta method applicable.

math.ST↗
arXiv2016

Quantifying minimal non-collinearity among random points

Let $φ_{n,K}$ denote the largest angle in all the triangles with vertices among the $n$ points selected at random in a compact convex subset $K$ of $\mathbb{R}^d$ with nonempty interior, where $d\ge2$. It is shown that the distribution of the random variable $λ_d(K)\,\frac{n^3}{3!}\,(π-φ_{n,K})^{d-1}$, where $λ_d(K)$ is a certain positive real number which depends only on the dimension $d$ and the shape of $K$, converges to the standard exponential distribution as $n\to\infty$. By using the Steiner symmetrization, it is also shown that $λ_d(K)$ -- which is referred to in the paper as the elongation of $K$ -- attains its minimum if and only if $K$ is a ball $B^{(d)}$ in $\mathbb{R}^d$. Finally, the asymptotics of $λ_d(B^{(d)})$ for large $d$ is determined.

math.PR↗
arXiv2016

Positive-part moments via the characteristic functions, and more general expressions

A unifying and generalizing approach to representations of the positive-part and absolute moments $\mathsf{E} X_+^p$ and $\mathsf{E}|X|^p$ of a random variable $X$ for real $p$ in terms of the characteristic function (c.f.) of $X$, as well as to related representations of the c.f.\ of $X_+$, generalized moments $\mathsf{E} X_+^p e^{iuX}$, truncated moments, and the distribution function is provided. Existing and new representations of these kinds are all shown to stem from a single basic representation. Computational aspects of these representations are addressed.

math.PR↗
arXiv2016

On a multidimensional spherically invariant extension of the Rademacher--Gaussian comparison

It is shown that \begin{equation*} \mathsf{P}(\|a_1U_1+\dots+a_nU_n\|>u)\le c\,\mathsf{P}(a\|Z_d\|>u) \end{equation*} for all real $u$, where $U_1,\dots,U_n$ are independent random vectors uniformly distributed on the unit sphere in $\mathbb{R}^d$, $a_1,\dots,a_n$ are any real numbers, $a:=\sqrt{(a_1^2+\dots+a_n^2)/d}$, $Z_d$ is a standard normal random vector in $\mathbb{R}^d$, and $c=2e^3/9=4.46\dots$. This constant factor is about $89$ times as small as the one in a recent result by Nayar and Tkocz, who proved, by a different method, a corresponding conjecture by Oleszkiewicz. As an immediate application, a corresponding upper bound on the tail probabilities for the norm of the sum of arbitrary independent spherically invariant random vectors is given.

math.PR↗
arXiv2016

Optimal binomial, Poisson, and normal left-tail domination for sums of nonnegative random variables

Let $X_1,\dots,X_n$ be independent nonnegative random variables (r.v.'s), with $S_n:=X_1+\dots+X_n$ and finite values of $s_i:=E X_i^2$ and $m_i:=E X_i>0$. Exact upper bounds on $E f(S_n)$ for all functions $f$ in a certain class $\mathcal{F}$ of nonincreasing functions are obtained, in each of the following settings: (i) $n,m_1,\dots,m_n,s_1,\dots,s_n$ are fixed; (ii) $n$, $m:=m_1+\dots+m_n$, and $s:=s_1+\dots+s_n$ are fixed; (iii)~only $m$ and $s$ are fixed. These upper bounds are of the form $E f(η)$ for a certain r.v. $η$. The r.v. $η$ and the class $\mathcal{F}$ depend on the choice of one of the three settings. In particular, $(m/s)η$ has the binomial distribution with parameters $n$ and $p:=m^2/(ns)$ in setting (ii) and the Poisson distribution with parameter $λ:=m^2/s$ in setting (iii). One can also let $η$ have the normal distribution with mean $m$ and variance $s$ in any of these three settings. In each of the settings, the class $\mathcal{F}$ contains, and is much wider than, the class of all decreasing exponential functions. As corollaries of these results, optimal in a certain sense upper bounds on the left-tail probabilities $P(S_n\le x)$ are presented, for any real $x$. In fact, more general settings than the ones described above are considered. Exact upper bounds on the exponential moments $E\exp\{hS_n\}$ for $h<0$, as well as the corresponding exponential bounds on the left-tail probabilities, were previously obtained by Pinelis and Utev. It is shown that the new bounds on the tails are substantially better.

math.PR↗
arXiv2016

Optimal-order bounds on the rate of convergence to normality in the multivariate delta method

Uniform and nonuniform Berry--Esseen (BE) bounds of optimal orders on the closeness to normality for general abstract nonlinear statistics are given, which are then used to obtain optimal bounds on the rate of convergence in the delta method for vector statistics. Specific applications to Pearson's, non-central Student's and Hotelling's statistics, sphericity test statistics, a regularized canonical correlation, and maximum likelihood estimators (MLEs) are given; all these uniform and nonuniform BE bounds appear to be the first known results of these kinds, except for uniform BE bounds for MLEs. When applied to the well-studied case of the central Student statistic, our general results compare well with known ones in that case, obtained previously by specialized methods. The proofs use a Stein-type method developed by Chen and Shao, a Cramér-type of tilt transform, exponential and Rosenthal-type inequalities for sums of random vectors established by Pinelis, Sakhanenko, and Utev, as well as a number of other, quite recent results motivated by this study. The method allows one to obtain bounds with explicit and rather moderate-size constants, at least as far as the uniform bounds are concerned. For instance, one has the uniform BE bound $3.61\mathbb{E}(Y_1^6+Z_1^6)\,(1+σ^{-3})/\sqrt n$ for the Pearson sample correlation coefficient based on independent identically distributed random pairs $(Y_1,Z_1),\dots,(Y_n,Z_n)$ with $\mathbb{E} Y_1=\mathbb{E} Z_1=\mathbb{E} Y_1Z_1=0$ and $\mathbb{E} Y_1^2=\mathbb{E} Z_1^2=1$, where $σ:=\sqrt{\mathbb{E} Y_1^2Z_1^2}$.

math.ST↗
arXiv2015

Convex cones of generalized multiply monotone functions and the dual cones

Let $n$ and $k$ be nonnegative integers such that $1\le k\le n+1$. The convex cone $\mathcal{F}_+^{k:n}$ of all functions $f$ on an arbitrary interval $I\subseteq\mathbb{R}$ whose derivatives $f^{(j)}$ of orders $j=k-1,\dots,n$ are nondecreasing is characterized in terms of extreme rays of the cone $\mathcal{F}_+^{k:n}$. A simple description of the convex cone dual to $\mathcal{F}_+^{k:n}$ is given. These results are useful in, and were motivated by, applications in probability. In fact, the results are obtained in a more general setting with certain generalized derivatives of $f$ of the $j$th order in place of $f^{(j)}$. Somewhat similar results were previously obtained in the case when the left endpoint of the interval $I$ is finite, with certain additional integrability conditions; such conditions fail to hold in the mentioned applications.

math.CA↗
arXiv2015

Exact bounds on the inverse Mills ratio and its derivatives

The inverse Mills ratio is $R:=φ/Ψ$, where $φ$ and $Ψ$ are, respectively, the probability density function and the tail function of the standard normal distribution. Exact bounds on $R(z)$ for complex $z$ with $\Re z\ge0$ are obtained, which then yield logarithmically exact bounds on high-order derivatives of $R$. The main idea of the proof is a non-asymptotic version of the so-called stationary-phase method.

math.CV↗
arXiv2015

Exact Rosenthal-type bounds

It is shown that, for any given $p\ge5$, $A>0$ and $B>0$, the exact upper bound on $\mathsf{E}|\sum X_i|^p$ over all independent zero-mean random variables (r.v.'s) $X_1,\ldots,X_n$ such that $\sum\mathsf{E}X_i^2=B$ and $\sum\mathsf{E}|X_i|^p=A$ equals $c^p\mathsf{E}|Π_λ-λ|^p$, where $(λ,c)\in(0,\infty)^2$ is the unique solution to the system of equations $c^pλ=A$ and $c^2λ=B$, and $Π_λ$ is a Poisson r.v. with mean $λ$. In fact, a more general result is obtained, as well as other related ones. As a tool used in the proof, a calculus of variations of moments of infinitely divisible distributions with respect to variations of the Lévy characteristics is developed.

math.PR↗
arXiv2015

A characterization of best unbiased estimators

A simple characterization of uniformly minimum variance unbiased estimators (UMVUEs) is provided (in the case when the sample space is finite) in terms of a linear independence condition on the likelihood functions corresponding to the possible samples. The crucial observation in the proof is that, if a UMVUE exists, then, after an appropriate cleaning of the parameter space, the nonzero likelihood functions are eigenvectors of an "artificial" matrix of Lagrange multipliers, and the values of the UMVUE are eigenvalues of that matrix. The characterization is then extended to best unbiased estimators with respect to arbitrary convex loss functions.

math.ST↗
arXiv2015

Explicit additive decomposition of norms on $\mathbb{R}^2$

A well-known result by Lindenstrauss is that any two-dimensional normed space can be isometrically imbedded into $L_1(0,1)$. We provide an explicit form of a such an imbedding. The proof is elementary and self-contained. Applications are given concerning the following: (i) explicit representations of the moments of the norm of a random vector $X$ in terms of the characteristic function and the Fourier--Laplace transform of the distribution of $X$; (ii) an explicit and partially improved form of the exact version of the Littlewood--Khinchin--Kahane inequality obtained by Latała and Oleszkiewicz; (iii) an extension of an inequality by Buja--Logan--Reeds--Shepp, arising from a statistical problem.

math.FA↗
arXiv2015

Exact upper and lower bounds on the difference between the arithmetic and geometric means

Let $X$ denote a nonnegative random variable with $\mathsf{E} X<\infty$. Upper and lower bounds on $\mathsf{E} X-\exp\mathsf{E}\ln X$ are obtained, which are exact, in terms of $V_X$ and $E_X$ for the upper bound and in terms of $V_X$ and $F_X$ for the lower bound, where $V_X:=\mathsf{Var}\sqrt X$, $E_X:=\mathsf{E}\big(\sqrt X-\sqrt{m_X}\,\big)^2$, $F_X:=\mathsf{E}\big(\sqrt{M_X}-\sqrt X\,\big)^2$, $m_X:=\inf S_X$, $M_X:=\sup S_X$, and $S_X$ is the support set of the distribution of $X$. Note that, if $X$ takes each of distinct real values $x_1,\dots,x_n$ with probability $1/n$, then $\mathsf{E} X$ and $\exp\mathsf{E}\ln X$ are, respectively, the arithmetic and geometric means of $x_1,\dots,x_n$.

math.PR↗