arXiv subjects

Art B. Owen

Publications and source records attributed to Art B. Owen.

At least 55 records · Page 3Linked to original sources

arXiv2021

Probing neural networks with t-SNE, class-specific projections and a guided tour

We use graphical methods to probe neural nets that classify images. Plots of t-SNE outputs at successive layers in a network reveal increasingly organized arrangement of the data points. They can also reveal how a network can diminish or even forget about within-class structure as the data proceeds through layers. We use class-specific analogues of principal components to visualize how succeeding layers separate the classes. These allow us to sort images from a given class from most typical to least typical (in the data) and they also serve as very useful projection coordinates for data visualization. We find them especially useful when defining versions guided tours for animated data visualization.

cs.LG↗

arXiv2021

The nonzero gain coefficients of Sobol's sequences are always powers of two

When a plain Monte Carlo estimate on $n$ samples has variance $σ^2/n$, then scrambled digital nets attain a variance that is $o(1/n)$ as $n\to\infty$. For finite $n$ and an adversarially selected integrand, the variance of a scrambled $(t,m,s)$-net can be at most $Γσ^2/n$ for a maximal gain coefficient $Γ<\infty$. The most widely used digital nets and sequences are those of Sobol'. It was previously known that $Γ\leqslant 2^t3^s$ for Sobol' points as well as Niederreiter-Xing points. In this paper we study nets in base $2$. We show that $Γ\leqslant2^{t+s-1}$ for nets. This bound is a simple, but apparently unnoticed, consequence of a microstructure analysis in Niederreiter and Pirsic (2001). We obtain a sharper bound that is smaller than this for some digital nets. We also show that all nonzero gain coefficients must be powers of two. A consequence of this latter fact is a simplified algorithm for computing gain coefficients of nets in base $2$.

math.NA↗

arXiv2021

Cohort Shapley value for algorithmic fairness

Cohort Shapley value is a model-free method of variable importance grounded in game theory that does not use any unobserved and potentially impossible feature combinations. We use it to evaluate algorithmic fairness, using the well known COMPAS recidivism data as our example. This approach allows one to identify for each individual in a data set the extent to which they were adversely or beneficially affected by their value of a protected attribute such as their race. The method can do this even if race was not one of the original predictors and even if it does not have access to a proprietary algorithm that has made the predictions. The grounding in game theory lets us define aggregate variable importance for a data set consistently with its per subject definitions. We can investigate variable importance for multiple quantities of interest in the fairness literature including false positive predictions.

cs.LG↗

arXiv2021

Quasi-Newton Quasi-Monte Carlo for variational Bayes

Many machine learning problems optimize an objective that must be measured with noise. The primary method is a first order stochastic gradient descent using one or more Monte Carlo (MC) samples at each step. There are settings where ill-conditioning makes second order methods such as L-BFGS more effective. We study the use of randomized quasi-Monte Carlo (RQMC) sampling for such problems. When MC sampling has a root mean squared error (RMSE) of $O(n^{-1/2})$ then RQMC has an RMSE of $o(n^{-1/2})$ that can be close to $O(n^{-3/2})$ in favorable settings. We prove that improved sampling accuracy translates directly to improved optimization. In our empirical investigations for variational Bayes, using RQMC with stochastic L-BFGS greatly speeds up the optimization, and sometimes finds a better parameter value than MC does.

cs.LG↗

arXiv2021

Backfitting for large scale crossed random effects regressions

Regression models with crossed random effect errors can be very expensive to compute. The cost of both generalized least squares and Gibbs sampling can easily grow as $N^{3/2}$ (or worse) for $N$ observations. Papaspiliopoulos et al. (2020) present a collapsed Gibbs sampler that costs $O(N)$, but under an extremely stringent sampling model. We propose a backfitting algorithm to compute a generalized least squares estimate and prove that it costs $O(N)$. A critical part of the proof is in ensuring that the number of iterations required is $O(1)$ which follows from keeping a certain matrix norm below $1-δ$ for some $δ>0$. Our conditions are greatly relaxed compared to those for the collapsed Gibbs sampler, though still strict. Empirically, the backfitting algorithm has a norm below $1-δ$ under conditions that are less strict than those in our assumptions. We illustrate the new algorithm on a ratings data set from Stitch Fix.

stat.ME↗

arXiv2021

Designing Experiments Informed by Observational Studies

The increasing availability of passively observed data has yielded a growing methodological interest in "data fusion." These methods involve merging data from observational and experimental sources to draw causal conclusions -- and they typically require a precarious tradeoff between the unknown bias in the observational dataset and the often-large variance in the experimental dataset. We propose an alternative approach to leveraging observational data, which avoids this tradeoff: rather than using observational data for inference, we use it to design a more efficient experiment. We consider the case of a stratified experiment with a binary outcome, and suppose pilot estimates for the stratum potential outcome variances can be obtained from the observational study. We extend results from Zhao et al. (2019) in order to generate confidence sets for these variances, while accounting for the possibility of unmeasured confounding. Then, we pose the experimental design problem as one of regret minimization, subject to the constraints imposed by our confidence sets. We show that this problem can be converted into a convex minimization and solved using conventional methods. Lastly, we demonstrate the practical utility of our methods using data from the Women's Health Initiative.

stat.ME↗

arXiv2020

Efficient estimation of the ANOVA mean dimension, with an application to neural net classification

The mean dimension of a black box function of $d$ variables is a convenient way to summarize the extent to which it is dominated by high or low order interactions. It is expressed in terms of $2^d-1$ variance components but it can be written as the sum of $d$ Sobol' indices that can be estimated by leave one out methods. We compare the variance of these leave one out methods: a Gibbs sampler called winding stairs, a radial sampler that changes each variable one at a time from a baseline, and a naive sampler that never reuses function evaluations and so costs about double the other methods. For an additive function the radial and winding stairs are most efficient. For a multiplicative function the naive method can easily be most efficient if the factors have high kurtosis. As an illustration we consider the mean dimension of a neural network classifier of digits from the MNIST data set. The classifier is a function of $784$ pixels. For that problem, winding stairs is the best algorithm. We find that inputs to the final softmax layer have mean dimensions ranging from $1.35$ to $2.0$.

stat.ME↗

arXiv2020

Density estimation by Randomized Quasi-Monte Carlo

We consider the problem of estimating the density of a random variable $X$ that can be sampled exactly by Monte Carlo (MC). We investigate the effectiveness of replacing MC by randomized quasi Monte Carlo (RQMC) or by stratified sampling over the unit cube, to reduce the integrated variance (IV) and the mean integrated square error (MISE) for kernel density estimators. We show theoretically and empirically that the RQMC and stratified estimators can achieve substantial reductions of the IV and the MISE, and even faster convergence rates than MC in some situations, while leaving the bias unchanged. We also show that the variance bounds obtained via a traditional Koksma-Hlawka-type inequality for RQMC are much too loose to be useful when the dimension of the problem exceeds a few units. We describe an alternative way to estimate the IV, a good bandwidth, and the MISE, under RQMC or stratification, and we show empirically that in some situations, the MISE can be reduced significantly even in high-dimensional settings.

math.ST↗

arXiv2020

Explaining black box decisions by Shapley cohort refinement

We introduce a variable importance measure to quantify the impact of individual input variables to a black box function. Our measure is based on the Shapley value from cooperative game theory. Many measures of variable importance operate by changing some predictor values with others held fixed, potentially creating unlikely or even logically impossible combinations. Our cohort Shapley measure uses only observed data points. Instead of changing the value of a predictor we include or exclude subjects similar to the target subject on that predictor to form a similarity cohort. Then we apply Shapley value to the cohort averages. We connect variable importance measures from explainable AI to function decompositions from global sensitivity analysis. We introduce a squared cohort Shapley value that splits previously studied Shapley effects over subjects, consistent with a Shapley axiom.

cs.LG↗

arXiv2020

Optimizing the tie-breaker regression discontinuity design

Motivated by customer loyalty plans and scholarship programs, we study tie-breaker designs which are hybrids of randomized controlled trials (RCTs) and regression discontinuity designs (RDDs). We quantify the statistical efficiency of a tie-breaker design in which a proportion $Δ$ of observed subjects are in the RCT. In a two line regression, statistical efficiency increases monotonically with $Δ$, so efficiency is maximized by an RCT. We point to additional advantages of tie-breakers versus RDD: for a nonparametric regression the boundary bias is much less severe and for quadratic regression, the variance is greatly reduced. For a two line model we can quantify the short term value of the treatment allocation and this comparison favors smaller $Δ$ with the RDD being best. We solve for the optimal tradeoff between these exploration and exploitation goals. The usual tie-breaker design applies an RCT on the middle $Δ$ subjects as ranked by the assignment variable. We quantify the efficiency of other designs such as experimenting only in the second decile from the top. We also show that in some general parametric models a Monte Carlo evaluation can be replaced by matrix algebra.

stat.ME↗

arXiv2020

A strong law of large numbers for scrambled net integration

This article provides a strong law of large numbers for integration on digital nets randomized by a nested uniform scramble. The motivating problem is optimization over some variables of an integral over others, arising in Bayesian optimization. This strong law requires that the integrand have a finite moment of order $p$ for some $p>1$. Previously known results implied a strong law only for Riemann integrable functions. Previous general weak laws of large numbers for scrambled nets require a square integrable integrand. We generalize from $L^2$ to $L^p$ for $p>1$ via the Riesz-Thorin interpolation theorem

math.NA↗

arXiv2019

Refiltering hypothesis tests to control sign error

A common, though not recommended statistical practice is to report confidence intervals if and only if they exclude a null value of 0. The resulting filtered confidence intervals generally do not have their nominal confidence level. More worryingly, in low power settings their center points will be much farther from zero than the true parameter is and they will frequently lie on the wrong side of zero. Many confidence intervals are constructed using an asymptotically Gaussian parameter estimate accompanied by a weakly consistent estimate of its variance. In these cases, we can subject the given confidence interval(s) to a second filtering step such that the probability of a sign error is controled. This refiltering step retains only those confidence intervals that are sufficiently well separated from the origin. It requires no assumptions on the dependencies among the test statistics.

math.ST↗

arXiv2019

Deterministic parallel analysis: An improved method for selecting factors and principal components

Factor analysis and principal component analysis (PCA) are used in many application areas. The first step, choosing the number of components, remains a serious challenge. Our work proposes improved methods for this important problem. One of the most popular state-of-the-art methods is Parallel Analysis (PA), which compares the observed factor strengths to simulated ones under a noise-only model. This paper proposes improvements to PA. We first de-randomize it, proposing Deterministic Parallel Analysis (DPA), which is faster and more reproducible than PA. Both PA and DPA are prone to a shadowing phenomenon in which a strong factor makes it hard to detect smaller but more interesting factors. We propose deflation to counter shadowing. We also propose to raise the decision threshold to improve estimation accuracy. We prove several consistency results for our methods, and test them in simulations. We also illustrate our methods on data from the Human Genome Diversity Project, where they significantly improve the accuracy.

stat.ME↗

arXiv2019

Mean Dimension of Ridge Functions

We consider the mean dimension of some ridge functions of spherical Gaussian random vectors of dimension $d$. If the ridge function is Lipschitz continuous, then the mean dimension remains bounded as $d\to\infty$. If instead, the ridge function is discontinuous, then the mean dimension depends on a measure of the ridge function's sparsity, and absent sparsity the mean dimension can grow proportionally to $\sqrt{d}$. Preintegrating a ridge function yields a new, potentially much smoother ridge function. We include an example where, if one of the ridge coefficients is bounded away from zero as $d\to\infty$, then preintegration can reduce the mean dimension from $O(\sqrt{d})$ to $O(1)$.

math.NA↗

arXiv2019

The square root rule for adaptive importance sampling

In adaptive importance sampling, and other contexts, we have $K>1$ unbiased and uncorrelated estimates $\hatμ_k$ of a common quantity $μ$. The optimal unbiased linear combination weights them inversely to their variances but those weights are unknown and hard to estimate. A simple deterministic square root rule based on a working model that $\mathrm{Var}(\hatμ_k)\propto k^{-1/2}$ gives an unbisaed estimate of $μ$ that is nearly optimal under a wide range of alternative variance patterns. We show that if $\mathrm{Var}(\hatμ_k)\propto k^{-y}$ for an unknown rate parameter $y\in [0,1]$ then the square root rule yields the optimal variance rate with a constant that is too large by at most $9/8$ for any $0\le y\le 1$ and any number $K$ of estimates. Numerical work shows that rule is similarly robust to some other patterns with mildly decreasing variance as $k$ increases.

math.ST↗

arXiv2019

Unreasonable effectiveness of Monte Carlo

This is a comment on the article "Probabilistic Integration: A Role in Statistical Computation?" by F.-X. Briol, C. J. Oates, M. Girolami, M. A. Osborne and D. Sejdinovic to appear in Statistical Science. There is a role for statistical computation in numerical integration. However, the competition from incumbent methods looks to be stiffer for this problem than for some of the newer problems being handled by probabilistic numerics. One of the challenges is the unreasonable effectiveness of the central limit theorem. Another is the unreasonable effectiveness of pseudorandom number generators. A third is the common $O(n^3)$ cost of methods based on Gaussian processes. Despite these advantages, the classical methods are weak in places where probabilistic methods could bring an improvement.

stat.CO↗

arXiv2018

Importance sampling the union of rare events with an application to power systems analysis

We consider importance sampling to estimate the probability $μ$ of a union of $J$ rare events $H_j$ defined by a random variable $\boldsymbol{x}$. The sampler we study has been used in spatial statistics, genomics and combinatorics going back at least to Karp and Luby (1983). It works by sampling one event at random, then sampling $\boldsymbol{x}$ conditionally on that event happening and it constructs an unbiased estimate of $μ$ by multiplying an inverse moment of the number of occuring events by the union bound. We prove some variance bounds for this sampler. For a sample size of $n$, it has a variance no larger than $μ(\barμ-μ)/n$ where $\barμ$ is the union bound. It also has a coefficient of variation no larger than $\sqrt{(J+J^{-1}-2)/(4n)}$ regardless of the overlap pattern among the $J$ events. Our motivating problem comes from power system reliability, where the phase differences between connected nodes have a joint Gaussian distribution and the $J$ rare events arise from unacceptably large phase differences. In the grid reliability problems even some events defined by $5772$ constraints in $326$ dimensions, with probability below $10^{-22}$, are estimated with a coefficient of variation of about $0.0024$ with only $n=10{,}000$ sample values.

stat.CO↗

arXiv2018

Effective dimension of some weighted pre-Sobolev spaces with dominating mixed partial derivatives

This paper considers two notions of effective dimension for quadrature in weighted pre-Sobolev spaces with dominating mixed partial derivatives. We begin by finding a ball in those spaces just barely large enough to contain a function with unit variance. If no function in that ball has more than $\varepsilon$ of its variance from ANOVA components involving interactions of order $s$ or more, then the space has effective dimension at most $s$ in the superposition sense. A similar truncation sense notion replaces the cardinality of the ANOVA component by the largest index it contains. Some Poincaré type inequalities are used to bound variance components by multiples of these space's squared norm and those in turn provide bounds on effective dimension. Very low effective dimension in the superposition sense holds for some spaces defined by product weights in which quadrature is strongly tractable. The superposition dimension is $O( \log(1/\varepsilon)/\log(\log(1/\varepsilon)))$ just like the superposition dimension used in the multidimensional decomposition method. Surprisingly, even spaces where all subset weights are equal, regardless of their cardinality or included indices, have low superposition dimension in this sense. This paper does not require periodicity of the integrands.

math.NA↗