arXiv subjects

Art B. Owen

Publications and source records attributed to Art B. Owen.

At least 19 recordsLinked to original sources

arXiv2026

On the optimization of discrepancy measures

Numerical optimizations can find point sets with lower discrepancies than prior methods based on algebra, and $L_2$ discrepancies are computationally attractive for this. The $L_2$ star discrepancy unreasonably emphasizes the origin and Matoušek pointed to a serious drawback. He showed that the $L_2$ star discrepancy of $n$ points at $(1,\dots,1)\in[0,1]^d$ is less than the root mean squared $L_2$ star discrepancy of $n$ random points until $n>(3/2)^d-1$. We introduce the \emph{average squared discrepancy} (ASD) which averages $2^d$ versions of the squared $L_2$ star discrepancy, one for each vertex of $[0,1]^d$. We show that it can be computed in $O(dn^2)$ work and that it does not suffer from Matoušek's repeated-point pathology. We inspect several $L_2$ discrepancies that do not favor the origin over other vertices. Only two of them avoid the repeated-point pathology: a symmetric discrepancy and a periodic discrepancy. The symmetric discrepancy has a twin-point pathology where point sets with only $2$ distinct points can be better than random points until $n\ge 3^d$. We also study weighted $L_2$ discrepancies introduced by Hickernell that use some product weights. Weighting lower dimensional projections reduces but does not eliminate the repeated-point pathologies. A range of numerical experiments showcase recent progress in the construction of sets with low $L_2$ discrepancies for many different measures. In our examples, optimizing the ASD also gives good results for \emph{the other} discrepancies we consider, especially in lower dimensions. This version corrects some of the threshold sample sizes n at which random points become better than n repeated points for some discrepancy measures. In particular, the periodic L_2 discrepancy has no repeated point pathology.

math.NA↗

arXiv2026

Randomized quasi-Monte Carlo integration

Quasi-Monte Carlo sampling is a numerical integration method that uses points with a space-filling property in $[0,1]^s$ designed to give better estimates than plain Monte Carlo methods do. For integrands of bounded variation in the sense of Hardy and Krause, errors of $O(n^{-1+ε})$ for any $ε>0$ are obtained from $n$ sample points. Randomized quasi-Monte Carlo (RQMC) points are individually uniformly distributed but collectively space-filling and then independent replications provide variance estimates. For smooth enough integrands the randomization can give a root mean squared error of $O(n^{-3/2+ε})$. This article explains RQMC for a statistical readership recounting some history and presenting some current directions.

stat.CO↗

arXiv2026

Walk on spheres and Array-RQMC

We use Array-RQMC sampling in a walk on spheres (WoS) algorithm for Dirichlet boundary value problems. On a collection of problems, we find that Array-RQMC-WoS reduces the Monte Carlo MSE or variance by factors ranging from $71$-fold to $3087$-fold at $n=2^{17}$ trajectories. The variance is known to be $o(1/n)$ but attains empirical rates between $n^{-1.4}$ and $n^{-1.8}$ in our examples. A simpler RQMC-WoS algorithm studied in Ho and Owen (2026) has more theoretical support but only reduced variance by 1.8 to 10.7-fold on the same set of examples. In order to explain this improvement, we introduce a column-wise mean dimension of the RQMC error based on Sobol' indices. It matches the usual mean dimension for Monte Carlo and the mean dimension of a dual lattice error for randomized lattices. We find for a gasket example from Crane et al. (2025) that the mean dimension of Array-RQMC-WoS errors is much higher than an analogous Array-MC-WoS algorithm has. v2 replaced v1's QMCPy lattice with Korobov lattices from LatNet Builder, but left the old abstract in the meta-data v3 corrects the v2 abstract in this meta data

math.NA↗

arXiv2026

Randomized quasi-Monte Carlo for walk on spheres

We investigate the use of randomized quasi-Monte Carlo (RQMC) in walk on spheres algorithms to solve boundary value problems for functions with Dirichlet boundary conditions in $\mathbb{R}^d$. For harmonic functions with $d=2$, the integrands of interest are periodic indicator functions over regions $Θ$ in the torus $\mathbb{T}^k$. We give conditions for $\partialΘ$ to have $k-1$ dimensional Minkowski content which allows us to use results of He and Wang (2015). The RQMC estimates involve multiple values of $k$. We see sampling variances decreasing with the number $n$ of sample points at slightly better than Monte Carlo rates. The median variance rate in $4$ RQMC methods over $5$ worked examples, including some with $d=3$ and some with nonzero source functions, was slightly better than $O(n^{-1.1})$. The variance reduction factors ranged from $1.8$ to $10.7$ at $n=2^{17}$. None of the four RQMC methods dominated the others. Changes: cite Liu (2025), describe the lattices that were used, describe usage of AI

math.NA↗

arXiv2026

Empirical Bernstein and betting confidence intervals for randomized quasi-Monte Carlo

Randomized quasi-Monte Carlo (RQMC) methods estimate the mean of a random variable by sampling an integrand at $n$ equidistributed points. For scrambled digital nets, the resulting variance is typically $\tilde O(n^{-θ})$ where $θ\in[1,3]$ depends on the smoothness of the integrand and $\tilde O$ neglects logarithmic factors. While RQMC can be far more accurate than plain Monte Carlo (MC) it remains difficult to get confidence intervals on RQMC estimates. We investigate some empirical Bernstein confidence intervals (EBCI) and hedged betting confidence intervals (HBCI), both from Waudby-Smith and Ramdas (2024), when the random variable of interest is subject to known bounds. When there are $N$ integrand evaluations partitioned into $R$ independent replicates of $n=N/R$ RQMC points, and the RQMC variance is $Θ(n^{-θ})$, then an oracle minimizing the width of a Bennett confidence interval would choose $n =Θ(N^{1/(θ+1)})$. The resulting intervals have a width that is $Θ(N^{-θ/(θ+1)})$. Our empirical investigations had optimal values of $n$ grow slowly with $N$, HBCI intervals that were usually narrower than the EBCI ones, and optimal values of $n$ for HBCI that were equal to or smaller than the ones for the oracle.

math.NA↗

arXiv2026

Quasi-Monte Carlo with one categorical variable

We study randomized quasi-Monte Carlo (RQMC) estimation of a multivariate integral where one of the variables takes only a finite number of values. This problem arises when the variable of integration is drawn from a mixture distribution as is common in importance sampling and also arises in some recent work on transport maps. We find that when integration error decreases at an RQMC rate that it is then important to oversample the smallest mixture components instead of using a proportional allocation. This can even improve the rate of convergence. The optimal allocations depend on the possibly unknown convergence rate. Designing the sample with an incorrect assumption on the rate still attains that convergence rate, with an inferior implied constant. The penalty for using a pessimistic rate is typically higher than for using an optimistic one. We also find that for the most accurate RQMC sampling methods, it is advantageous to arrange that our $n=2^m$ randomized Sobol' points split into subsample sizes that are also powers of $2$.

stat.CO↗

arXiv2025

Scalable solution to crossed random effects model with random slopes

The crossed random effects model is widely used, finding applications in various fields such as longitudinal studies, e-commerce, and recommender systems, among others. However, these models encounter scalability challenges, as the computational time for standard algorithms grows superlinearly with the number N of observations in the data set, commonly $Ω(N^{3/2})$ or worse. Recent published works present scalable methods for crossed random effects in linear models and some generalized linear models, but those methods only allow for random intercepts. In this paper, we devise scalable algorithms for models that include random slopes. This addition brings substantial difficulty in estimating the random-effect covariance matrices in a scalable way. We address this issue by using a variational EM algorithm. Our proposed approach accommodates both diagonal covariance matrices and cases where no structure is assumed-a scenario common in fields such as psychology and neuroscience. In simulations, the proposed method is substantially faster than standard methods for large $N$. It is also more efficient than ordinary least squares which has a problem of greatly underestimating the sampling uncertainty in parameter estimates. We illustrate the new method on a MovieLens dataset, as well as a large data set (five million observations) from the online retailer Stitch Fix.

stat.ME↗

arXiv2025

Zero variance self-normalized importance sampling via estimating equations

In ordinary importance sampling with a nonnegative integrand there exists an importance sampling strategy with zero variance. Practical sampling strategies are often based on approximating that optimal solution, potentially approaching zero variance. There is a positivisation extension of that method to handle integrands that take both positive and negative values. Self-normalized importance sampling uses a ratio estimate, for which the optimal sampler does not have zero variance and so zero variance cannot even be approached in practice. Strategies that separately estimate the numerator and denominator of that ratio can approach zero variance. This paper develops another zero variance solution for self-normalized importance sampling. The first step is to write the desired expectation as the zero of an estimating equation using Fieller's technique. Then we apply the positivisation strategy to the estimating equation. This paper give conditions for existence and uniqueness of the sample solution to the estimating equation. Then it give conditions for consistency and asymptotic normality and an expression for the asymptotic variance. The sample size multiplied by the variance of the asymptotic formula becomes arbitrarily close to zero for certain sampling strategies.

math.ST↗

arXiv2025

Better bootstrap t confidence intervals for the mean

This article explores combinations of weighted bootstraps, like the Bayesian bootstrap, with the bootstrap $t$ method for setting approximate confidence intervals for the mean of a random variable in small samples. For this problem the usual bootstrap $t$ has good coverage but provides intervals with long and highly variable lengths. Those intervals can have infinite length not just for tiny $n$, when the data have a discrete distribution. The BC$_a$ bootstrap produces shorter intervals but tends to severely under-cover the mean. Bootstrapping the studentized mean with weights from a Beta$(1/2,3/2)$ distribution is shown to attain second order accuracy. It never yields infinite length intervals and the mean square bootstrap $t$ statistic is finite when there are at least three distinct values in the data, or two distinct values appearing at least three times each. In a range of small sample settings, the beta bootstrap $t$ intervals have closer to nominal coverage than the BC$_a$ and shorter length than the multinomial bootstrap $t$. The paper includes a lengthy discussion of the difficulties in constructing a utility function to evaluate nonparametric approximate confidence intervals.

math.ST↗

arXiv2025

Error estimation for quasi-Monte Carlo

Quasi-Monte Carlo sampling can attain far better accuracy than plain Monte Carlo sampling. However, with plain Monte Carlo sampling it is much easier to estimate the attained accuracy. This article describes methods old and new to quantify the error in quasi-Monte Carlo estimates. An important challenge in this setting is that the goal of getting accuracy conflicts with that of estimating the attained accuracy. A related challenge is that rigorous uncertainty quantifications can be extremely conservative. A recent surprise is that some RQMC estimates have nearly symmetric distributions and that has the potential to allow confidence intervals that do not require either a central limit theorem or a consistent variance estimate.

math.NA↗

arXiv2025

Coverage errors for Student's t confidence intervals comparable to those in Hall (1988)

Table 1 of Hall (1988) contains asymptotic coverage error formulas for some nonparametric approximate 95\% confidence intervals for the mean based on $n$ IID samples. The table includes an entry for an interval based on the central limit theorem using Gaussian quantiles and the Gaussian maximum likelihood variance estimate. It is missing an entry for the very widely used Student's $t$ confidence intervals. This note develops such a formula. The impetus to revisit this issue arose from the surprisingly robust performance of confidence intervals based on Student's t statistic in randomized quasi-Monte Carlo sampling. Hall's table had $0.14κ-2.12γ^2-3.35$ for normal theory intervals; the corresponding entry for Student's $t$ is $0.14κ-2.12γ^2$. An earlier version of this note reported that it corrected some coverage error formulas in Hall (1988). Two-sided errors take the form $2Φ^{-1}(0.975)(Aκ+ γ^2+C)φ(1.96)/n +O(1/n^{3/2})$ where the error may well be $O(n^{-2})$. Hall's table showed $Φ^{-1}(0.975)(Aκ+ Bγ^2+C)$. The version intended as a correction had $2(Aκ+ Bγ^2+C)$, wider by about $2/1.96\doteq1.02$. So, Hall's table really is proportional to the two-sided coverage errors.

math.ST↗

arXiv2025

Constrained Design of a Binary Instrument in a Partially Linear Model

We study the question of how best to assign an encouragement in a randomized encouragement study. In our setting, units arrive with covariates, receive a nudge toward treatment or control, acquire one of those statuses in a way that need not align with the nudge, and finally have a response observed. The nudge can be modeled as a binary instrument if one assumes that it affects the response only via the treatment status. Our goal is to assign the nudge as a function of covariates in a way that best estimates the local average treatment effect (LATE). We assume a partially linear model, wherein the baseline model is non-parametric and the treatment term is linear in the covariates. Under this model, we outline a two-stage procedure to consistently and optimally estimate the LATE. Though the variance of the LATE is intractable, we derive a finite sample approximation and thus a design criterion to minimize. This criterion is convex, allowing for constraints that might arise for budgetary or ethical reasons. We prove conditions under which our solution asymptotically recovers the lowest true variance among all possible nudge propensities. A one-stage version of the algorithm is consistent but not necessarily optimal. We apply our method to a semi-synthetic example involving triage in an emergency department and find significant gains relative to a regression discontinuity design.

stat.ME↗

arXiv2025

Consistent and Scalable Composite Likelihood Estimation of Probit Models with Crossed Random Effects

Estimation of crossed random effects models commonly requires computational costs that grow faster than linearly in the sample size $N$, often as fast as $Ω(N^{3/2})$, making them unsuitable for large data sets. For non-Gaussian responses, integrating out the random effects to get a marginal likelihood brings significant challenges, especially for high dimensional integrals where the Laplace approximation might not be accurate. We develop a composite likelihood approach to probit models that replaces the crossed random effects model with some hierarchical models that require only one-dimensional integrals. We show how to consistently estimate the crossed effects model parameters from the hierarchical model fits. We find that the computation scales linearly in the sample size. We illustrate the method on about five million observations from Stitch Fix where the crossed effects formulation would require an integral of dimension larger than $700{,}000$.

stat.ME↗

arXiv2025

Skewness of a randomized quasi-Monte Carlo estimate

Some recent work on confidence intervals for randomized quasi-Monte Carlo (RQMC) sampling found a surprising result: ordinary Student $t$ 95% confidence intervals based on a modest number of replicates were seen to be very effective and even more reliable than some bootstrap $t$ intervals that were expected to be best. One potential explanation is that those RQMC estimates have small skewness. In this paper we give conditions under which the skewness is $O(n^ε)$ for any $ε>0$, so 'almost $O(1)$'. Under a random generator matrix model, we can improve this rate to $O(n^{-1/2+ε})$ with very high probability. We also improve some probabilistic bounds on the distribution of the quality parameter $t$ for a digital net in a prime base under random sampling of generator matrices.

math.NA↗

arXiv2025

Optimal Correlation for Bernoulli Trials with Covariates

Given covariates for $n$ units, each of which is to receive a treatment with probability $1/2$, we study the question of how best to correlate their treatment assignments to minimize the variance of the IPW estimator of the average treatment effect. Past work by \cite{bai2022} found that the optimal stratified experiment is a matched-pair design, where the matching depends on oracle knowledge of the distributions of potential outcomes given covariates. We study the strictly broader class of all admissible correlation structures, for which \cite{cytrynbaum2023} recently showed that the optimal design is to divide the units into two clusters and uniformly assign treatment to exactly one of them. This design can be computed by solving a 0-1 knapsack problem that uses the same oracle information. We derive a novel proof of this fact using a result about admissible Bernoulli correlations. We also show how to construct a shift-invariant version of the design that still improves on the optimal stratified design by ensuring that exactly half of the units are treated. A method with just two clusters is not robust to a bad proxy for the oracle, and we mitigate this with a hybrid that uses $O(n^α)$ clusters for $0<α<1$. Under certain assumptions, we also derive a CLT for the IPW estimator under these designs and a consistent estimator of the variance. We compare the proposed designs to the optimal stratified design in simulated examples and find strong performance.

stat.ME↗

arXiv2024

Multivariate Tie-breaker Designs

In a tie-breaker design (TBD), subjects with high values of a running variable are given some (usually desirable) treatment, subjects with low values are not, and subjects in the middle are randomized. TBDs are intermediate between regression discontinuity designs (RDDs) and randomized controlled trials (RCTs). TBDs allow a tradeoff between the resource allocation efficiency of an RDD and the statistical efficiency of an RCT. We study a model where the expected response is one multivariate regression for treated subjects and another for control subjects. We propose a prospective D-optimality, analogous to Bayesian optimal design, to understand design tradeoffs without reference to a specific data set. For given covariates, we show how to use convex optimization to choose treatment probabilities that optimize this criterion. We can incorporate a variety of constraints motivated by economic and ethical considerations. In our model, D-optimality for the treatment effect coincides with D-optimality for the whole regression, and, without constraints, an RCT is globally optimal. We show that a monotonicity constraint favoring more deserving subjects induces sparsity in the number of distinct treatment probabilities. We apply the convex optimization solution to a semi-synthetic example involving triage data from the MIMIC-IV-ED database.

stat.ME↗

arXiv2024

Computable error bounds for quasi-Monte Carlo using points with non-negative local discrepancy

Let $f:[0,1]^d\to\mathbb{R}$ be a completely monotone integrand as defined by Aistleitner and Dick (2015) and let points $\boldsymbol{x}_0,\dots,\boldsymbol{x}_{n-1}\in[0,1]^d$ have a non-negative local discrepancy (NNLD) everywhere in $[0,1]^d$. We show how to use these properties to get a non-asymptotic and computable upper bound for the integral of $f$ over $[0,1]^d$. An analogous non-positive local discrepancy (NPLD) property provides a computable lower bound. It has been known since Gabai (1967) that the two dimensional Hammersley points in any base $b\ge2$ have non-negative local discrepancy. Using the probabilistic notion of associated random variables, we generalize Gabai's finding to digital nets in any base $b\ge2$ and any dimension $d\ge1$ when the generator matrices are permutation matrices. We show that permutation matrices cannot attain the best values of the digital net quality parameter when $d\ge3$. As a consequence the computable absolutely sure bounds we provide come with less accurate estimates than the usual digital net estimates do in high dimensions. We are also able to construct high dimensional rank one lattice rules that are NNLD. We show that those lattices do not have good discrepancy properties: any lattice rule with the NNLD property in dimension $d\ge2$ either fails to be projection regular or has all its points on the main diagonal. Complete monotonicity is a very strict requirement that for some integrands can be mitigated via a control variate.

math.NA↗

arXiv2024

Biases in estimates of air pollution impacts: the role of omitted variables and measurement errors

Observational studies often use linear regression to assess the effect of ambient air pollution on outcomes of interest, such as human health indicators or crop yields. Yet pollution datasets are typically noisy and include only a subset of the potentially relevant pollutants, giving rise to both measurement error bias (MEB) and omitted variable bias (OVB). While it is well understood that these biases exist, less is understood about whether these biases tend to be positive or negative, even though it is sometimes falsely claimed that measurement error simply biases regression coefficient estimates towards zero. In this paper, we study the direction of these biases under the realistic assumptions that the concentrations of different types of air pollutants are positively correlated with each other and that each type of pollutant has a nonpositive association with the outcome variable. We demonstrate both theoretically and using simulations that under these two assumptions, the OVB will typically be negative and that more often than not the MEB for null pollutants or for pollutants that are perfectly measured will be negative. We also use a crop yield and air pollution dataset to show that these biases tend to be negative in the setting of our motivating application. We do this by introducing a validation scheme that does not require knowing the true coefficients. While this paper is motivated by studies assessing the effect of air pollutants on crop yields, the findings are also relevant to regression-based studies assessing the effect of air pollutants on human health outcomes. The validation scheme can also be used to empirically study OVB or MEB in other contexts.

stat.AP↗