arXiv subjects

Andrew D. Smith

Publications and source records attributed to Andrew D. Smith.

3 recordsLinked to original sources

arXiv2025

Measures and Models of Non-Monotonic Dependence

A margin-free measure of bivariate association generalizing Spearman's rho to the case of non-monotonic dependence is defined in terms of two square integrable functions on the unit interval. Properties of generalized Spearman correlation are investigated when the functions are piecewise continuous and strictly monotonic, with particular focus on the special cases where the functions are drawn from orthonormal bases defined by Legendre polynomials and cosine functions. For continuous random variables, generalized Spearman correlation is treated as a copula-based measure and shown to depend on a pair of uniform-distribution-preserving (udp) transformations determined by the underlying functions. Bounds for generalized Spearman correlation are derived and a novel technique referred to as stochastic inversion of udp transformations is used to construct singular copulas that attain the bounds and parametric copulas with densities that interpolate between the bounds and model different degrees of non-monotonic dependence. Sample analogues of generalized Spearman correlation are proposed and their asymptotic and small-sample properties are investigated. Potential applications of the theory are demonstrated including: exploratory analyses of the dependence structures of datasets and their symmetries; elicitation of functions maximizing generalized Spearman correlation via expansions in orthonormal basis functions; and construction of tractable probability densities to model a wide variety of non-monotonic dependencies.

stat.ME↗

arXiv2022

On attainability of Kendall's tau matrices and concordance signatures

Methods are developed for checking and completing systems of bivariate and multivariate Kendall's tau concordance measures in applications where only partial information about dependencies between variables is available. The concept of a concordance signature of a multivariate continuous distribution is introduced; this is the vector of concordance probabilities for margins of all orders. It is shown that every attainable concordance signature is equal to the concordance signature of a unique mixture of the extremal copulas, that is the copulas with extremal correlation matrices consisting exclusively of 1's and -1's. A method of estimating an attainable concordance signature from data is derived and shown to correspond to using standard estimates of Kendall's tau in the absence of ties. The set of attainable Kendall rank correlation matrices of multivariate continuous distributions is proved to be identical to the set of convex combinations of extremal correlation matrices, a set known as the cut polytope. A methodology for testing the attainability of concordance signatures using linear optimization and convex analysis is provided. The elliptical copulas are shown to yield a strict subset of the attainable concordance signatures as well as a strict subset of the attainable Kendall rank correlation matrices; the Student t copula is seen to converge, as the degrees of freedom tend to zero, to a mixture of extremal copulas sharing its concordance signature with all elliptical distributions that have the same correlation matrix. A characterization of the attainable signatures of equiconcordant copulas is given.

math.ST↗

arXiv2018

Estimating the number of species to attain sufficient representation in a random sample

The statistical problem of using an initial sample to estimate the number of species in a larger sample has found important applications in fields far removed from ecology. Here we address the general problem of estimating the number of species that will be represented by at least a number r of observations in a future sample. The number r indicates species with sufficient observations, which are commonly used as a necessary condition for any robust statistical inference. We derive a procedure to construct consistent estimators that apply universally for a given population: once constructed, they can be evaluated as a simple function of r. Our approach is based on a relation between the number of species represented at least r times and the higher derivatives of the expected number of species discovered per unit of time. Combining this relation with a rational function approximation, we propose nonparametric estimators that are accurate for both large values of r and long-range extrapolations. We further show that our estimators retain asymptotic behaviors that are essential for applications on large-scale datasets. We evaluate the performance of this approach by both simulation and real data applications for inferences of the vocabulary of Shakespeare and Dickens, the topology of a Twitter social network, and molecular diversity in DNA sequencing data.

stat.ME↗