arXiv subjects

Harry Crane

Publications and source records attributed to Harry Crane.

At least 19 recordsLinked to original sources

arXiv2023

Root and community inference on the latent growth process of a network

Many existing statistical models for networks overlook the fact that many real world networks are formed through a growth process. To address this, we introduce the PAPER (Preferential Attachment Plus Erdős--Rényi) model for random networks, where we let a random network G be the union of a preferential attachment (PA) tree T and additional Erdős--Rényi (ER) random edges. The PA tree component captures the underlying growth/recruitment process of a network where vertices and edges are added sequentially, while the ER component can be regarded as random noise. Given only a single snapshot of the final network G, we study the problem of constructing confidence sets for the early history, in particular the root node, of the unobserved growth process; the root node can be patient zero in a disease infection network or the source of fake news in a social media network. We propose an inference algorithm based on Gibbs sampling that scales to networks with millions of nodes and provide theoretical analysis showing that the expected size of the confidence set is small so long as the noise level of the ER edges is not too large. We also propose variations of the model in which multiple growth processes occur simultaneously, reflecting the growth of multiple communities, and we use these models to provide a new approach to community detection.

stat.ME↗

arXiv2021

Inference on the History of a Randomly Growing Tree

The spread of infectious disease in a human community or the proliferation of fake news on social media can be modeled as a randomly growing tree-shaped graph. The history of the random growth process is often unobserved but contains important information such as the source of the infection. We consider the problem of statistical inference on aspects of the latent history using only a single snapshot of the final tree. Our approach is to apply random labels to the observed unlabeled tree and analyze the resulting distribution of the growth process, conditional on the final outcome. We show that this conditional distribution is tractable under a shape-exchangeability condition, which we introduce here, and that this condition is satisfied for many popular models for randomly growing trees such as uniform attachment, linear preferential attachment and uniform attachment on a $D$-regular tree. For inference of the root under shape-exchangeability, we propose O(n log n) time algorithms for constructing confidence sets with valid frequentist coverage as well as bounds on the expected size of the confidence sets. We also provide efficient sampling algorithms that extend our methods to a wide class of inference problems.

math.PR↗

arXiv2018

Pattern Avoidance for Random Permutations

Using techniques from Poisson approximation, we prove explicit error bounds on the number of permutations that avoid any pattern. Most generally, we bound the total variation distance between the joint distribution of pattern occurrences and a corresponding joint distribution of independent Bernoulli random variables, which as a corollary yields a Poisson approximation for the distribution of the number of occurrences of any pattern. We also investigate occurrences of consecutive patterns in random Mallows permutations, of which uniform random permutations are a special case. These bounds allow us to estimate the probability that a pattern occurs any number of times and, in particular, the probability that a random permutation avoids a given pattern.

math.CO↗

arXiv2017

Why "Redefining Statistical Significance" Will Not Improve Reproducibility and Could Make the Replication Crisis Worse

A recent proposal to "redefine statistical significance" (Benjamin, et al. Nature Human Behaviour, 2017) claims that false positive rates "would immediately improve" by factors greater than two and replication rates would double simply by changing the conventional cutoff for 'statistical significance' from P<0.05 to P<0.005. I analyze the veracity of these claims, focusing especially on how Benjamin, et al neglect the effects of P-hacking in assessing the impact of their proposal. My analysis shows that once P-hacking is accounted for the perceived benefits of the lower threshold all but disappear, prompting two main conclusions: (i) The claimed improvements to false positive rate and replication rate in Benjamin, et al (2017) are exaggerated and misleading. (ii) There are plausible scenarios under which the lower cutoff will make the replication crisis worse.

stat.AP↗

arXiv2017

Relational exchangeability

A relationally exchangeable structure is a random combinatorial structure whose law is invariant with respect to relabeling its relations, as opposed to its elements. Aside from exchangeable random partitions, examples include edge exchangeable random graphs and hypergraphs, path exchangeable processes, and a range of other network-like structures that arise in statistical applications. We prove a de Finetti-type structure theorem for the general class of relationally exchangeable structures.

math.ST↗

arXiv2017

Rethinking probabilistic prediction in the wake of the 2016 U.S. presidential election

To many statisticians and citizens, the outcome of the most recent U.S. presidential election represents a failure of data-driven methods on the grandest scale. This impression has led to much debate and discussion about how the election predictions went awry -- Were the polls inaccurate? Were the models wrong? Did we misinterpret the probabilities? -- and how they went right -- Perhaps the analyses were correct even though the predictions were wrong, that's just the nature of probabilistic forecasting. With this in mind, we analyze the election outcome with respect to a core set of effectiveness principles. Regardless of whether and how the election predictions were right or wrong, we argue that they were ineffective in conveying the extent to which the data was informative of the outcome and the level of uncertainty in making these assessments. Among other things, our analysis sheds light on the shortcomings of the classical interpretations of probability and its communication to consumers in the form of predictions. We present here an alternative approach, based on a notion of validity, which offers two immediate insights for predictive inference. First, the predictions are more conservative, arguably more realistic, and come with certain guarantees on the probability of an erroneous prediction. Second, our approach easily and naturally reflects the (possibly substantial) uncertainty about the model by outputting plausibilities instead of probabilities. Had these simple steps been taken by the popular prediction outlets, the election outcome may not have been so shocking.

stat.OT↗

arXiv2016

A framework for statistical network modeling

Basic principles of statistical inference are commonly violated in network data analysis. Under the current approach, it is often impossible to identify a model that accommodates known empirical behaviors, possesses crucial inferential properties, and accurately models the data generating process. In the absence of one or more of these properties, sensible inference from network data cannot be assured. Our proposed framework decomposes every network model into a (relatively) exchangeable data generating process} and a sampling mechanism that relates observed data to the population network. This framework, which encompasses all models in current use as well as many new models, such as edge exchangeable and relationally exchangeable models, that lie outside the existing paradigm, offers a sound context within which to develop theory and methods for network analysis.

math.ST↗

arXiv2016

Combinatorial Levy processes

Combinatorial Levy processes evolve on general state spaces of countable combinatorial structures. In this setting, the usual Levy process properties of stationary, independent increments are defined in an unconventional way in terms of the symmetric difference operation on sets. In discrete time, the description of combinatorial Levy processes gives rise to the notion of combinatorial random walks. These processes behave differently than random walks and Levy processes on other state spaces. Standard examples include processes on sets, graphs, and n-ary relations, but the framework permits far more general possibilities. The main theorems characterize both finite and infinite state space combinatorial Levy processes by a unique sigma-finite measure. Under the additional assumption of exchangeability, we obtain a more explicit characterization by which every exchangeable combinatorial Levy process corresponds to a Poisson point process on the same state space. Associated behavior of the projection into a space of limiting objects reflects certain structural features of the covering process.

math.PR↗

arXiv2016

Edge exchangeable models for network data

Exchangeable models for countable vertex-labeled graphs cannot replicate the large sample behaviors of sparsity and power law degree distribution observed in many network datasets. Out of this mathematical impossibility emerges the question of how network data can be modeled in a way that reflects known empirical behaviors and respects basic statistical principles. We address this question by observing that edges, not vertices, act as the statistical units in networks constructed from interaction data, making a theory of edge-labeled networks more natural for many applications. In this context we introduce the concept of {\em edge exchangeability}, which unlike its vertex exchangeable counterpart admits models for networks with sparse and/or power law structure. Our characterization of edge exchangeable networks gives rise to a class of nonparametric models, akin to graphon models in the vertex exchangeable setting. Within this class, we identify a tractable family of distributions with a clear interpretation and suitable theoretical properties, whose significance in estimation, prediction, and testing we demonstrate.

math.ST↗

arXiv2016

The probability of avoiding consecutive patterns in the Mallows distribution

We use various combinatorial and probabilistic techniques to study growth rates for the probability that a random permutation from the Mallows distribution avoids consecutive patterns. The Mallows distribution behaves like a $q$-analogue of the uniform distribution by weighting each permutation $π$ by $q^{inv(π)}$, where $inv(π)$ is the number of inversions in $π$ and $q$ is a positive, real-valued parameter. We prove that the growth rate exists for all patterns and all $q>0$, and we generalize Goulden and Jackson's cluster method to keep track of the number of inversions in permutations avoiding a given consecutive pattern. Using singularity analysis, we approximate the growth rates for length-3 patterns, monotone patterns, and non-overlapping patterns starting with 1, and we compare growth rates between different patterns. We also use Stein's method to show that, under certain assumptions on $q$, the length of $σ$, and $inv(σ)$, the number of occurrences of a given pattern $σ$ is well approximated by the normal distribution.

math.CO↗

arXiv2016

The structure of combinatorial Markov processes

Every exchangeable Feller process taking values in a suitably nice combinatorial state space can be constructed by a system of iterated random Lipschitz functions. In discrete time, the construction proceeds by iterated application of independent, identically distributed functions, while in continuous time the random functions occur as the atoms of a time homogeneous Poisson point process. We further show that every exchangeable Feller process projects to a Feller process in an appropriate limit space, akin to the projection of partition-valued processes into the ranked-simplex and graph-valued processes into the space of graph limits. Together, our main theorems establish common structural features shared by all exchangeable combinatorial Feller processes, regardless of the dynamics or resident state space, thereby generalizing behaviors previously observed for exchangeable coalescent and fragmentation processes as well as other combinatorial stochastic processes. If, in addition, an exchangeable Feller process evolves on a state space satisfying the $n$-disjoint amalgamation property for all $n\geq1$, then its jump measure can be decomposed explicitly in the sense of Lévy--Itô--Khintchine.

math.PR↗

arXiv2016

Relative exchangeability with equivalence relations

We describe an Aldous--Hoover-type characterization of random relational structures that are exchangeable relative to a fixed structure which may have various equivalence relations. Our main theorem gives the common generalization of the results on relative exchangeability due to Ackerman \cite{Ackerman2015} and Crane and Towsner \cite{CraneTowsner2015} and hierarchical exchangeability results due to Austin and Panchenko \cite{AustinPanchenko2014}.

math.LO↗

arXiv2015

Reversible Markov structures on divisible set partitions

We study $k$-divisible partition structures, which are families of random set partitions whose block sizes are divisible by an integer $k=1,2,\ldots$. In this setting, exchangeability corresponds to the usual invariance under relabeling by arbitrary permutations; however, for $k>1$, the ordinary deletion maps on partitions no longer preserve divisibility, and so a random deletion procedure is needed to obtain a partition structure. We describe explicit Chinese restaurant-type seating rules for generating families of exchangeable $k$-divisible partitions that are consistent under random deletion. We further introduce the notion of {\em Markovian partition structures}, which are ensembles of exchangeable Markov chains on $k$-divisible partitions that are consistent under a random process of {\em Markovian deletion}. The Markov chains we study are reversible and refine the class of Markov chains introduced in {\em J.\ Appl.\ Probab.}~{\bf48}(3):778--791.

math.ST↗

arXiv2015

Relatively exchangeable structures

We study random relational structures that are \emph{relatively exchangeable}---that is, whose distributions are invariant under the automorphisms of a reference structure $\mathfrak{M}$. When $\mathfrak{M}$ has {\em trivial definable closure}, every relatively exchangeable structure satisfies a general Aldous--Hoover-type representation. If $\mathfrak{M}$ satisfies the stronger properties of {\em ultrahomogeneity} and {\em $n$-disjoint amalgamation property} ($n$-DAP) for every $n\geq1$, then relatively exchangeable structures have a more precise description whereby each component depends locally on $\mathfrak{M}$.

math.LO↗

arXiv2015

Community detection for interaction networks

In many applications, it is common practice to obtain a network from interaction counts by thresholding each pairwise count at a prescribed value. Our analysis calls attention to the dependence of certain methods, notably Newman--Girvan modularity, on the choice of threshold. Essentially, the threshold either separates the network into clusters automatically, making the algorithm's job trivial, or erases all structure in the data, rendering clustering impossible. By fitting the original interaction counts as given, we show that minor modifications to classical statistical methods outperform the prevailing approaches for community detection from interaction datasets. We also introduce a new hidden Markov model for inferring community structures that vary over time. We demonstrate each of these features on three real datasets: the karate club dataset, voting data from the U.S.\ Senate (2001--2003), and temporal voting data for the U.S. Supreme Court (1990--2004).

cs.SI↗

arXiv2015

Atypical scaling behavior persists in real world interaction networks

Scale-free power law structure describes complex networks derived from a wide range of real world processes. The extensive literature focuses almost exclusively on networks with power law exponent strictly larger than 2, which can be explained by constant vertex growth and preferential attachment. The complementary scale-free behavior in the range between 1 and 2 has been mostly neglected as atypical because there is no known generating mechanism to explain how networks with this property form. However, empirical observations reveal that scaling in this range is an inherent feature of real world networks obtained from repeated interactions within a population, as in social, communication, and collaboration networks. A generative model explains the observed phenomenon through the realistic dynamics of constant edge growth and a positive feedback mechanism. Our investigation, therefore, yields a novel empirical observation grounded in a strong theoretical basis for its occurrence.

cs.SI↗

arXiv2015

Exchangeable Markov processes on graphs: Feller case

The transition law of every exchangeable Feller process on the space of countable graphs is determined by a $σ$-finite measure on the space of $\{0,1\}\times\{0,1\}$-valued arrays. In discrete-time, this characterization amounts to a construction from an independent, identically distributed sequence of exchangeable random functions. In continuous-time, the behavior is enriched by a Lévy--Itô-type decomposition of the jump measure into mutually singular components that govern global, vertex-level, and edge-level dynamics. Furthermore, every such process almost surely projects to a Feller process in the space of graph limits.

math.PR↗

arXiv2015

Time-varying network models

We introduce the exchangeable rewiring process for modeling time-varying networks. The process fulfills fundamental mathematical and statistical properties and can be easily constructed from the novel operation of random rewiring. We derive basic properties of the model, including consistency under subsampling, exchangeability, and the Feller property. A reversible sub-family related to the Erdős-Rényi model arises as a special case.

math.ST↗