arXiv subjects

Jonathan Terhorst

Publications and source records attributed to Jonathan Terhorst.

14 recordsLinked to original sources

arXiv2026

Rates of forgetting for the sequentially Markov coalescent

The sequentially Markov coalescent (SMC) is a Markov jump process which models correlations in local genealogies across a chromosome. It has been used as a theoretical tool for studying linkage disequilibrium and identity-by-descent, and it also forms the basis of a class of statistical procedures for estimating population history and inferring ancestry. In this paper, we study the rate at which SMC forgets its initial condition in the pairwise setting. For the embedded jump chain, we prove geometric ergodicity in total variation, with explicit constants. For the continuous process, by contrast, the total variation distance from stationarity decays as $\asymp 1/\ell$ in genetic distance $\ell$. We obtain analogous results for the closely related SMC' process using a novel time-change argument. One application of these results is to justify heuristic approximations used in the literature that treat distant loci as evolving independently.

math.PR↗

arXiv2026

Pair-based estimators of infection and removal rates for stochastic epidemic models

Stochastic epidemic models can estimate infection and removal rates, and derived quantities such as the basic reproductive number ($R_0$), when both infection and removal times are observed. In practice, however, removal times are often available while infection times are not, and existing methods that rely only on removal times can become unstable or biased. We study inference for stochastic SIR/SEIR models in a partial--observation setting. We develop imputation--based estimators that use a small calibration sample of fully observed infectious periods, derive closed--form expressions for the pairwise exposure terms they require, and use a studentized parametric bootstrap for bias correction and uncertainty quantification. In simulations, removal time--only methods performed poorly in moderate to large $R_0$ scenarios, while observing even tens of complete infectious periods substantially improved the estimation of the infection rate. A reanalysis of the 1861 Hagelloch measles outbreak under simulated missingness recovered stable qualitative differences in transmission between school classes. Based on our results, we advocate for the targeted collection of a modest number of complete infectious periods as a means of improving surveillance in the early stages of an epidemic.

stat.ME↗

arXiv2026

Population-scale Ancestral Recombination Graphs with tskit 1.0

Ancestral recombination graphs (ARGs) are an increasingly important component of population and statistical genetics. The tskit library has become key infrastructure for the field, providing an expressive and general representation of ARGs together with a suite of efficient fundamental operations. In this note, we announce tskit version 1.0, describe its underlying rationale, and document its stability guarantees. These guarantees provide a foundation for durable computational artefacts and support long-term reproducibility of code and analyses.

q-bio.PE↗

arXiv2025

Dirichlet moment tensors and the correspondence between admixture and mixture of product models

Understanding posterior contraction behavior in Bayesian hierarchical models is of fundamental importance, but progress in this question is relatively sparse in comparison to the theory of density estimation. In this paper, we study two classes of hierarchical models for grouped data, where observations within groups are exchangeable. Using moment tensor decomposition of the distribution of the latent variables, we establish a precise equivalence between the class of Admixture models (such as Latent Dirichlet Allocation) and the class of Mixture of products of multinomial distributions. This correspondence enables us to leverage the result from the latter class of models, which are more well-understood, so as to arrive at the identifiability and posterior contraction rates in both classes under conditions much weaker than in existing literature. For instance, our results shed light on cases where the topics are not linearly independent or the number of topics is misspecified in the admixture setting. Finally, we analyze individual documents' latent allocation performance via the borrowing of strength properties of hierarchical Bayesian modeling. Many illustrations and simulations are provided to support the theory.

math.ST↗

arXiv2024

Dendrogram of mixing measures: Hierarchical clustering and model selection for finite mixture models

We present a new way to summarize and select mixture models via the hierarchical clustering tree (dendrogram) constructed from an overfitted latent mixing measure. Our proposed method bridges agglomerative hierarchical clustering and mixture modeling. The dendrogram's construction is derived from the theory of convergence of the mixing measures, and as a result, we can both consistently select the true number of mixing components and obtain the pointwise optimal convergence rate for parameter estimation from the tree, even when the model parameters are only weakly identifiable. In theory, it explicates the choice of the optimal number of clusters in hierarchical clustering. In practice, the dendrogram reveals more information on the hierarchy of subpopulations compared to traditional ways of summarizing mixture models. Several simulation studies are carried out to support our theory. We also illustrate the methodology with an application to single-cell RNA sequence analysis.

stat.ME↗

arXiv2022

Exact and arbitrarily accurate non-parametric two-sample tests based on rank spacings

A common method for deriving non-parametric tests is to reformulate a parametric test in terms of sample ranks. Despite being distribution free (even in finite samples), the resulting tests often display remarkable asymptotic power properties, typically matching the efficiency of their parametric counterpart. Empirically, these favorable power properties have been shown to persist in non-asymptotic regimes as well, prompting the need for finite-sample characterizations of the corresponding rank-based statistics. Here, we provide such characterization for the family of weighted $p$-norms of rank spacings, which includes the classical tests of Mann-Whitney, Dixon, and various generalizations thereof. For $p=1$, we provide exact expressions for the involved distributions, while for $p>1$ we describe the associated moment sequences and derive an algorithm to recover the distributions of interest from these sequences in a fast and stable manner. We use this framework to develop a new family of non-parametric tests mirroring properties of generalized likelihood-ratios, prove new tail bounds for Dixon's and Greenwood's statistics, and prove a previously formulated conjecture regarding the global efficiency of rank-based tests against the $F$-test in the context of scale-families.

math.ST↗

arXiv2021

A linear adjustment based approach to posterior drift in transfer learning

We present a new model and methods for the posterior drift problem where the regression function in the target domain is modeled as a linear adjustment (on an appropriate scale) of that in the source domain, an idea that inherits the simplicity and the usefulness of generalized linear models and accelerated failure time models from the classical statistics literature, and study the theoretical properties of our proposed estimator in the binary classification problem. Our approach is shown to be flexible and applicable in a variety of statistical settings, and can be adopted to transfer learning problems in various domains including epidemiology, genetics and biomedicine. As a concrete application, we illustrate the power of our approach through mortality prediction for British Asians by borrowing strength from similar data from the larger pool of British Caucasians, using the UK Biobank data.

stat.ME↗

arXiv2020

Explaining Groups of Points in Low-Dimensional Representations

A common workflow in data exploration is to learn a low-dimensional representation of the data, identify groups of points in that representation, and examine the differences between the groups to determine what they represent. We treat this workflow as an interpretable machine learning problem by leveraging the model that learned the low-dimensional representation to help identify the key differences between the groups. To solve this problem, we introduce a new type of explanation, a Global Counterfactual Explanation (GCE), and our algorithm, Transitive Global Translations (TGT), for computing GCEs. TGT identifies the differences between each pair of groups using compressed sensing but constrains those pairwise differences to be consistent among all of the groups. Empirically, we demonstrate that TGT is able to identify explanations that accurately explain the model while being relatively sparse, and that these explanations match real patterns in the data.

cs.LG↗

arXiv2018

Inference of Population History using Coalescent HMMs: Review and Outlook

Studying how diverse human populations are related is of historical and anthropological interest, in addition to providing a realistic null model for testing for signatures of natural selection or disease associations. Furthermore, understanding the demographic histories of other species is playing an increasingly important role in conservation genetics. A number of statistical methods have been developed to infer population demographic histories using whole-genome sequence data, with recent advances focusing on allowing for more flexible modeling choices, scaling to larger data sets, and increasing statistical power. Here we review coalescent hidden Markov models, a powerful class of population genetic inference methods that can effectively utilize linkage disequilibrium information. We highlight recent advances, give advice for practitioners, point out potential pitfalls, and present possible future research directions.

q-bio.PE↗

arXiv2015

Fundamental limits on the accuracy of demographic inference based on the sample frequency spectrum

The sample frequency spectrum (SFS) of DNA sequences from a collection of individuals is a summary statistic which is commonly used for parametric inference in population genetics. Despite the popularity of SFS-based inference methods, currently little is known about the information-theoretic limit on the estimation accuracy as a function of sample size. Here, we show that using the SFS to estimate the size history of a population has a minimax error of at least $O(1/\log s)$, where $s$ is the number of independent segregating sites used in the analysis. This rate is exponentially worse than known convergence rates for many classical estimation problems in statistics. Another surprising aspect of our theoretical bound is that it does not depend on the dimension of the SFS, which is related to the number of sampled individuals. This means that, for a fixed number $s$ of segregating sites considered, using more individuals does not help to reduce the minimax error bound. Our result pertains to populations that have experienced a bottleneck, and we argue that it can be expected to apply to many populations in nature.

q-bio.PE↗

arXiv2015

Efficient computation of the joint sample frequency spectra for multiple populations

A wide range of studies in population genetics have employed the sample frequency spectrum (SFS), a summary statistic which describes the distribution of mutant alleles at a polymorphic site in a sample of DNA sequences. In particular, recently there has been growing interest in analyzing the joint SFS data from multiple populations to infer parameters of complex demographic histories, including variable population sizes, population split times, migration rates, admixture proportions, and so on. Although much methodological progress has been made, existing SFS-based inference methods suffer from numerical instability and high computational complexity when multiple populations are involved and the sample size is large. In this paper, we present new analytic formulas and algorithms that enable efficient computation of the expected joint SFS for multiple populations related by a complex demographic model with arbitrary population size histories (including piecewise exponential growth). Our results are implemented in a new software package called momi (MOran Models for Inference). Through an empirical study involving tens of populations, we demonstrate our improvements to numerical stability and computational complexity.

math.PR↗

arXiv2014

Communication-Efficient Distributed Dual Coordinate Ascent

Communication remains the most significant bottleneck in the performance of distributed optimization algorithms for large-scale machine learning. In this paper, we propose a communication-efficient framework, CoCoA, that uses local computation in a primal-dual setting to dramatically reduce the amount of necessary communication. We provide a strong convergence rate analysis for this class of algorithms, as well as experiments on real-world distributed datasets with implementations in Spark. In our experiments, we find that as compared to state-of-the-art mini-batch versions of SGD and SDCA algorithms, CoCoA converges to the same .001-accurate solution quality on average 25x as quickly.

cs.LG↗

arXiv2014

SMaSH: A Benchmarking Toolkit for Human Genome Variant Calling

Motivation: Computational methods are essential to extract actionable information from raw sequencing data, and to thus fulfill the promise of next-generation sequencing technology. Unfortunately, computational tools developed to call variants from human sequencing data disagree on many of their predictions, and current methods to evaluate accuracy and computational performance are ad-hoc and incomplete. Agreement on benchmarking variant calling methods would stimulate development of genomic processing tools and facilitate communication among researchers. Results: We propose SMaSH, a benchmarking methodology for evaluating human genome variant calling algorithms. We generate synthetic datasets, organize and interpret a wide range of existing benchmarking data for real genomes, and propose a set of accuracy and computational performance metrics for evaluating variant calling methods on this benchmarking data. Moreover, we illustrate the utility of SMaSH to evaluate the performance of some leading single nucleotide polymorphism (SNP), indel, and structural variant calling algorithms. Availability: We provide free and open access online to the SMaSH toolkit, along with detailed documentation, at smash.cs.berkeley.edu.

q-bio.GN↗

arXiv2011

The Kalmanson Complex

Let X be a finite set of cardinality n. The Kalmanson complex K_n is the simplicial complex whose vertices are non-trivial X-splits, and whose facets are maximal circular split systems over X. In this paper we examine K_n from three perspectives. In addition to the T-theoretic description, we show that K_n has a geometric realization as the Kalmanson conditions on a finite metric. A third description arises in terms of binary matrices which possess the circular ones property. We prove the equivalence of these three definitions. This leads to a simplified proof of the well-known equivalence between Kalmanson and circular decomposable metrics, as well as a partial description of the f-vector of K_n.

math.CO↗