arXiv subjects

Jordan Rodu

Publications and source records attributed to Jordan Rodu.

13 recordsLinked to original sources

arXiv2026

Semantic Alignment of AI Models: Concept Collapse, Checkpoint Dynamics, and Cross-Lingual Transfer

Language model benchmarking is a difficult task. Outcome reasoning alone does not test the model's conceptualization of language and popular open-source benchmarks are quickly saturated or ingested as training data. It is important to test the model's output, but augmenting these tests by characterizing semantic structure gives more insight to how models relate abstract concepts. However, the high dimensional embedding spaces are not easy to interpret. This work demonstrates how topological methods can be used to rigorously compare these spaces to low dimensional and interpretable baselines like ontologies and curated knowledge graphs. These multi-modal alignment tests make it possible to track model adaptations and test phrase understanding across multiple languages.

cs.CL↗

arXiv2026

Persistent Convolution: A Topological Framework for AI Alignment Testing and Semantic Space Characterization

Modern opaque AI models prize performance over interpretability, which makes testing difficult. However, formal statistical tests conducted on a model's embedding space can provide robust characterizations of semantic structure, concept separation, and knowledge graph alignment. Model developers would benefit from a model comparison technique that leverages human-curated knowledge structures to test alignment. The scale of the input space for even relatively simple tasks motivates the need for alignment checks that augment standard outcome reasoning. This work develops and demonstrates a topology-based multi-modal alignment test to make deployment, selection, and comparison of opaque models more interpretable. These methods also offer an intuitive connection to possibility theory and a unified decision theoretic framework from data to deployment.

stat.ML↗

arXiv2026

Data Reuse and the Long Shadow of Error: Splitting, Subsampling, and Prospectively Managing Inferential Errors

When multiple investigators analyze a common dataset, the data reuse induces dependence across testing procedures, affecting the distribution of errors. Existing techniques of managing dependent tests require either cross-study coordination or post-hoc correction. These methods do not apply to the current practice of uncoordinated groups of researchers independently evaluating hypotheses on a shared dataset. We investigate the use of subsampling techniques implemented at the level of individual investigators to remedy dependence with minimal coordination. To this end, we establish the asymptotic joint normality of test statistics for the class of asymptotically linear test statistics, decomposing the covariance matrix as the product of a data overlap term and a test statistic association term. This decomposition shows that controlling data overlap is sufficient to control dependence, which we formalize through the notion of Expected Variance Ratio. This enables the closed form derivation of the variance of the joint rejection region under the global null as a function of pairwise correlations of test statistics. We adopt mean-variance portfolio theory to measure risk, defining the Expected Variance Ratio (EVR) as the ratio of the expected variance of the Type I error count to the independent baseline. We show that data splitting is asymptotically optimal among rules that ensure exact independence. We then use concentration inequalities to establish that subsampling techniques implementable by individual investigators can ensure an EVR close to $1$. Finally, we show that such subsampling techniques are able to simultaneously perform a number of tests while ensuring sufficient power and that the bounded EVR is $O\left(\frac{1}{r^2}\right)$ compared to data splitting's $O\left(\frac{1}{r}\right)$, where $r$ is the per-statistic fraction of data required.

math.ST↗

arXiv2026

Synthetic Data, Information, and Prior Knowledge: Why Synthetic Data Augmentation to Boost Sample Doesn't Work for Statistical Inference

The use of synthetic data to deidentify data and to improve predictive models is well-attested to. The augmentation of datasets using synthetically generated data is an alluring proposition: in the best case, it generates realistic data \textit{in silico} at a fraction of the cost of authentic data which may be found \textit{in vivo} or \textit{in vitro}. This poses novel epistemic challenges. We contend that synthetic data augmentation is best understood as a novel way of accounting for prior knowledge. In this manuscript, we propose a definition of synthetic distributions and analyze how synthetic data augmentation interplays with standard accounts of maximum likelihood and Bayesian estimation. We observe that the marginal Fisher information contributed by synthetic data processes is subject to fundamental bounds, and enumerate obstacles to the use of synthetic data augmentation to aid in inferential tasks. We then articulate a Bayesian formulation of the way that synthetic data augmentation can be coherently understood, but argue that naive approaches to the specification of the prior are epistemically unjustifiable. This suggests that enhanced scrutiny must be placed on identifying justifiable priors to warrant the use and inclusion of data drawn from specific synthetic distributions. While our analysis shows the challenges and limitations of using synthetic data augmentation to improve upon traditional statistical model reasoning, it does suggest that augmentation is the principal approach analysts using outcome reasoning (i.e. using train/test splits to justify the analysis) can constrain an otherwise high-dimensional model space, providing an alternative to trying to encode the constraints into the potentially complex architecture of the algorithm.

stat.ME↗

arXiv2025

Data Gluttony: Epistemic Risks, Dependent Testing and Data Reuse in Large Datasets

Large-scale registries have collected vast amounts of data which has enabled investigators to efficiently conduct studies of observational data. Common practice is for investigators to use all data meeting the inclusion criteria of their study to perform their analysis. We term this common practice data gluttony. It has apparent formal justification insofar as this approach maximizes per-study power. But this comes at a cost: data reuse affects the shape of the tail distribution of inferential errors. Using the theory of risk orderings we demonstrate how positively dependent testing procedures result in strictly riskier distributions of inferential error. We identify two remedies to this state of affairs: research portfolio optimization and what we term data temperance. Research portfolio optimization requires that we formulate the enterprise of inference in a utility theoretic framework: associated to each hypothesis to be evaluated is some utility dependent on its truth as well as the impact of the statistical decision rendered on the basis of the data. Under certain models of data governance, this approach can be used to optimally allocate data usage across multiple inferential tasks. On the other hand, data temperance is a more flexible strategy for managing the distribution of inferential errors. Data temperance is the principle that an investigator use only as much data as is necessary to perform the task at hand. This is possible due to the diminishing marginal returns in power and precision in sample size. We analyze the effectiveness of data temperance at reducing the dependence across testing and develop a theory of the capacity of a static database to sustain large numbers of inferential tasks with low probability of inducing pairwise dependent testing procedures.

math.ST↗

arXiv2023

Change Point Detection with Conceptors

Offline change point detection retrospectively locates change points in a time series. Many nonparametric methods that target i.i.d. mean and variance changes fail in the presence of nonlinear temporal dependence, and model based methods require a known, rigid structure. For the at most one change point problem, we propose use of a conceptor matrix to learn the characteristic dynamics of a baseline training window with arbitrary dependence structure. The associated echo state network acts as a featurizer of the data, and change points are identified from the nature of the interactions between the features and their relationship to the baseline state. This model agnostic method can suggest potential locations of interest that warrant further study. We prove that, under mild assumptions, the method provides a consistent estimate of the true change point, and quantile estimates are produced via a moving block bootstrap of the original data. The method is evaluated with clustering metrics and Type 1 error control on simulated data, and applied to publicly available neural data from rats experiencing bouts of non-REM sleep prior to exploration of a radial maze. With sufficient spacing, the framework provides a simple extension to the sparse, multiple change point problem.

stat.ML↗

arXiv2023

Nonlinear Permuted Granger Causality

Granger causal inference is a contentious but widespread method used in fields ranging from economics to neuroscience. The original definition addresses the notion of causality in time series by establishing functional dependence conditional on a specified model. Adaptation of Granger causality to nonlinear data remains challenging, and many methods apply in-sample tests that do not incorporate out-of-sample predictability, leading to concerns of model overfitting. To allow for out-of-sample comparison, a measure of functional connectivity is explicitly defined using permutations of the covariate set. Artificial neural networks serve as featurizers of the data to approximate any arbitrary, nonlinear relationship, and consistent estimation of the variance for each permutation is shown under certain conditions on the featurization process and the model residuals. Performance of the permutation method is compared to penalized variable selection, naive replacement, and omission techniques via simulation, and it is applied to neuronal responses of acoustic stimuli in the auditory cortex of anesthetized rats. Targeted use of the Granger causal framework, when prior knowledge of the causal mechanisms in a dataset are limited, can help to reveal potential predictive relationships between sets of variables that warrant further study.

stat.ME↗

arXiv2023

Bridging the Usability Gap: Theoretical and Methodological Advances for Spectral Learning of Hidden Markov Models

The Baum-Welch (B-W) algorithm is the most widely accepted method for inferring hidden Markov models (HMM). However, it is prone to getting stuck in local optima, and can be too slow for many real-time applications. Spectral learning of HMMs (SHMM), based on the method of moments (MOM) has been proposed in the literature to overcome these obstacles. Despite its promises, asymptotic theory for SHMM has been elusive, and the long-run performance of SHMM can degrade due to unchecked propagation of error. In this paper, we (1) provide an asymptotic distribution for the approximate error of the likelihood estimated by SHMM, (2) propose a novel algorithm called projected SHMM (PSHMM) that mitigates the problem of error propagation, and (3) develop online learning variants of both SHMM and PSHMM that accommodate potential nonstationarity. We compare the performance of SHMM with PSHMM and estimation through the B-W algorithm on both simulated data and data from real world applications, and find that PSHMM not only retains the computational advantages of SHMM, but also provides more robust estimation and forecasting.

stat.ML↗

arXiv2021

Trees in transformers: a theoretical analysis of the Transformer's ability to represent trees

Transformer networks are the de facto standard architecture in natural language processing. To date, there are no theoretical analyses of the Transformer's ability to capture tree structures. We focus on the ability of Transformer networks to learn tree structures that are important for tree transduction problems. We first analyze the theoretical capability of the standard Transformer architecture to learn tree structures given enumeration of all possible tree backbones, which we define as trees without labels. We then prove that two linear layers with ReLU activation function can recover any tree backbone from any two nonzero, linearly independent starting backbones. This implies that a Transformer can learn tree structures well in theory. We conduct experiments with synthetic data and find that the standard Transformer achieves similar accuracy compared to a Transformer where tree position information is explicitly encoded, albeit with slower convergence. This confirms empirically that Transformers can learn tree structures.

cs.CL↗

arXiv2020

When black box algorithms are (not) appropriate: a principled prediction-problem ontology

In the 1980s a new, extraordinarily productive way of reasoning about algorithms emerged. In this paper, we introduce the term "outcome reasoning" to refer to this form of reasoning. Though outcome reasoning has come to dominate areas of data science, it has been under-discussed and its impact under-appreciated. For example, outcome reasoning is the primary way we reason about whether ``black box'' algorithms are performing well. In this paper we analyze outcome reasoning's most common form (i.e., as "the common task framework") and its limitations. We discuss why a large class of prediction-problems are inappropriate for outcome reasoning. As an example, we find the common task framework does not provide a foundation for the deployment of an algorithm in a real world situation. Building off of its core features, we identify a class of problems where this new form of reasoning can be used in deployment. We purposefully develop a novel framework so both technical and non-technical people can discuss and identify key features of their prediction problem and whether or not it is suitable for outcome reasoning.

stat.OT↗

arXiv2015

Locating recombination hot spots in genomic sequences through the singular value decomposition

Locating recombination hotspots in genomic data is an important but difficult task. Current methods frequently rely on estimating complicated models at high computational cost. In this paper we develop an extremely fast, scalable method for inferring recombination hot spots in a population of genomic sequences that is based on the singular value decomposition. Our method performs well in several synthetic data scenarios. We also apply our technique to a real data investigation of the evolution of drug therapy resistance in a population of HIV genomic sequences. Finally, we compare our method both on real and simulated data to a state of the art algorithm.

stat.AP↗

arXiv2012

Two Step CCA: A new spectral method for estimating vector models of words

Unlabeled data is often used to learn representations which can be used to supplement baseline features in a supervised learner. For example, for text applications where the words lie in a very high dimensional space (the size of the vocabulary), one can learn a low rank "dictionary" by an eigen-decomposition of the word co-occurrence matrix (e.g. using PCA or CCA). In this paper, we present a new spectral method based on CCA to learn an eigenword dictionary. Our improved procedure computes two set of CCAs, the first one between the left and right contexts of the given word and the second one between the projections resulting from this CCA and the word itself. We prove theoretically that this two-step procedure has lower sample complexity than the simple single step procedure and also illustrate the empirical efficacy of our approach and the richness of representations learned by our Two Step CCA (TSCCA) procedure on the tasks of POS tagging and sentiment classification.

cs.CL↗

arXiv2012

Spectral dimensionality reduction for HMMs

Hidden Markov Models (HMMs) can be accurately approximated using co-occurrence frequencies of pairs and triples of observations by using a fast spectral method in contrast to the usual slow methods like EM or Gibbs sampling. We provide a new spectral method which significantly reduces the number of model parameters that need to be estimated, and generates a sample complexity that does not depend on the size of the observation vocabulary. We present an elementary proof giving bounds on the relative accuracy of probability estimates from our model. (Correlaries show our bounds can be weakened to provide either L1 bounds or KL bounds which provide easier direct comparisons to previous work.) Our theorem uses conditions that are checkable from the data, instead of putting conditions on the unobservable Markov transition matrix.

stat.ML↗