arXiv subjects

Neil D. Lawrence

Publications and source records attributed to Neil D. Lawrence.

At least 19 recordsLinked to original sources

arXiv2026

The No Barber Principle: Towards Formalised Selection in the Inaccessible Game

The inaccessible game (Lawrence, 2025, 2026) is an information-theoretic dynamical system governed by three information loss axioms, a marginal entropy conservation constraint and maximum entropy dynamics. In this paper we look at selection in the game. Our aim is to develop a selection policy for the game rules based on a minimal set of assumptions. We seek necessary consistency constraints for self-determining dynamical systems. Specifically, we suggest that rules that quantify over distinctions they cannot internally represent risk impredicative-style circularity. Our criterion is motivated by an analogy with Russell's paradox. We formulate a no-barber principle which prohibits dynamics that appeal to external adjudicators or structure lying outside the system. To motivate our principle we examine Russell's paradox through its structural formalisation as a Lawvere diagonalisation. The marginal-entropy conservation in the game is a nontrivial entropy constraint which prohibits external structure. Through the no-barber principle we argue (i) the classical category FinProb, in which Shannon entropy is characterised, is cartesian and provides canonical diagonal (copying) maps that make Lawvere-style constructions expressible and is structurally incompatible with the no-copying instantiation of the no-barber principle studied here. (ii) the noncommutative category NCFinProb, in which von Neumann entropy is characterised, is symmetric monoidal and lacks canonical copying maps, making it a more natural candidate for the game's internal language.

math.CT↗

arXiv2026

On the Value of Tokeniser Pretraining in Physics Foundation Models

We investigate the impact of tokeniser pretraining on the accuracy and efficiency of physics emulation. Modern high-resolution simulations produce vast volumes of data spanning diverse physical regimes and scales. Training foundation models to learn the dynamics underlying such data enables the modelling of complex multiphysics phenomena, especially in data-limited settings. The emerging class of physics foundation models typically aims to learn two tasks jointly: (i) extracting compact representations of high-resolution spatiotemporal data, and (ii) capturing governing physical dynamics. However, learning both tasks from scratch simultaneously can impede the effectiveness of either process. We show that pretraining the tokeniser with an autoencoding objective prior to training the dynamics model enhances computational efficiency for physics emulation. Notably, the magnitude of this benefit depends on domain alignment: pretraining on the same physical system as the emulation task yields the largest improvements, while pretraining on other systems provides moderate gains. In-domain pretraining reduces VRMSE by 64% after 10,500 training steps compared to training from scratch. To our knowledge, this is the first systematic investigation of tokeniser pretraining for physics foundation models. We further introduce flexible spatiotemporal compression operations that extend causal convolutions to support runtime-adjustable compression ratios, enabling efficient adaptation to diverse downstream tasks. Our findings provide practical guidance for training efficient physics emulators and highlight the importance of strategic pretraining data selection.

cs.LG↗

arXiv2026

The Origin of the Inaccessible Game

The inaccessible game is an information-geometric framework where dynamics of information loss emerge from maximum entropy production under marginal-entropy conservation. We study the game's starting state, the origin. Classical Shannon entropy forbids a representation with zero joint entropy and positive marginal entropies: non-negativity of conditional entropy rules this out. Replacing Shannon with von Neumann entropy within the Baez Fritz Leinster Parzygnat categorical framework removes this obstruction and admits a well-defined origin: a globally pure state with maximally mixed marginals, selected up to local-unitary equivalence. At this LME origin, marginal-entropy conservation becomes a second-order geometric condition. Because the marginal-entropy sum is saturated termwise, the constraint gradient vanishes and first-order tangency is vacuous; admissible directions are selected by the kernel of the constraint Hessian, characterised by the marginal-preserving tangent space. We derive the constrained gradient flow in the matrix exponential family and show that, as the origin is approached, the affine time parameter degenerates. This motivates an axiomatically distinguished reparametrisation, entropy time $t$, defined by $dH/dt = c$ for fixed constant $c>0$. In this parametrisation, the infinite affine-time approach to the boundary maps to a finite entropy-time interval. The constrained dynamics split into a symmetric dissipative component realising SEA and a reversible component represented as unitary evolution. As in the classical game, marginal-entropy conservation is equivalent to conservation of a sum of local modular Hamiltonian expectations, a state-dependent "modular energy"; in Gibbs regimes where local modular generators become approximately parameter-invariant, this reduces to familiar fixed-energy constraints from nonequilibrium thermodynamics.

cs.IT↗

arXiv2026

Optimising for Energy Efficiency and Performance in Machine Learning

The ubiquity of machine learning (ML) and the demand for ever-larger models bring an increase in energy consumption and environmental impact. However, little is known about the energy scaling laws in ML, and existing research focuses on training cost -- ignoring the larger cost of inference. Furthermore, tools for measuring the energy consumption of ML do not provide actionable feedback. To address these gaps, we developed Energy Consumption Optimiser (ECOpt): a hyperparameter tuner that optimises for energy efficiency and model performance. ECOpt quantifies the trade-off between these metrics as an interpretable Pareto frontier. This enables ML practitioners to make informed decisions about energy cost and environmental impact, while maximising the benefit of their models and complying with new regulations. Using ECOpt, we show that parameter and floating-point operation counts can be unreliable proxies for energy consumption, and observe that the energy efficiency of Transformer models for text generation is relatively consistent across hardware. These findings motivate measuring and publishing the energy metrics of ML models. We further show that ECOpt can have a net positive environmental impact and use it to uncover seven models for CIFAR-10 that improve upon the state of the art, when considering accuracy and energy efficiency together.

cs.LG↗

arXiv2025

The Inaccessible Game

In this paper we introduce the inaccessible game, an information-theoretic dynamical system constructed from four axioms. The first three axioms are known and define \emph{information loss} in the system. The fourth is a novel \emph{information isolation} axiom that assumes our system is isolated from observation, making it observer-independent and exchangeable. Under this isolation axiom, total marginal entropy is conserved: $\sum_i h_i = C$. We consider maximum entropy production in the game and show that the dynamics exhibit a GENERIC-like structure combining reversible and irreversible components.

cs.IT↗

arXiv2025

Revisiting Deep AC-OPF

Recent work has proposed machine learning (ML) approaches as fast surrogates for solving AC optimal power flow (AC-OPF), with claims of significant speed-ups and high accuracy. In this paper, we revisit these claims through a systematic evaluation of ML models against a set of simple yet carefully designed linear baselines. We introduce OPFormer-V, a transformer-based model for predicting bus voltages, and compare it to both the state-of-the-art DeepOPF-V model and simple linear methods. Our findings reveal that, while OPFormer-V improves over DeepOPF-V, the relative gains of the ML approaches considered are less pronounced than expected. Simple linear baselines can achieve comparable performance. These results highlight the importance of including strong linear baselines in future evaluations.

eess.SY↗

arXiv2025

Transformers as Unrolled Inference in Probabilistic Laplacian Eigenmaps: An Interpretation and Potential Improvements

We propose a probabilistic interpretation of transformers as unrolled inference steps assuming a probabilistic Laplacian Eigenmaps model from the ProbDR framework. Our derivation shows that at initialisation, transformers perform "linear" dimensionality reduction. We also show that within the transformer block, a graph Laplacian term arises from our arguments, rather than an attention matrix (which we interpret as an adjacency matrix). We demonstrate that simply subtracting the identity from the attention matrix (and thereby taking a graph diffusion step) improves validation performance on a language model and a simple vision transformer.

cs.LG↗

arXiv2025

Code Roulette: How Prompt Variability Affects LLM Code Generation

Code generation is one of the most active areas of application of Large Language Models (LLMs). While LLMs lower barriers to writing code and accelerate development process, the overall quality of generated programs depends on the quality of given prompts. Specifically, functionality and quality of generated code can be sensitive to user's background and familiarity with software development. It is therefore important to quantify LLM's sensitivity to variations in the input. To this end we propose an evaluation pipeline for LLM code generation with a focus on measuring sensitivity to prompt augmentations, completely agnostic to a specific programming tasks and LLMs, and thus widely applicable. We provide extensive experimental evidence illustrating utility of our method and share our code for the benefit of the community.

cs.SE↗

arXiv2025

Protein Language Model Zero-Shot Fitness Predictions are Improved by Inference-only Dropout

Protein Language Models (PLMs) such as ESM2 have been shown to be capable of zero-shot prediction of critical scalar properties of proteins (fitness). In this work, we show that injecting a dropout layer at inference time between a PLM's featurizer/embedding layer and its transformer, and averaging its output akin to Monte-Carlo dropout increases zero-shot performance on a subset of the ProteinGym dataset. This is the case even when the model was not trained with dropouts to begin with, and does not require retraining or finetuning of the PLM. A dropout of 0.1 seems performant across all models.

cs.LG↗

arXiv2024

The Systems Engineering Approach in Times of Large Language Models

Using Large Language Models (LLMs) to address critical societal problems requires adopting this novel technology into socio-technical systems. However, the complexity of such systems and the nature of LLMs challenge such a vision. It is unlikely that the solution to such challenges will come from the Artificial Intelligence (AI) community itself. Instead, the Systems Engineering approach is better equipped to facilitate the adoption of LLMs by prioritising the problems and their context before any other aspects. This paper introduces the challenges LLMs generate and surveys systems research efforts for engineering AI-based systems. We reveal how the systems engineering principles have supported addressing similar issues to the ones LLMs pose and discuss our findings to provide future directions for adopting LLMs.

cs.AI↗

arXiv2024

On Feature Learning for Titi Monkey Activity Detection

This paper, a technical summary of our preceding publication, introduces a robust machine learning framework for the detection of vocal activities of Coppery titi monkeys. Utilizing a combination of MFCC features and a bidirectional LSTM-based classifier, we effectively address the challenges posed by the small amount of expert-annotated vocal data available. Our approach significantly reduces false positives and improves the accuracy of call detection in bioacoustic research. Initial results demonstrate an accuracy of 95\% on instance predictions, highlighting the effectiveness of our model in identifying and classifying complex vocal patterns in environmental audio recordings. Moreover, we show how call classification can be done downstream, paving the way for real-world monitoring.

cs.SD↗

arXiv2024

Towards One Model for Classical Dimensionality Reduction: A Probabilistic Perspective on UMAP and t-SNE

This paper shows that dimensionality reduction methods such as UMAP and t-SNE, can be approximately recast as MAP inference methods corresponding to a model introduced in Ravuri et al. (2023), that describes the graph Laplacian (an estimate of the data precision matrix) using a Wishart distribution, with a mean given by a non-linear covariance function evaluated on the latents. This interpretation offers deeper theoretical and semantic insights into such algorithms, and forging a connection to Gaussian process latent variable models by showing that well-known kernels can be used to describe covariances implied by graph Laplacians. We also introduce tools with which similar dimensionality reduction methods can be studied.

stat.ML↗

arXiv2024

Requirements are All You Need: The Final Frontier for End-User Software Engineering

What if end users could own the software development lifecycle from conception to deployment using only requirements expressed in language, images, video or audio? We explore this idea, building on the capabilities that generative Artificial Intelligence brings to software generation and maintenance techniques. How could designing software in this way better serve end users? What are the implications of this process for the future of end-user software engineering and the software development lifecycle? We discuss the research needed to bridge the gap between where we are today and these imagined systems of the future.

cs.SE↗

arXiv2024

Scalable Amortized GPLVMs for Single Cell Transcriptomics Data

Dimensionality reduction is crucial for analyzing large-scale single-cell RNA-seq data. Gaussian Process Latent Variable Models (GPLVMs) offer an interpretable dimensionality reduction method, but current scalable models lack effectiveness in clustering cell types. We introduce an improved model, the amortized stochastic variational Bayesian GPLVM (BGPLVM), tailored for single-cell RNA-seq with specialized encoder, kernel, and likelihood designs. This model matches the performance of the leading single-cell variational inference (scVI) approach on synthetic and real-world COVID datasets and effectively incorporates cell-cycle and batch information to reveal more interpretable latent structures as we demonstrate on an innate immunity dataset.

stat.ML↗

arXiv2024

Self-sustaining Software Systems (S4): Towards Improved Interpretability and Adaptation

Software systems impact society at different levels as they pervasively solve real-world problems. Modern software systems are often so sophisticated that their complexity exceeds the limits of human comprehension. These systems must respond to changing goals, dynamic data, unexpected failures, and security threats, among other variable factors in real-world environments. Systems' complexity challenges their interpretability and requires autonomous responses to dynamic changes. Two main research areas explore autonomous systems' responses: evolutionary computing and autonomic computing. Evolutionary computing focuses on software improvement based on iterative modifications to the source code. Autonomic computing focuses on optimising systems' performance by changing their structure, behaviour, or environment variables. Approaches from both areas rely on feedback loops that accumulate knowledge from the system interactions to inform autonomous decision-making. However, this knowledge is often limited, constraining the systems' interpretability and adaptability. This paper proposes a new concept for interpretable and adaptable software systems: self-sustaining software systems (S4). S4 builds knowledge loops between all available knowledge sources that define modern software systems to improve their interpretability and adaptability. This paper introduces and discusses the S4 concept.

cs.SE↗

arXiv2023

Automated discovery of trade-off between utility, privacy and fairness in machine learning models

Machine learning models are deployed as a central component in decision making and policy operations with direct impact on individuals' lives. In order to act ethically and comply with government regulations, these models need to make fair decisions and protect the users' privacy. However, such requirements can come with decrease in models' performance compared to their potentially biased, privacy-leaking counterparts. Thus the trade-off between fairness, privacy and performance of ML models emerges, and practitioners need a way of quantifying this trade-off to enable deployment decisions. In this work we interpret this trade-off as a multi-objective optimization problem, and propose PFairDP, a pipeline that uses Bayesian optimization for discovery of Pareto-optimal points between fairness, privacy and utility of ML models. We show how PFairDP can be used to replicate known results that were achieved through manual constraint setting process. We further demonstrate effectiveness of PFairDP with experiments on multiple models and datasets.

cs.LG↗

arXiv2023

Dimensionality Reduction as Probabilistic Inference

Dimensionality reduction (DR) algorithms compress high-dimensional data into a lower dimensional representation while preserving important features of the data. DR is a critical step in many analysis pipelines as it enables visualisation, noise reduction and efficient downstream processing of the data. In this work, we introduce the ProbDR variational framework, which interprets a wide range of classical DR algorithms as probabilistic inference algorithms in this framework. ProbDR encompasses PCA, CMDS, LLE, LE, MVU, diffusion maps, kPCA, Isomap, (t-)SNE, and UMAP. In our framework, a low-dimensional latent variable is used to construct a covariance, precision, or a graph Laplacian matrix, which can be used as part of a generative model for the data. Inference is done by optimizing an evidence lower bound. We demonstrate the internal consistency of our framework and show that it enables the use of probabilistic programming languages (PPLs) for DR. Additionally, we illustrate that the framework facilitates reasoning about unseen data and argue that our generative models approximate Gaussian processes (GPs) on manifolds. By providing a unified view of DR, our framework facilitates communication, reasoning about uncertainties, model composition, and extensions, particularly when domain knowledge is present.

stat.ML↗

arXiv2023

Causal fault localisation in dataflow systems

Dataflow computing was shown to bring significant benefits to multiple niches of systems engineering and has the potential to become a general-purpose paradigm of choice for data-driven application development. One of the characteristic features of dataflow computing is the natural access to the dataflow graph of the entire system. Recently it has been observed that these dataflow graphs can be treated as complete graphical causal models, opening opportunities to apply causal inference techniques to dataflow systems. In this demonstration paper we aim to provide the first practical validation of this idea with a particular focus on causal fault localisation. We provide multiple demonstrations of how causal inference can be used to detect software bugs and data shifts in multiple scenarios with three modern dataflow engines.

cs.SE↗