arXiv subjects

Simon Segert

Publications and source records attributed to Simon Segert.

7 recordsLinked to original sources

arXiv2025

A Probabilistic Basis for Low-Rank Matrix Learning

Low rank inference on matrices is widely conducted by optimizing a cost function augmented with a penalty proportional to the nuclear norm $\Vert \cdot \Vert_*$. However, despite the assortment of computational methods for such problems, there is a surprising lack of understanding of the underlying probability distributions being referred to. In this article, we study the distribution with density $f(X)\propto e^{-\lambda\Vert X\Vert_*}$, finding many of its fundamental attributes to be analytically tractable via differential geometry. We use these facts to design an improved MCMC algorithm for low rank Bayesian inference as well as to learn the penalty parameter $\lambda$, obviating the need for hyperparameter tuning when this is difficult or impossible. Finally, we deploy these to improve the accuracy and efficiency of low rank Bayesian matrix denoising and completion algorithms in numerical experiments.

stat.ML↗

arXiv2025

A Group Theoretic Analysis of the Symmetries Underlying Base Addition and Their Learnability by Neural Networks

A major challenge in the use of neural networks both for modeling human cognitive function and for artificial intelligence is the design of systems with the capacity to efficiently learn functions that support radical generalization. At the roots of this is the capacity to discover and implement symmetry functions. In this paper, we investigate a paradigmatic example of radical generalization through the use of symmetry: base addition. We present a group theoretic analysis of base addition, a fundamental and defining characteristic of which is the carry function -- the transfer of the remainder, when a sum exceeds the base modulus, to the next significant place. Our analysis exposes a range of alternative carry functions for a given base, and we introduce quantitative measures to characterize these. We then exploit differences in carry functions to probe the inductive biases of neural networks in symmetry learning, by training neural networks to carry out base addition using different carries, and comparing efficacy and rate of learning as a function of their structure. We find that even simple neural networks can achieve radical generalization with the right input format and carry function, and that learnability is closely correlated with carry function structure. We then discuss the relevance this has for cognitive science and machine learning.

cs.LG↗

arXiv2025

The generalized Alice HH vs Bob HT problem

In 2024, Daniel Litt posed a simple coinflip game pitting Alice's "Heads-Heads" vs Bob's "Heads-Tails": who is more likely to win if they score 1 point per occurrence of their substring in a sequence of n fair coinflips? This attracted over 1 million views on X and quickly spawned several articles explaining the counterintuitive solution. We study the generalized game, where the set of coin outcomes, {Heads, Tails}, is generalized to an arbitrary finite alphabet A, and where Alice's and Bob's substrings are any finite A-strings of the same length. We find that the winner of Litt's game can be determined by a single quantity which measures the amount of prefix/suffix self-overlaps in each string; whoever's string has more overlaps loses. For example, "Heads-Tails" beats "Heads-Heads" in the original problem because "Heads-Heads" has a prefix/suffix overlap of length 1 while "Heads-Tails" has none. The method of proof is to develop a precise Edgeworth expansion for discreteMarkov chains, and apply this to calculate Alice's and Bob's probability to win the game correct to order O(1/n).

math.PR↗

arXiv2024

A proof that HT is more likely to outnumber HH than vice versa in a sequence of n coin flips

Consider the following probability puzzle: A fair coin is flipped n times. For each HT in the resulting sequence, Bob gets a point, and for each HH Alice gets a point. Who is more likely to win? We provide a proof that Bob wins more often for every n>=3. As a byproduct, we derive the asymptotic form of the difference in win probabilities, and obtain an efficient algorithms for their calculation.

math.CO↗

arXiv2023

Flat Minima in Linear Estimation and an Extended Gauss Markov Theorem

We consider the problem of linear estimation, and establish an extension of the Gauss-Markov theorem, in which the bias operator is allowed to be non-zero but bounded with respect to a matrix norm of Schatten type. We derive simple and explicit formulas for the optimal estimator in the cases of Nuclear and Spectral norms (with the Frobenius case recovering ridge regression). Additionally, we analytically derive the generalization error in multiple random matrix ensembles, and compare with Ridge regression. Finally, we conduct an extensive simulation study, in which we show that the cross-validated Nuclear and Spectral regressors can outperform Ridge in several circumstances.

cs.LG↗

arXiv2023

The Relational Bottleneck as an Inductive Bias for Efficient Abstraction

A central challenge for cognitive science is to explain how abstract concepts are acquired from limited experience. This has often been framed in terms of a dichotomy between connectionist and symbolic cognitive models. Here, we highlight a recently emerging line of work that suggests a novel reconciliation of these approaches, by exploiting an inductive bias that we term the relational bottleneck. In that approach, neural networks are constrained via their architecture to focus on relations between perceptual inputs, rather than the attributes of individual inputs. We review a family of models that employ this approach to induce abstractions in a data-efficient manner, emphasizing their potential as candidate models for the acquisition of abstract concepts in the human mind and brain.

cs.AI↗

arXiv2023

Beyond Transformers for Function Learning

The ability to learn and predict simple functions is a key aspect of human intelligence. Recent works have started to explore this ability using transformer architectures, however it remains unclear whether this is sufficient to recapitulate the extrapolation abilities of people in this domain. Here, we propose to address this gap by augmenting the transformer architecture with two simple inductive learning biases, that are directly adapted from recent models of abstract reasoning in cognitive science. The results we report demonstrate that these biases are helpful in the context of large neural network models, as well as shed light on the types of inductive learning biases that may contribute to human abilities in extrapolation.

cs.LG↗