arXiv subjects

Albert Kjøller Jacobsen

Publications and source records attributed to Albert Kjøller Jacobsen.

6 recordsLinked to original sources

arXiv2026

Beyond Laplace: Closed-form wrapped Gaussian posterior approximations on statistical manifolds

In Bayesian statistics, the Laplace approximation provides a computationally efficient approximation to posterior distributions. However, its Gaussian form restricts it to elliptical shapes, limiting its ability to capture important posterior features such as skewness, heavy tails, and narrow high-probability regions. Recent work has addressed this limitation by exploiting Riemannian geometry to push forward Gaussian distributions from the tangent space to a chosen manifold, referred to wrapped Gaussians. While offering greater flexibility, they introduce substantial computational challenges. Sampling requires solving geodesic equations through the exponential map and density evaluation additionally depends on the logarithmic map and Jacobi fields, involving costly differential equation solvers alongside full matrices inverse operations, Christoffel symbols and curvature tensors. To overcome these limitations, we employ the theory of contrast functions to derive tractable approximations of the logarithmic and exponential maps on statistical manifolds endowed with the Fisher--Rao metric and the prior distribution geometry. The resulting methodology bypass the need to compute these geometric quantities and numerical solvers thereby removing the principal computational bottlenecks of existing wrapped Gaussian approaches. Empirical results across a range of models demonstrate that the proposed approximation captures complex posterior geometries while remaining orders of magnitude faster than current geometric state-of-the-art approximation.

stat.ME↗

arXiv2026

Don't Stop Me Yet: Sampling Loss Minima via Dissipative Riemannian Mechanics

The minima of modern neural network loss functions are typically not isolated, rather they form connected components of reparameterization invariant solutions on the training data. Analytically characterizing these solutions is a hard problem, but sampling approaches are feasible. By construction, existing methods either spread over low-loss regions, and thus do not sample reparameterization invariant solutions exactly, or are inherently local, which limits exploration of other minima valleys. We propose sampling such reparameterization invariant models using a dynamical system based on kinetic energy, subject to a gravitational pull and a friction term that dissipates energy from the system. Our proposed sampler, DiMS, is guaranteed to sample exactly from the minimum level sets and depends on physically motivated hyperparameters which allows control over the exploration capabilities of the sampler. We consider uncertainty quantification in Bayesian inference as the motivating problem and observe improved performance compared to previously proposed approaches.

cs.LG↗

arXiv2026

Reducing Memorisation in Generative Models via Riemannian Bayesian Inference

Modern generative models can produce realistic samples, however, balancing memorisation and generalisation remains an open problem. We approach this challenge from a Bayesian perspective by focusing on the parameter space of flow matching and diffusion models and constructing a predictive posterior that better captures the variability of the data distribution. In particular, we capture the geometry of the loss using a Riemannian metric and leverage a flexible approximate posterior that adapts to the local structure of the loss landscape. This approach allows us to sample generative models that resemble the original model, but exhibit reduced memorisation. Empirically, we demonstrate that the proposed approach reduces memorisation while preserving generalisation. Further, we provide a theoretical analysis of our method, which explains our findings. Overall, our work illustrates how considering the geometry of the loss enables effective use of the parameter space, even for complex high-dimensional generative models.

cs.LG↗

arXiv2025

Staying on the Manifold: Geometry-Aware Noise Injection

It has been shown that perturbing the input during training implicitly regularises the gradient of the learnt function, leading to smoother models and enhancing generalisation. However, previous research mostly considered the addition of ambient noise in the input space, without considering the underlying structure of the data. In this work, we propose several strategies of adding geometry-aware input noise that accounts for the lower dimensional manifold the input space inhabits. We start by projecting ambient Gaussian noise onto the tangent space of the manifold. In a second step, the noise sample is mapped on the manifold via the associated geodesic curve. We also consider Brownian motion noise, which moves in random steps along the manifold. We show that geometry-aware noise leads to improved generalisation and robustness to hyperparameter selection on highly curved manifolds, while performing at least as well as training without noise on simpler manifolds. Our proposed framework extends to data manifolds approximated by generative models and we observe similar trends on the MNIST digits dataset.

cs.LG↗

arXiv2025

Monge SAM: Robust Reparameterization-Invariant Sharpness-Aware Minimization Based on Loss Geometry

Recent studies on deep neural networks show that flat minima of the loss landscape correlate with improved generalization. Sharpness-aware minimization (SAM) efficiently finds flat regions by updating the parameters according to the gradient at an adversarial perturbation. The perturbation depends on the Euclidean metric, making SAM non-invariant under reparametrizations, which blurs sharpness and generalization. We propose Monge SAM (M-SAM), a reparametrization invariant version of SAM by considering a Riemannian metric in the parameter space induced naturally by the loss surface. Compared to previous approaches, M-SAM works under any modeling choice, relies only on mild assumptions while being as computationally efficient as SAM. We theoretically argue that M-SAM varies between SAM and gradient descent (GD), which increases robustness to hyperparameter selection and reduces attraction to suboptimal equilibria like saddle points. We demonstrate this behavior both theoretically and empirically on a multi-modal representation alignment task.

cs.LG↗

arXiv2025

How Redundant Is the Transformer Stack in Speech Representation Models?

Self-supervised speech representation models, particularly those leveraging transformer architectures, have demonstrated remarkable performance across various tasks such as speech recognition, speaker identification, and emotion detection. Recent studies on transformer models revealed a high redundancy between layers and the potential for significant pruning, which we will investigate here for transformer-based speech representation models. We perform a detailed analysis of layer similarity in speech representation models using three similarity metrics: cosine similarity, centered kernel alignment, and mutual nearest-neighbor alignment. Our findings reveal a block-like structure of high similarity, suggesting two main processing steps and significant redundancy of layers. We demonstrate the effectiveness of pruning transformer-based speech representation models without the need for post-training, achieving up to 40% reduction in transformer layers while maintaining over 95% of the model's predictive capacity. Furthermore, we employ a knowledge distillation method to substitute the entire transformer stack with mimicking layers, reducing the network size 95-98% and the inference time by up to 94%. This substantial decrease in computational load occurs without considerable performance loss, suggesting that the transformer stack is almost completely redundant for downstream applications of speech representation models.

eess.AS↗