arXiv subjects

Michael Munn

Publications and source records attributed to Michael Munn.

At least 19 recordsLinked to original sources

arXiv2025

Equivalence of Context and Parameter Updates in Modern Transformer Blocks

Recent research has established that the impact of context in a vanilla transformer can be represented implicitly by forming a token-dependent, rank-1 patch to its MLP weights. This work extends that foundational theory to the diverse architectures of modern Large Language Models. We first demonstrate a precise, analytical solution for a Gemma-style transformer block, proving that the entire effect of a context can be perfectly mapped to rank-1 patches on its MLP weight matrices and a patch to the RMSNorm scale. We then generalize this result, providing a constructive proof and algorithm for multi-layer models. To unify these findings, we introduce a general framework centered on two core properties: input controllability and output controllability. We prove that a perfect implicit weight patch is possible for any MLP block where the inner function is input-controllable and the outer function is output-controllable. This provides a simpler and more powerful lens for understanding how transformer models transmute prompts into effective weights. This setup generalizes to a wide range of modern LLM architectures including gating, pre-/post-norm, mixture of experts and sequential/parallel transformer blocks.

cs.LG↗

arXiv2025

Transmuting prompts into weights

A growing body of research has demonstrated that the behavior of large language models can be effectively controlled at inference time by directly modifying their internal states, either through vector additions to their activations or through updates to their weight matrices. These techniques, while powerful, are often guided by empirical heuristics, such as deriving ``steering vectors'' from the average activations of contrastive prompts. Building on the foundational work of Dherin et al. (2025), who discovered that a prompt's influence mathematically maps to token-dependent implicit weight updates and introduced the initial concept of a static thought patch for prompt compression, we elevate this framework into a robust algorithm for direct model editing. We derive a principled method for condensing this transient information into token-independent thought vectors and thought matrices. These constructs provide a theoretical explanation for existing vector-and-matrix-based model editing techniques and offer a direct, computationally-grounded method for transmuting textual input into reusable weight updates for complex architectures and new knowledge injection.

cs.LG↗

arXiv2025

On residual network depth

Deep residual architectures, such as ResNet and the Transformer, have enabled models of unprecedented depth, yet a formal understanding of why depth is so effective remains an open question. A popular intuition, following Veit et al. (2016), is that these residual networks behave like ensembles of many shallower models. Our key finding is an explicit analytical formula that verifies this ensemble perspective, proving that increasing network depth is mathematically equivalent to expanding the size of this implicit ensemble. Furthermore, our expansion reveals a hierarchical ensemble structure in which the combinatorial growth of computation paths leads to an explosion in the output signal, explaining the historical necessity of normalization layers in training deep models. This insight offers a first principles explanation for the historical dependence on normalization layers and sheds new light on a family of successful normalization-free techniques like SkipInit and Fixup. However, while these previous approaches infer scaling factors through optimizer analysis or a heuristic analogy to Batch Normalization, our work offers the first explanation derived directly from the network's inherent functional structure. Specifically, our Residual Expansion Theorem reveals that scaling each residual module provides a principled solution to taming the combinatorial explosion inherent to these architectures. We further show that this scaling acts as a capacity controls that also implicitly regularizes the model's complexity.

cs.LG↗

arXiv2025

Learning without training: The implicit dynamics of in-context learning

One of the most striking features of Large Language Models (LLMs) is their ability to learn in-context. Namely at inference time an LLM is able to learn new patterns without any additional weight update when these patterns are presented in the form of examples in the prompt, even if these patterns were not seen during training. The mechanisms through which this can happen are still largely unknown. In this work, we show that the stacking of a self-attention layer with an MLP allows the transformer block to implicitly modify the weights of the MLP layer according to the context. We argue through theoretical analysis and experimentation that this simple mechanism may help explain why LLMs demonstrate capabilities of in-context learning, beyond what is captured during training. Specifically, we show that a standard forward pass with context is mathematically equivalent to a forward pass without context but with the MLP weights updated by a minimal low-rank update representing the context.

cs.CL↗

arXiv2025

Learning by solving differential equations

Modern deep learning algorithms use variations of gradient descent as their main learning methods. Gradient descent can be understood as the simplest Ordinary Differential Equation (ODE) solver; namely, the Euler method applied to the gradient flow differential equation. Since Euler, many ODE solvers have been devised that follow the gradient flow equation more precisely and more stably. Runge-Kutta (RK) methods provide a family of very powerful explicit and implicit high-order ODE solvers. However, these higher-order solvers have not found wide application in deep learning so far. In this work, we evaluate the performance of higher-order RK solvers when applied in deep learning, study their limitations, and propose ways to overcome these drawbacks. In particular, we explore how to improve their performance by naturally incorporating key ingredients of modern neural network optimizers such as preconditioning, adaptive learning rates, and momentum.

cs.LG↗

arXiv2025

How iteration order influences convergence and stability in deep learning

Despite exceptional achievements, training neural networks remains computationally expensive and is often plagued by instabilities that can degrade convergence. While learning rate schedules can help mitigate these issues, finding optimal schedules is time-consuming and resource-intensive. This work explores theoretical issues concerning training stability in the constant-learning-rate (i.e., without schedule) and small-batch-size regime. Surprisingly, we show that the composition order of gradient updates affects stability and convergence in gradient-based optimizers. We illustrate this new line of thinking using backward-SGD, which produces parameter iterates at each step by reverting the usual forward composition order of batch gradients. Our theoretical analysis shows that in contractive regions (e.g., around minima) backward-SGD converges to a point while the standard forward-SGD generally only converges to a distribution. This leads to improved stability and convergence which we demonstrate experimentally. While full backward-SGD is computationally intensive in practice, it highlights that the extra freedom of modifying the usual iteration composition by reusing creatively previous batches at each optimization step may have important beneficial effects in improving training. Our experiments provide a proof of concept supporting this phenomenon. To our knowledge, this represents a new and unexplored avenue in deep learning optimization.

cs.LG↗

arXiv2024

A Bayesian Model Selection Criterion for Selecting Pretraining Checkpoints

Recent advances in artificial intelligence have been fueled by the development of foundation models such as BERT, GPT, T5, and Vision Transformers. These models are first pretrained on vast and diverse datasets and then adapted to specific downstream tasks, often with significantly less data. However, the mechanisms behind the success of this ubiquitous pretrain-then-adapt paradigm remain underexplored, particularly the characteristics of pretraining checkpoints that enhance downstream adaptation. We introduce a Bayesian model selection criterion, called the downstream free energy, which quantifies a checkpoint's adaptability by measuring the concentration of nearby favorable parameters for the downstream task. We demonstrate that this Bayesian model selection criterion can be effectively implemented without access to the downstream data or prior knowledge of the downstream task. Furthermore, we provide empirical evidence that the criterion reliably correlates with improved finetuning performance, offering a principled approach to predicting model adaptability.

cs.LG↗

arXiv2024

A Margin-based Multiclass Generalization Bound via Geometric Complexity

There has been considerable effort to better understand the generalization capabilities of deep neural networks both as a means to unlock a theoretical understanding of their success as well as providing directions for further improvements. In this paper, we investigate margin-based multiclass generalization bounds for neural networks which rely on a recent complexity measure, the geometric complexity, developed for neural networks. We derive a new upper bound on the generalization error which scales with the margin-normalized geometric complexity of the network and which holds for a broad family of data distributions and model classes. Our generalization bound is empirically investigated for a ResNet-18 model trained with SGD on the CIFAR-10 and CIFAR-100 datasets with both original and random labels.

stat.ML↗

arXiv2024

The Impact of Geometric Complexity on Neural Collapse in Transfer Learning

Many of the recent remarkable advances in computer vision and language models can be attributed to the success of transfer learning via the pre-training of large foundation models. However, a theoretical framework which explains this empirical success is incomplete and remains an active area of research. Flatness of the loss surface and neural collapse have recently emerged as useful pre-training metrics which shed light on the implicit biases underlying pre-training. In this paper, we explore the geometric complexity of a model's learned representations as a fundamental mechanism that relates these two concepts. We show through experiments and theory that mechanisms which affect the geometric complexity of the pre-trained network also influence the neural collapse. Furthermore, we show how this effect of the geometric complexity generalizes to the neural collapse of new classes as well, thus encouraging better performance on downstream tasks, particularly in the few-shot setting.

cs.LG↗

arXiv2023

Unified Functional Hashing in Automatic Machine Learning

The field of Automatic Machine Learning (AutoML) has recently attained impressive results, including the discovery of state-of-the-art machine learning solutions, such as neural image classifiers. This is often done by applying an evolutionary search method, which samples multiple candidate solutions from a large space and evaluates the quality of each candidate through a long training process. As a result, the search tends to be slow. In this paper, we show that large efficiency gains can be obtained by employing a fast unified functional hash, especially through the functional equivalence caching technique, which we also present. The central idea is to detect by hashing when the search method produces equivalent candidates, which occurs very frequently, and this way avoid their costly re-evaluation. Our hash is "functional" in that it identifies equivalent candidates even if they were represented or coded differently, and it is "unified" in that the same algorithm can hash arbitrary representations; e.g. compute graphs, imperative code, or lambda functions. As evidence, we show dramatic improvements on multiple AutoML domains, including neural architecture search and algorithm discovery. Finally, we consider the effect of hash collisions, evaluation noise, and search distribution through empirical analysis. Altogether, we hope this paper may serve as a guide to hashing techniques in AutoML.

cs.LG↗

arXiv2022

Why neural networks find simple solutions: the many regularizers of geometric complexity

In many contexts, simpler models are preferable to more complex models and the control of this model complexity is the goal for many methods in machine learning such as regularization, hyperparameter tuning and architecture design. In deep learning, it has been difficult to understand the underlying mechanisms of complexity control, since many traditional measures are not naturally suitable for deep neural networks. Here we develop the notion of geometric complexity, which is a measure of the variability of the model function, computed using a discrete Dirichlet energy. Using a combination of theoretical arguments and empirical results, we show that many common training heuristics such as parameter norm regularization, spectral norm regularization, flatness regularization, implicit gradient regularization, noise regularization and the choice of parameter initialization all act to control geometric complexity, providing a unifying framework in which to characterize the behavior of deep learning models.

cs.LG↗

arXiv2021

The Geometric Occam's Razor Implicit in Deep Learning

In over-parameterized deep neural networks there can be many possible parameter configurations that fit the training data exactly. However, the properties of these interpolating solutions are poorly understood. We argue that over-parameterized neural networks trained with stochastic gradient descent are subject to a Geometric Occam's Razor; that is, these networks are implicitly regularized by the geometric model complexity. For one-dimensional regression, the geometric model complexity is simply given by the arc length of the function. For higher-dimensional settings, the geometric model complexity depends on the Dirichlet energy of the function. We explore the relationship between this Geometric Occam's Razor, the Dirichlet energy and other known forms of implicit regularization. Finally, for ResNets trained on CIFAR-10, we observe that Dirichlet energy measurements are consistent with the action of this implicit Geometric Occam's Razor.

cs.LG↗

arXiv2020

On Weak Super Ricci Flow through Neckpinch

In this article, we study the Ricci flow neckpinch in the context of metric measure spaces. We introduce the notion of a Ricci flow metric measure spacetime and of a weak (refined) super Ricci flow associated to convex cost functions (cost functions which are increasing convex functions of the distance function). Our definition of a weak super Ricci flow is based on the coupled contraction property for suitably defined diffusions on maximal diffusion components. In our main theorem, we show that if a non-degenerate spherical neckpinch can be continued beyond the singular time by a smooth forward evolution then the corresponding Ricci flow metric measure spacetime through the singularity is a weak super Ricci flow for a (and therefore for all) convex cost functions if and only if the single point pinching phenomenon holds at singular times; i.e., if singularities form on a finite number of totally geodesic hypersurfaces of the form $\{x\} \times \sphere^n$. We also show the spacetime is a refined weak super Ricci flow if and only if the flow is a smooth Ricci flow with possibly singular final time.

math.DG↗

arXiv2020

COT-GAN: Generating Sequential Data via Causal Optimal Transport

We introduce COT-GAN, an adversarial algorithm to train implicit generative models optimized for producing sequential data. The loss function of this algorithm is formulated using ideas from Causal Optimal Transport (COT), which combines classic optimal transport methods with an additional temporal causality constraint. Remarkably, we find that this causality condition provides a natural framework to parameterize the cost function that is learned by the discriminator as a robust (worst-case) distance, and an ideal mechanism for learning time dependent data distributions. Following Genevay et al.\ (2018), we also include an entropic penalization term which allows for the use of the Sinkhorn algorithm when computing the optimal transport cost. Our experiments show effectiveness and stability of COT-GAN when generating both low- and high-dimensional time series data. The success of the algorithm also relies on a new, improved version of the Sinkhorn divergence which demonstrates less bias in learning.

stat.ML↗

arXiv2017

Geometric singularities and a flow tangent to the Ricci flow

We consider a geometric flow introduced by Gigli and Mantegazza which, in the case of smooth compact manifolds with smooth metrics, is tangen- tial to the Ricci flow almost-everywhere along geodesics. To study spaces with geometric singularities, we consider this flow in the context of smooth manifolds with rough metrics with sufficiently regular heat kernels. On an appropriate non- singular open region, we provide a family of metric tensors evolving in time and provide a regularity theory for this flow in terms of the regularity of the heat kernel. When the rough metric induces a metric measure space satisfying a Riemannian Curvature Dimension condition, we demonstrate that the distance induced by the flow is identical to the evolving distance metric defined by Gigli and Mantegazza on appropriate admissible points. Consequently, we demonstrate that a smooth compact manifold with a finite number of geometric conical singularities remains a smooth manifold with a smooth metric away from the cone points for all future times. Moreover, we show that the distance induced by the evolving metric tensor agrees with the flow of RCD(K, N) spaces defined by Gigli-Mantegazza.

math.DG↗

arXiv2016

Three-Dimensional Alexandrov spaces with positive or nonnegative Ricci curvature

We study closed three-dimensional Alexandrov spaces with a lower Ricci curvature bound in the $\mathsf{CD}^*(K,N)$ sense, focusing our attention on those with positive or nonnegative Ricci curvature. First, we show that a closed three-dimensional $\mathsf{CD}^*(2,3)$-Alexandrov space must be homeomorphic to a spherical space form or to the suspension of $\mathbb{R}P^2$. We then classify closed three-dimensional $\mathsf{CD}^*(0,3)$-Alexandrov spaces.

math.DG↗

arXiv2015

Intrinsic flat convergence with bounded Ricci curvature

In this paper we address the relationship between Gromov-Hausdorff limits and intrinsic flat limits of complete Riemannian manifolds. In \cite{SormaniWenger2010, SormaniWenger2011}, Sormani-Wenger show that for a sequence of Riemannian manifolds with nonnegative Ricci curvature, a uniform upper bound on diameter, and non-collapsed volume, the intrinsic flat limit exists and agrees with the Gromov-Hausdorff limit. This can be viewed as a non-cancellation theorem showing that for such sequences, points don't cancel each other out in the limit. Here we prove a similar no-cancellation theorem, replacing the assumption of nonnegative Ricci curvature with a two-sided bound on Ricci curvature. This version corrects a mistake in the previous version of this paper (where we assume only an arbitrary lower Ricci bound) which was due to a crucial error in one of our supporting theorems for that argument.

math.MG↗

arXiv2014

Alexandrov spaces with large volume growth

Let $(X,d)$ be an $n$-dimensional Alexandrov space whose Hausdorff measure $\mathcal{H}^n$ satisfies a condition giving the metric measure space $(X,d,\mathcal{H}^n)$ a notion of having nonnegative Ricci curvature. We examine the influence of large volume growth on these spaces and generalize some classical arguments from Riemannian geometry showing that when the volume growth is sufficiently large, then $(X,d,\mathcal{H}^n)$ has finite topological type.

math.MG↗