arXiv subjects

Alexander Hsu

Publications and source records attributed to Alexander Hsu.

9 recordsLinked to original sources

arXiv2026

Training-Free Universal Approximation by Prompting Random Transformers

How expressive is prompting a transformer? Answering this question is important for separating the roles of prompting, architecture, and pretraining in transformer models, and for determining whether task-specific behavior must be stored in model weights or can instead be induced at inference time through the prompt. We show, in an approximation-theoretic sense, that pretraining is optional: a single-layer softmax attention network with random, untrained weights can approximate any H\"older function on a compact manifold when steered by an appropriate soft prompt. Guided by the connection between softmax attention and kernel methods, we construct explicit soft prompts (a prompt per target function, independent of the query) as solutions to linear systems matching attention logits to Gaussian kernel exponents, under which the frozen transformer emulates the classical Nadaraya-Watson kernel estimator. The construction requires only a mild rank condition on the weights, which we show holds almost surely under Gaussian initialization. The prompted network inherits the theoretical guarantees of kernel regression, leading to universal approximation theorems with minimax-optimal rates that depend on the intrinsic dimension. We further quantify the cost of prompting, exposing a tradeoff between the norm of the constructed soft prompt tokens, prompt length, and hidden dimension. Numerical experiments corroborate the constructions and predicted rates.

cs.LG↗

arXiv2026

Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizer

Pre-trained transformers are able to learn from examples provided as part of the prompt without any weight updates, a remarkable ability known as in-context learning (ICL). Despite its demonstrated efficacy across various domains, the theoretical understanding of ICL is still developing. Whereas most existing theory has focused on linear models, we study ICL in the nonlinear regression setting. Through the interaction mechanism in attention, we explicitly construct transformer networks to realize nonlinear features, such as polynomial or spline bases, which span a wide class of functions. Based on this construction, we establish a framework to analyze end-to-end in-context nonlinear regression with the constructed features. Our theory provides finite-sample generalization error bounds in terms of context length and training set size. We numerically validate the theory on synthetic regression tasks.

cs.LG↗

arXiv2025

Operator Learning at Machine Precision

Neural operator learning methods have garnered significant attention in scientific computing for their ability to approximate infinite-dimensional operators. However, increasing their complexity often fails to substantially improve their accuracy, leaving them on par with much simpler approaches such as kernel methods and more traditional reduced-order models. In this article, we set out to address this shortcoming and introduce CHONKNORIS (Cholesky Newton--Kantorovich Neural Operator Residual Iterative System), an operator learning paradigm that can achieve machine precision. CHONKNORIS draws on numerical analysis: many nonlinear forward and inverse PDE problems are solvable by Newton-type methods. Rather than regressing the solution operator itself, our method regresses the Cholesky factors of the elliptic operator associated with Tikhonov-regularized Newton--Kantorovich updates. The resulting unrolled iteration yields a neural architecture whose machine-precision behavior follows from achieving a contractive map, requiring far lower accuracy than end-to-end approximation of the solution operator. We benchmark CHONKNORIS on a range of nonlinear forward and inverse problems, including a nonlinear elliptic equation, Burgers' equation, a nonlinear Darcy flow problem, the Calder\'{o}n problem, an inverse wave scattering problem, and a problem from seismic imaging. We also present theoretical guarantees for the convergence of CHONKNORIS in terms of the accuracy of the emulated Cholesky factors. Additionally, we introduce a foundation model variant, FONKNORIS (Foundation Newton--Kantorovich Neural Operator Residual Iterative System), which aggregates multiple pre-trained CHONKNORIS experts for diverse PDEs to emulate the solution map of a novel nonlinear PDE. Our FONKNORIS model is able to accurately solve unseen nonlinear PDEs such as the Klein--Gordon and Sine--Gordon equations.

cs.LG↗

arXiv2025

Understanding In-Context Learning on Structured Manifolds: Bridging Attention to Kernel Methods

While in-context learning (ICL) has achieved remarkable success in natural language and vision domains, its theoretical understanding-particularly in the context of structured geometric data-remains unexplored. This paper initiates a theoretical study of ICL for regression of H\"older functions on manifolds. We establish a novel connection between the attention mechanism and classical kernel methods, demonstrating that transformers effectively perform kernel-based prediction at a new query through its interaction with the prompt. This connection is validated by numerical experiments, revealing that the learned query-prompt scores for H\"older functions are highly correlated with the Gaussian kernel. Building on this insight, we derive generalization error bounds in terms of the prompt length and the number of training tasks. When a sufficient number of training tasks are observed, transformers give rise to the minimax regression rate of H\"older functions on manifolds, which scales exponentially with respect to the prompt length with the exponent depending on the intrinsic dimension of the manifold, rather than the ambient space dimension. Our result also characterizes how the generalization error scales with the number of training tasks, shedding light on the complexity of transformers as in-context kernel algorithm learners. Our findings provide foundational insights into the role of geometry in ICL and novels tools to study ICL of nonlinear models.

cs.LG↗

arXiv2025

Data-Efficient Kernel Methods for Learning Differential Equations and Their Solution Operators: Algorithms and Error Analysis

We introduce a novel kernel-based framework for learning differential equations and their solution maps that is efficient in data requirements, in terms of solution examples and amount of measurements from each example, and computational cost, in terms of training procedures. Our approach is mathematically interpretable and backed by rigorous theoretical guarantees in the form of quantitative worst-case error bounds for the learned equation. Numerical benchmarks demonstrate significant improvements in computational complexity and robustness while achieving one to two orders of magnitude improvements in terms of accuracy compared to state-of-the-art algorithms.

stat.ML↗

arXiv2024

Raking mortality rates across cause, population group and geography with uncertainty quantification

The Global Burden of Diseases, Injuries, and Risk Factors Study (GBD) is the single largest and most detailed scientific effort ever conducted to quantify levels and trends in health. This global health model to estimate mortality rates and other health metrics is run at different scales, leading to large data sets of results for a global region and its different sub-regions, or for a cause of death and different sub-causes for example. These models do not necessarily lead to consistent data tables where, for instance, the sum of the number of deaths for each of the sub-regions is equal to the number of deaths for the global region. Raking is widely used in survey inference and global health models to adjust the observations in contingency tables to given marginals, in the latter case reconciling estimates between models with different granularities. The results of global health models usually associate to the point estimates an uncertainty, such as standard deviations or confidence intervals. In this paper, we propose an uncertainty propagation approach that obtains, at the cost of a single solve, nearly the same uncertainty estimates as computationally intensive Monte Carlo techniques that pass thousands of observed and marginal samples through the entire raking process. We introduce a convex optimization approach that provides a unified framework to raking extensions such as uncertainty propagation, raking with differential weights, raking with different loss functions in order to ensure that bounds on estimates are respected, verifying the feasibility of the constraints, raking to margins either as hard constraints or as aggregate observations, and handling missing data.

stat.ME↗

arXiv2020

Time cells might be optimized for predictive capacity, not redundancy reduction or memory capacity

Recently, researchers have found time cells in the hippocampus that appear to contain information about the timing of past events. Some researchers have argued that time cells are taking a Laplace transform of their input in order to reconstruct the past stimulus. We argue that stimulus prediction, not stimulus reconstruction or redundancy reduction, is in better agreement with observed responses of time cells. In the process, we introduce new analyses of nonlinear, continuous-time reservoirs that model these time cells.

q-bio.NC↗

arXiv2020

The layer number of grids

The peeling process is defined as follows: starting with a finite point set $X \subset \mathbb{R}^d$, we repeatedly remove the set of vertices of the convex hull of the current set of points. The number of peeling steps needed to completely delete the set $X$ is called the layer number of $X$. In this paper, we study the layer number of the $d$-dimensional integer grid $[n]^d$. We prove that for every $d \geq 1$, the layer number of $[n]^d$ is at least $\Omega\left(n^\frac{2d}{d+1}\right)$. On the other hand, we show that for every $d\geq 3$, it takes at most $O(n^{d - 9/11})$ steps to fully remove $[n]^d$. Our approach is based on an enhancement of the method used by Har-Peled and Lidick\'{y} for solving the 2-dimensional case.

math.MG↗

arXiv2019

Covering point-sets with parallel hyperplanes and sparse signal recovery

We give a new deterministic construction of integer sensing matrices that can be used for the recovery of integer-valued signals in compressed sensing. This is a family of $n \times d$ integer matrices, $d \geq n$, with bounded sup-norm and the property that no $\ell$ column vectors are linearly dependent, $\ell \leq n$. Further, if $\ell \leq o(\log n)$ then $d/n \to \infty$ as $n \to \infty$. Our construction comes from particular sets of difference vectors of point-sets in $\mathbb R^n$ that cannot be covered by few parallel hyperplanes. We construct examples of such sets on the $0, \pm 1$ grid and use them for the matrix construction. We also show a connection of our constructions to a simple version of the Tarski plank problem.

math.CO↗