Searcharxiv⌕ Search

arXiv subjects

Hayden Schaeffer

Publications and source records attributed to Hayden Schaeffer.

At least 19 recordsLinked to original sources
arXiv2026

Neural Network Approximation and Structural Complexity of Semilinear PDEs with Lower Bounds

Neural networks are commonly used to learn the solution operators of PDEs such as the Navier-Stokes equations and viscous conservation laws. On standard benchmarks, moderately sized and often shallow networks achieve small errors that decay at algebraic rates with the network size. These observations are not well described by the current approximation theory, which treats the solution operator as a generic map and derives worst-case bounds over large classes of operators. We show that the structure of the equations and their properties explain the observed behavior. For semilinear parabolic equations, the approximation costs separate into two terms, where one is controlled by an invariant of the governing law and the other by the data. The constitutive law can be parameterized as a single block whose size for accuracy $η$ is $O(η^{-pχ}\logη^{-1})$ for every $p>p_f$, where the structural exponent $p_f$ only depends on the computation graph of $f$. This exponent does not depend on the dimension and is in fact zero for polynomial laws, so for the two-dimensional Navier-Stokes equations the nonlinear block costs only $O(\logη^{-1})$ parameters. The data enters through encoding layers with a bounded number of learned weights per mode, so neither term grows with the grid resolution or the time step. While the analysis only uses the max-norm error of the pointwise block, we show that one obtains approximation bounds in a stronger norm along the whole trajectory. For viscous Hamilton-Jacobi equations with a $C^s$ Hamiltonian, the approximation requires on the order of $η^{-1/s}$ parameters, and we prove a matching lower bound, up to logarithms. This shows that solution operators for this class of problems have low structural complexity, even though they are nonlinear maps between infinite-dimensional spaces.

math.NA↗
arXiv2026

MintFlow: Minimal Trajectory Intervention for Constrained Flow Matching

Flow matching models excel at generative modeling, and many downstream applications require their samples to satisfy prescribed constraints, such as observed measurements and physical laws. However, existing constrained samplers often face a trade-off: \textit{enforcing constraints can substantially displace samples from the pretrained data distribution}. To address this trade-off, we introduce \textbf{MintFlow}, a training-free constrained sampling framework that formulates constraint enforcement as a minimal intervention on the pretrained flow trajectory. MintFlow seeks the minimal perturbation of an intermediate flow state such that its subsequent evolution under the pretrained flow field satisfies the target constraint. By minimally perturbing the flow state while keeping the pretrained flow field unchanged, MintFlow enforces the constraint while minimizing unnecessary deviation from the pretrained distribution. An adjoint formulation yields a closed-form expression for this perturbation, eliminating expensive iterative optimization. Furthermore, MintFlow adaptively selects the intervention time to balance the required perturbation magnitude with its amplification by the remaining flow. Across a range of tasks in generative vision and physical system modeling, MintFlow achieves competitive constraint satisfaction while preserving the pretrained generative distribution substantially better than state-of-the-art constrained methods.

cs.AI↗
arXiv2026

Loss Curvature as a Sampling-Free Proxy for Memorization in Diffusion Models

Understanding the mechanism behind why diffusion models memorize their training data remains an open question in generative modeling. In this paper, we examine this question from two perspectives: (a) the model-level perspective concerning the onset of memorization in a model during training, and (b) the sample-level perspective concerning the proclivity of an individual sample to be memorized by a given model. We show that the so-called input loss curvature, or the Frobenius norm of the Hessian of the diffusion model's training loss, is an effective metric for quantifying both perspectives on memorization. In particular, we empirically demonstrate that input curvatures averaged across training and validation samples can identify when a model is likely to start regurgitating its training data when tracked over training time. However, this metric is not sufficient for measuring an individual sample's likelihood of being memorized. To address this perspective, we introduce a counterfactual definition of memorization in diffusion models, extending the analogous notion from supervised learning proposed by [Feldman, 2019]. We prove that this counterfactual memorization is controlled by the input curvature of the leave-one-out model, thus establishing a direct theoretical link between a sample's local loss landscape and its generation likelihood. We validate our derived bounds through several empirical experiments and show that input-dependent curvature provides a proxy for memorization.

cs.LG↗
arXiv2026

MVNN: A Measure-Valued Neural Network for Learning McKean-Vlasov Dynamics from Particle Data

Collective behaviors that emerge from interactions are fundamental to numerous biological systems. To learn such interacting forces from observations, we introduce a measure-valued neural network that infers measure-dependent interaction (drift) terms directly from particle-trajectory observations. The proposed architecture generalizes standard neural networks to operate on probability measures by learning cylindrical features, using an embedding network that produces scalable distribution-to-vector representations. On the theory side, we establish well-posedness of the resulting dynamics and prove propagation-of-chaos for the associated interacting-particle system. We further show universal approximation and quantitative approximation rates under a low-dimensional measure-dependence assumption. Numerical experiments on first and second order systems, including deterministic and stochastic Motsch-Tadmor dynamics, two-dimensional attraction-repulsion aggregation, Cucker-Smale dynamics, and a hierarchical multi-group system, demonstrate accurate prediction and strong out-of-distribution generalization.

math.NA↗
arXiv2026

Wasserstein Moment Nudging for Vlasov-Poisson Data Assimilation

We introduce a continuous data assimilation method for particle-in-cell simulations of the Vlasov-Poisson equation when only hydrodynamic moments are observed. The forecast state is an empirical measure on phase space, whereas the observed fields (density, bulk velocity, and temperature) constrain only a few velocity moments and leave the velocity-space shape of the distribution undetermined. We construct the moment feedback as a Wasserstein gradient flow of a moment-mismatch functional over phase-space measures. The resulting drift acts directly on particle positions and velocities, couples the density, momentum, and energy residuals through a single variational structure, and vanishes on the entire moment-compatible set. Under the standard Wasserstein metric, the energy residual produces a position correction that grows quadratically with the particle speed, and the particle system falls outside standard well-posedness theory. Our primary formulation pairs the quadratic moment mismatch with a velocity-weighted Wasserstein metric that penalizes spatial transport at large peculiar velocity relative to the observed bulk flow, which removes this growth. A direction-split variant retains the plain metric instead. Under the same weighted metric, an alternative moment-relative-entropy functional yields an affine, shape-preserving velocity correction and explicit global moment balances for the space-inhomogeneous system. We prove that the finite-particle scheme with linear Lenard-Bernstein collisions is globally well posed. In 1D1V and 2D2V experiments with several collision models, the nudged formulations reduce bulk-velocity and temperature errors by up to two orders of magnitude relative to an unassimilated run.

math.NA↗
arXiv2026

Kernel Methods for Learning Operators with Multiple Inputs and Outputs

Learning mappings between infinite-dimensional objects is a central challenge in scientific machine learning. We introduce a general kernel-based encoder-decoder framework for operator learning that separates observation, representation, learning, and reconstruction. We develop this framework for multi-input, multi-output operator learning, where operators map between products of potentially distinct function spaces. Our approximation theory shows that, although the number of inputs and outputs can increase, the convergence rate is governed by the most challenging constituent approximation problem rather than the overall problem dimension. The framework leads to practical kernel methods with closed-form training and inference, combining mathematical tractability with computational efficiency. We further specialize the approach to multiple operator learning by introducing KernelMO, a family of kernel methods with complementary operator-valued and product-space formulations. Across five families of parametric partial differential equations, the proposed methods achieve competitive or state-of-the-art predictive accuracy while reducing training and inference costs relative to neural operator architectures and deep learning based models, offering an efficient and lightweight alternative.

cs.LG↗
arXiv2026

Multiscale Nudging: From Macroscopic Observations to Microscopic Dynamics

We introduce a measure-based nudging framework for assimilating macroscopic observations into microscopic mean-field particle dynamics. The central difficulty is a representation mismatch: the forecast is a labeled particle system, while the observations specify only a smoothed, permutation-invariant density. To address this mismatch, we define the forecast-observation discrepancy as a quadratic functional on probability measures after applying the same smoothing operator used by the observation process. The Wasserstein gradient of this functional induces a transport velocity on state space, which yields a particle-level correction without constructing particle-to-particle matching, linearizing the dynamics, or estimating ensemble covariances. For a fixed observation scale, we prove well-posedness of the assimilated McKean-Vlasov dynamics and propagation of chaos for the interacting particle approximation. Under exact smoothed observations and an observability condition at the kernel scale, we establish an $L^2$-stability estimate showing exponential decay up to a bias floor controlled by model misspecification. Numerical experiments on linear, bimodal, chaotic, kinetic, and collective-motion systems demonstrate that the method can recover macroscopic structure from incomplete density-level observations.

math.NA↗
arXiv2026

Neural Scaling Laws of Deep ReLU and Deep Operator Network: A Theoretical Study

Neural scaling laws play a pivotal role in the performance of deep neural networks and have been observed in a wide range of tasks. However, a complete theoretical framework for understanding these scaling laws remains underdeveloped. In this paper, we explore the neural scaling laws for deep operator networks, which involve learning mappings between function spaces, with a focus on the Chen and Chen style architecture. These approaches, which include the popular Deep Operator Network (DeepONet), approximate the output functions using a linear combination of learnable basis functions and coefficients that depend on the input functions. We establish a theoretical framework to quantify the neural scaling laws by analyzing its approximation and generalization errors. We articulate the relationship between the approximation and generalization errors of deep operator networks and key factors such as network model size and training data size. Moreover, we address cases where input functions exhibit low-dimensional structures, allowing us to derive tighter error bounds. These results also hold for deep ReLU networks and other similar structures. Our results offer a partial explanation of the neural scaling laws in operator learning and provide a theoretical foundation for their applications.

cs.LG↗
arXiv2026

Multiple Neural Operators Achieve Near-Optimal Rates for Multi-Task Learning

We study the approximation and statistical complexity of learning collections of operators in a shared multi-task setting, with a focus on the Multiple Neural Operators (MNO) architecture. For broad classes of Lipschitz multiple operator maps, we derive near-optimal upper bounds for approximation and statistical generalization. On the lower-bound side, we establish a curse of parametric complexity and prove corresponding minimax rates. Together, these results show that shared representations across tasks do not increase the overall cost: multi-task operator learning follows the same scaling laws as single operator learning. We also compare MNO with a multi-task extension of DeepONet based on concatenated task inputs and show that, from a worst-case approximation-complexity perspective, both architectures satisfy essentially the same asymptotic rates.

cs.LG↗
arXiv2026

Generalization Bounds and Statistical Guarantees for Multi-Task and Multiple Operator Learning with MNO Networks

Multiple operator learning concerns learning operator families $\{G[α]:U\to V\}_{α\in W}$ indexed by an operator descriptor $α$. Training data are collected hierarchically by sampling operator instances $α$, then input functions $u$ per instance, and finally evaluation points $x$ per input, yielding noisy observations of $G[α][u](x)$. While recent work has developed expressive multi-task and multiple operator learning architectures and approximation-theoretic scaling laws, quantitative statistical generalization guarantees remain limited. We provide a covering-number-based generalization analysis for separable models, focusing on the Multiple Neural Operator (MNO) architecture: we first derive explicit metric-entropy bounds for hypothesis classes given by linear combinations of products of deep ReLU subnetworks, and then combine these complexity bounds with approximation guarantees for MNO to obtain an explicit approximation-estimation tradeoff for the expected test error on new (unseen) triples $(α,u,x)$. The resulting bound makes the dependence on the hierarchical sampling budgets $(n_α,n_u,n_x)$ transparent and yields an explicit learning-rate statement in the operator-sampling budget $n_α$, providing a sample-complexity characterization for generalization across operator instances. The structure and architecture can also be viewed as a general purpose solver or an example of a "small'' PDE foundation model, where the triples are one form of multi-modality.

cs.LG↗
arXiv2026

Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentum

Efficient stochastic optimization typically integrates an update direction that performs well in the deterministic regime with a mechanism adapting to stochastic perturbations. While Adam uses adaptive moment estimates to promote stability, Muon utilizes the weight layers' matrix structure via orthogonalized momentum, showing superior performance in large language model training. We propose a new optimizer and a diagonal extension, NAMO and NAMO-D, providing the first principled integration of orthogonalized momentum with norm-based Adam-type noise adaptation. NAMO scales orthogonalized momentum using a single adaptive stepsize, preserving orthogonality while improving upon Muon at negligible additional cost. NAMO-D instead right-multiplies orthogonalized momentum by a diagonal matrix with clamped entries. This design enables neuron-wise noise adaptation and aligns with the common near block-diagonal Hessian structure. Under standard assumptions, we establish optimal convergence rates for both algorithms in the deterministic setting and show that, in the stochastic setting, their convergence guarantees adapt to the noise level of stochastic gradients. Experiments on pretraining GPT-2 models demonstrate improved performance of both NAMO and NAMO-D compared to the AdamW and Muon baselines, with NAMO-D achieving further gains over NAMO via an additional clamping hyperparameter that balances the competing goals of maintaining a well-conditioned update direction and leveraging fine-grained noise adaptation.

cs.LG↗
arXiv2026

VICON: Vision In-Context Operator Networks for Multi-Physics Fluid Dynamics Prediction

In-Context Operator Networks (ICONs) have demonstrated the ability to learn operators across diverse partial differential equations using few-shot, in-context learning. However, existing ICONs process each spatial point as an individual token, severely limiting computational efficiency when handling dense data in higher spatial dimensions. We propose Vision In-Context Operator Networks (VICON), which integrates vision transformer architectures to efficiently process 2D data through patch-wise operations while preserving ICON's adaptability to multiphysics systems and varying timesteps. Evaluated across three fluid dynamics benchmarks, VICON significantly outperforms state-of-the-art baselines: DPOT and MPP, reducing the averaged last-step rollout error by 37.9% compared to DPOT and 44.7% compared to MPP, while requiring only 72.5% and 34.8% of their respective inference times. VICON naturally supports flexible rollout strategies with varying timestep strides, enabling immediate deployment in imperfect measurement systems where sampling frequencies may differ or frames might be dropped - common challenges in real-world settings - without requiring retraining or interpolation. In these realistic scenarios, VICON exhibits remarkable robustness, experiencing only 24.41% relative performance degradation compared to 71.37%-74.49% degradation in baseline methods, demonstrating its versatility for deploying in realistic applications. Our scripts for processing datasets and code are publicly available at https://github.com/Eydcao/VICON.

cs.LG↗
arXiv2025

PI-MFM: Physics-informed multimodal foundation model for solving partial differential equations

Partial differential equations (PDEs) govern a wide range of physical systems, and recent multimodal foundation models have shown promise for learning PDE solution operators across diverse equation families. However, existing multi-operator learning approaches are data-hungry and neglect physics during training. Here, we propose a physics-informed multimodal foundation model (PI-MFM) framework that directly enforces governing equations during pretraining and adaptation. PI-MFM takes symbolic representations of PDEs as the input, and automatically assembles PDE residual losses from the input expression via a vectorized derivative computation. These designs enable any PDE-encoding multimodal foundation model to be trained or adapted with unified physics-informed objectives across equation families. On a benchmark of 13 parametric one-dimensional time-dependent PDE families, PI-MFM consistently outperforms purely data-driven counterparts, especially with sparse labeled spatiotemporal points, partially observed time domains, or few labeled function pairs. Physics losses further improve robustness against noise, and simple strategies such as resampling collocation points substantially improve accuracy. We also analyze the accuracy, precision, and computational cost of automatic differentiation and finite differences for derivative computation within PI-MFM. Finally, we demonstrate zero-shot physics-informed fine-tuning to unseen PDE families: starting from a physics-informed pretrained model, adapting using only PDE residuals and initial/boundary conditions, without any labeled solution data, rapidly reduces test errors to around 1% and clearly outperforms physics-only training from scratch. These results show that PI-MFM provides a practical and scalable path toward data-efficient, transferable PDE solvers.

cs.LG↗
arXiv2025

Regularized Random Fourier Features and Finite Element Reconstruction for Operator Learning in Sobolev Space

Operator learning is a data-driven approximation of mappings between infinite-dimensional function spaces, such as the solution operators of partial differential equations. Kernel-based operator learning can offer accurate, theoretically justified approximations that require less training than standard methods. However, they can become computationally prohibitive for large training sets and can be sensitive to noise. We propose a regularized random Fourier feature (RRFF) approach, coupled with a finite element reconstruction map (RRFF-FEM), for learning operators from noisy data. The method uses random features drawn from multivariate Student's $t$ distributions, together with frequency-weighted Tikhonov regularization that suppresses high-frequency noise. We establish high-probability bounds on the extreme singular values of the associated random feature matrix and show that when the number of features $N$ scales like $m \log m$ with the number of training samples $m$, the system is well-conditioned, which yields estimation and generalization guarantees. Detailed numerical experiments on benchmark PDE problems, including advection, Burgers', Darcy flow, Helmholtz, Navier-Stokes, and structural mechanics, demonstrate that RRFF and RRFF-FEM are robust to noise and achieve improved performance with reduced training time compared to the unregularized random feature model, while maintaining competitive accuracy relative to kernel and neural operator tests.

cs.LG↗
arXiv2025

Finite Element Representation Network (FERN) for Operator Learning with a Localized Trainable Basis

We propose a finite-element local basis-based operator learning framework for solving partial differential equations (PDEs). Operator learning aims to approximate mappings from input functions to output functions, where the latter are typically represented using basis functions. While non-learnable bases reduce training costs, learnable bases offer greater flexibility but often require deep network architectures with a large number of trainable parameters. Existing approaches typically rely on deep global bases; however, many PDE solutions exhibit local behaviors such as shocks, sharp gradients, etc., and in parametrized PDE settings, these localized features may appear in different regions of the domain across different training and testing samples. Motivated by the use of local bases in finite element methods (FEM) for function approximation, we develop a shallow neural network architecture that constructs adaptive FEM bases. By adopting suitable activation functions, such as ReLU, the FEM bases can be assembled exactly within the network, introducing no additional approximation error in the basis construction process. This design enables the learning procedure to naturally mimic the adaptive refinement mechanism of FEM, allowing the network to discover basis functions tailored to intrinsic solution features such as shocks. The proposed learnable adaptive bases are then employed to represent the solution (output function) of the PDE. This framework reduces the number of trainable parameters while maintaining high approximation accuracy, effectively combining the adaptivity of FEM with the expressive power of operator learning. To evaluate performance, we validate the proposed method on seven families of PDEs with diverse characteristics, demonstrating its accuracy, efficiency, and robustness.

math.NA↗
arXiv2025

A Deep Learning Framework for Multi-Operator Learning: Architectures and Approximation Theory

While many problems in machine learning focus on learning mappings between finite-dimensional spaces, scientific applications require approximating mappings between function spaces, i.e., operators. We study the problem of learning collections of operators and provide both theoretical and empirical advances. We distinguish between two regimes: (i) multiple operator learning, where a single network represents a continuum of operators parameterized by a parametric function, and (ii) learning several distinct single operators, where each operator is learned independently. For the multiple operator case, we introduce two new architectures, $\mathrm{MNO}$ and $\mathrm{MONet}$, and establish universal approximation results in three settings: continuous, integrable, or Lipschitz operators. For the latter, we further derive explicit scaling laws that quantify how the network size must grow to achieve a target approximation accuracy. For learning several single operators, we develop a framework for balancing architectural complexity across subnetworks and show how approximation order determines computational efficiency. Empirical experiments on parametric PDE benchmarks confirm the strong expressive power and efficiency of the proposed architectures. Overall, this work establishes a unified theoretical and practical foundation for scalable neural operator learning across multiple operators.

cs.LG↗
arXiv2025

Differentially Private Random Feature Model

Designing privacy-preserving machine learning algorithms has received great attention in recent years, especially in the setting when the data contains sensitive information. Differential privacy (DP) is a widely used mechanism for data analysis with privacy guarantees. In this paper, we produce a differentially private random feature model. Random features, which were proposed to approximate large-scale kernel machines, have been used to study privacy-preserving kernel machines as well. We consider the over-parametrized regime (more features than samples) where the non-private random feature model is learned via solving the min-norm interpolation problem, and then we apply output perturbation techniques to produce a private model. We show that our method preserves privacy and derive a generalization error bound for the method. To the best of our knowledge, we are the first to consider privacy-preserving random feature models in the over-parametrized regime and provide theoretical guarantees. We empirically compare our method with other privacy-preserving learning methods in the literature as well. Our results show that our approach is superior to the other methods in terms of generalization performance on synthetic data and benchmark data sets. Additionally, it was recently observed that DP mechanisms may exhibit and exacerbate disparate impact, which means that the outcomes of DP learning algorithms vary significantly among different groups. We show that both theoretically and empirically, random features have the potential to reduce disparate impact, and hence achieve better fairness.

cs.LG↗
arXiv2025

Cauchy Random Features for Operator Learning in Sobolev Space

Operator learning is the approximation of operators between infinite dimensional Banach spaces using machine learning approaches. While most progress in this area has been driven by variants of deep neural networks such as the Deep Operator Network and Fourier Neural Operator, the theoretical guarantees are often in the form of a universal approximation property. However, the existence theorems do not guarantee that an accurate operator network is obtainable in practice. Motivated by the recent kernel-based operator learning framework, we propose a random feature operator learning method with theoretical guarantees and error bounds. The random feature method can be viewed as a randomized approximation of a kernel method, which significantly reduces the computation requirements for training. We provide a generalization error analysis for our proposed random feature operator learning method along with comprehensive numerical results. Compared to kernel-based method and neural network methods, the proposed method can obtain similar or better test errors across benchmarks examples with significantly reduced training times. An additional advantages it that our implementation is simple and does require costly computational resources, such as GPU.

cs.LG↗