arXiv subjects

Nick Alger

Publications and source records attributed to Nick Alger.

8 recordsLinked to original sources

arXiv2026

Tucker Tensor Train Taylor Series

Learning derivative-accurate surrogates for implicit simulators is a key challenge in scientific machine learning. High-order Taylor surrogates have long been considered intractable in high dimensions, because the derivative tensors are enormous and accessible only through probes. We make such surrogates tractable with the Tucker tensor train Taylor series (T4S), a local surrogate that represents each derivative tensor of a truncated Taylor expansion as a Tucker tensor train. T4S targets a different learning problem than global operator learning: rather than training from input-output pairs at many parameter values, it is trained from random directionally symmetric derivative probes at a single expansion point. Computing $m$ probes of the $k$th derivative requires only $O(mk)$ linearized solves sharing one operator, cheaper than the $O(m)$ nonlinear solves for function evaluations or $O(m\,2^k)$ linearized solves for asymmetric probes. We develop derivative-informed dimension reduction, Riemannian Gauss-Newton and Cauchy SGD fitting algorithms with rank continuation, requiring little hyperparameter tuning, and fast sweeping routines for the Riemannian Jacobian. We prove representational guarantees under spectral decay of the input covariance. Experiments show that our methods match quasi-optimal T3-SVD accuracy on random tensors from probes alone, up to data-limited ranks, and recover high-order Taylor structure in Poisson PDE examples.

math.NA↗

arXiv2025

Accelerating seismic inversion and uncertainty quantification with efficient high-rank Hessian approximations

Efficient high-rank approximations of the Hessian can accelerate seismic full waveform inversion (FWI) and uncertainty quantification (UQ). In FWI, approximations of the inverse of the Hessian may be used as preconditioners for Newton-type or quasi-Newton algorithms, reducing computational costs and improving recovery in deeper subsurface regions. In Bayesian UQ, Hessian approximations enable the construction of Markov chain Monte Carlo (MCMC) proposals that capture the directional scalings of the posterior, enhancing the efficiency of MCMC. Computing the exact Hessian is intractable for large-scale problems because the Hessian is accessible only through matrix-vector products, and performing each matrix-vector product requires costly solution of wave equations. Moreover, the Hessian is high-rank, which means that low-rank methods, often employed in large-scale inverse problems, are inefficient. We adapt two existing high-rank Hessian approximations -- the point spread function method and the pseudo-differential operator probing method. Building on an observed duality between these approaches, we develop a novel method that unifies their complementary strengths. We validate these methods on a synthetic quadratic model and on the Marmousi model. Numerical experiments show that these high-rank Hessian approximations substantially reduce the computational costs in FWI. In UQ, MCMC samples computed using no Hessian approximation or a low-rank approximation explore the posterior slowly, providing little meaningful statistical information after tens of thousands of iterations and underestimating the variance. At the same time, the effective sample size is overestimated, providing false confidence. In contrast, MCMC samples generated using the high-rank Hessian approximations provide meaningful statistical information about the posterior and more accurately assess the posterior variance.

math.NA↗

arXiv2023

Point spread function approximation of high rank Hessians with locally supported non-negative integral kernels

We present an efficient matrix-free point spread function (PSF) method for approximating operators that have locally supported non-negative integral kernels. The method computes impulse responses at scattered points, and interpolates these impulse responses to approximate integral kernel entries. Impulse responses are computed by applying the operator to Dirac comb batches of point sources, which are chosen via an ellipsoid packing procedure. Evaluation of kernel entries allows us to construct a hierarchical matrix approximation of the operator, which is used for further matrix computations. We illustrate the end-to-end method on a blur problem, then use the method to build preconditioners for the Hessian in two inverse problems governed by partial differential equations (PDEs): inversion for the basal friction coefficient in an ice sheet flow problem and for the initial condition in an advective-diffusive transport problem. While for many ill-posed inverse problems the Hessian of the data misfit term exhibits a low rank structure, and hence a low rank approximation is suitable, for many problems of practical interest the numerical rank of the Hessian is still large. But Hessian impulse responses typically become more local as the numerical rank increases, which benefits the PSF method. Numerical results reveal that the PSF preconditioner clusters the spectrum of the preconditioned Hessian near one, yielding roughly 5x-10x reductions in the required number of PDE solves, as compared to regularization preconditioning and no preconditioning. We also present a numerical study for the influence of various parameters (that control the shape of the impulse responses) on the effectiveness of the advection-diffusion Hessian approximation. The results show that the PSF-based preconditioners are able to form good approximations of high-rank Hessians using a small number of operator applications.

math.NA↗

arXiv2020

Tensor train construction from tensor actions, with application to compression of large high order derivative tensors

We present a method for converting tensors into tensor train format based on actions of the tensor as a vector-valued multilinear function. Existing methods for constructing tensor trains require access to "array entries" of the tensor and are therefore inefficient or computationally prohibitive if the tensor is accessible only through its action, especially for high order tensors. Our method permits efficient tensor train compression of large high order derivative tensors for nonlinear mappings that are implicitly defined through the solution of a system of equations. Array entries of these derivative tensors are not directly accessible, but actions of these tensors can be computed efficiently via a procedure that we discuss. Such tensors are often amenable to tensor train compression in theory, but until now no efficient algorithm existed to convert them into tensor train format. We demonstrate our method by compressing a Hilbert tensor of size $41 \times 42 \times 43 \times 44 \times 45$, and by forming high order (up to $5^\text{th}$ order derivatives/$6^\text{th}$ order tensors) Taylor series surrogates of the noise-whitened parameter-to-output map for a stochastic partial differential equation with boundary output.

math.NA↗

arXiv2020

Low Rank Saddle Free Newton: A Scalable Method for Stochastic Nonconvex Optimization

In modern deep learning, highly subsampled stochastic approximation (SA) methods are preferred to sample average approximation (SAA) methods because of large data sets as well as generalization properties. Additionally, due to perceived costs of forming and factorizing Hessians, second order methods are not used for these problems. In this work we motivate the extension of Newton methods to the SA regime, and argue for the use of the scalable low rank saddle free Newton (LRSFN) method, which avoids forming the Hessian in favor of making a low rank approximation. Additionally, LRSFN can facilitate fast escape from indefinite regions leading to better optimization solutions. In the SA setting, iterative updates are dominated by stochastic noise, and stability of the method is key. We introduce a continuous time stability analysis framework, and use it to demonstrate that stochastic errors for Newton methods can be greatly amplified by ill-conditioned Hessians. The LRSFN method mitigates this stability issue via Levenberg-Marquardt damping. However, generally the analysis shows that second order methods with stochastic Hessian and gradient information may need to take small steps, unlike in deterministic problems. Numerical results show that LRSFN can escape indefinite regions that other methods have issues with; and even under restrictive step length conditions, LRSFN can outperform popular first order methods on large scale deep learning tasks in terms of generalizability for equivalent computational work.

math.OC↗

arXiv2019

Inexact Newton Methods for Stochastic Nonconvex Optimization with Applications to Neural Network Training

We study stochastic inexact Newton methods and consider their application in nonconvex settings. Building on the work of [R. Bollapragada, R. H. Byrd, and J. Nocedal, IMA Journal of Numerical Analysis, 39 (2018), pp. 545--578] we derive bounds for convergence rates in expected value for stochastic low rank Newton methods, and stochastic inexact Newton Krylov methods. These bounds quantify the errors incurred in subsampling the Hessian and gradient, as well as in approximating the Newton linear solve, and in choosing regularization and step length parameters. We deploy these methods in training convolutional autoencoders for the MNIST and CIFAR10 data sets. Numerical results demonstrate that, relative to first order methods, these stochastic inexact Newton methods often converge faster, are more cost-effective, and generalize better.

math.OC↗

arXiv2018

Scalable matrix-free adaptive product-convolution approximation for locally translation-invariant operators

We present an adaptive grid matrix-free operator approximation scheme based on a "product-convolution" interpolation of convolution operators. This scheme is appropriate for operators that are locally translation-invariant, even if these operators are high-rank or full-rank. Such operators arise in Schur complement methods for solving partial differential equations (PDEs), as Hessians in PDE-constrained optimization and inverse problems, as integral operators, as covariance operators, and as Dirichlet-to-Neumann maps. Constructing the approximation requires computing the impulse responses of the operator to point sources centered on nodes in an adaptively refined grid of sample points. A randomized a-posteriori error estimator drives the adaptivity. Once constructed, the approximation can be efficiently applied to vectors using the fast Fourier transform. The approximation can be efficiently converted to hierarchical matrix ($H$-matrix) format, then inverted or factorized using scalable $H$-matrix arithmetic. The quality of the approximation degrades gracefully as fewer sample points are used, allowing cheap lower quality approximations to be used as preconditioners. This yields an automated method to construct preconditioners for locally translation-invariant Schur complements. We directly address issues related to boundaries and prove that our scheme eliminates boundary artifacts. We test the scheme on a spatially varying blurring kernel, on the non-local component of an interface Schur complement for the Poisson operator, and on the data misfit Hessian for an advection dominated advection-diffusion inverse problem. Numerical results show that the scheme outperforms existing methods.

math.NA↗

arXiv2016

A data scalable augmented Lagrangian KKT preconditioner for large scale inverse problems

Current state of the art preconditioners for the reduced Hessian and the Karush-Kuhn-Tucker (KKT) operator for large scale inverse problems are typically based on approximating the reduced Hessian with the regularization operator. However, the quality of this approximation degrades with increasingly informative observations or data. Thus the best case scenario from a scientific standpoint (fully informative data) is the worse case scenario from a computational perspective. In this paper we present an augmented Lagrangian-type preconditioner based on a block diagonal approximation of the augmented upper left block of the KKT operator. The preconditioner requires solvers for two linear subproblems that arise in the augmented KKT operator, which we expect to be much easier to precondition than the reduced Hessian. Analysis of the spectrum of the preconditioned KKT operator indicates that the preconditioner is effective when the regularization is chosen appropriately. In particular, it is effective when the regularization does not over-penalize highly informed parameter modes and does not under-penalize uninformed modes. Finally, we present a numerical study for a large data/low noise Poisson source inversion problem, demonstrating the effectiveness of the preconditioner. In this example, three MINRES iterations on the KKT system with our preconditioner results in a reconstruction with better accuracy than 50 iterations of CG on the reduced Hessian system with regularization preconditioning.

math.NA↗