arXiv subjects

Yunan Yang

Publications and source records attributed to Yunan Yang.

At least 55 records · Page 3Linked to original sources

arXiv2023

Norm-dependent convergence and stability of the inverse scattering series for diffuse and scalar waves

This work analyzes the forward and inverse scattering series for scalar waves based on the Helmholtz equation and the diffuse waves from the time-independent diffusion equation, which are important PDEs in various applications. Different from previous works, which study the radius of convergence for the forward and inverse scattering series, the stability, and the approximation error of the series under the $L^p$ norms, we study these quantities under the Sobolev $H^s$ norm, which associates with a general class of $L^2$-based function spaces. The $H^s$ norm has a natural spectral bias based on its definition in the Fourier domain: the case $s<0$ biases towards the lower frequencies, while the case $s>0$ biases towards the higher frequencies. We compare the stability estimates using different $H^s$ norms for both the parameter and data domains and provide a theoretical justification for the frequency weighting techniques in practical inversion procedures. We also provide numerical inversion examples to demonstrate the differences in the inverse scattering radius of convergence under different metric spaces.

math.AP↗

arXiv2023

Efficient Natural Gradient Descent Methods for Large-Scale PDE-Based Optimization Problems

We propose efficient numerical schemes for implementing the natural gradient descent (NGD) for a broad range of metric spaces with applications to PDE-based optimization problems. Our technique represents the natural gradient direction as a solution to a standard least-squares problem. Hence, instead of calculating, storing, or inverting the information matrix directly, we apply efficient methods from numerical linear algebra. We treat both scenarios where the Jacobian, i.e., the derivative of the state variable with respect to the parameter, is either explicitly known or implicitly given through constraints. We can thus reliably compute several natural NGDs for a large-scale parameter space. In particular, we are able to compute Wasserstein NGD in thousands of dimensions, which was believed to be out of reach. Finally, our numerical results shed light on the qualitative differences between the standard gradient descent and various NGD methods based on different metric spaces in nonconvex optimization problems.

math.OC↗

arXiv2022

Tuning Frequency Bias in Neural Network Training with Nonuniform Data

Small generalization errors of over-parameterized neural networks (NNs) can be partially explained by the frequency biasing phenomenon, where gradient-based algorithms minimize the low-frequency misfit before reducing the high-frequency residuals. Using the Neural Tangent Kernel (NTK), one can provide a theoretically rigorous analysis for training where data are drawn from constant or piecewise-constant probability densities. Since most training data sets are not drawn from such distributions, we use the NTK model and a data-dependent quadrature rule to theoretically quantify the frequency biasing of NN training given fully nonuniform data. By replacing the loss function with a carefully selected Sobolev norm, we can further amplify, dampen, counterbalance, or reverse the intrinsic frequency biasing in NN training.

cs.LG↗

arXiv2022

Monte Carlo Gradient in Optimization Constrained by Radiative Transport Equation

Can Monte Carlo (MC) solvers be directly used in gradient-based methods for PDE-constrained optimization problems? In these problems, a gradient of the loss function is typically presented as a product of two PDE solutions, one for the forward equation and the other for the adjoint. When MC solvers are used, the numerical solutions are Dirac measures. As such, one immediately faces the difficulty in explaining the multiplication of two measures. This suggests that MC solvers are naturally incompatible with gradient-based optimization under PDE constraints. In this paper, we study two different strategies to overcome the difficulty. One is to adopt the Discrete-Then-Optimize technique and conduct the full optimization on the algebraic system, avoiding the Dirac measures. The second strategy stays within the Optimize-Then-Discretize framework. We propose a correlated simulation where, instead of using MC solvers separately for both forward and adjoint problems, we recycle the samples in the forward simulation in the adjoint solver. This frames the adjoint solution as a test function, and hence allows a rigorous convergence analysis. The investigation is presented through the lens of the radiative transfer equation, either in the inverse setting from optical imaging or in the optimal control framework. We detail the algorithm development, convergence analysis, and complexity cost. Numerical evidence is also presented to demonstrate the claims.

math.NA↗

arXiv2022

Adjoint DSMC for Nonlinear Spatially-Homogeneous Boltzmann Equation With a General Collision Model

We derive an adjoint method for the Direct Simulation Monte Carlo (DSMC) method for the spatially homogeneous Boltzmann equation with a general collision law. This generalizes our previous results in [Caflisch, R., Silantyev, D. and Yang, Y., 2021. Journal of Computational Physics, 439, p.110404], which was restricted to the case of Maxwell molecules, for which the collision rate is constant. The main difficulty in generalizing the previous results is that a rejection sampling step is required in the DSMC algorithm in order to handle the variable collision rate. We find a new term corresponding to the so-called score function in the adjoint equation and a new adjoint Jacobian matrix capturing the dependence of the collision parameter on the velocities. The new formula works for a much more general class of collision models.

math.NA↗

arXiv2022

Optimal Transport for Parameter Identification of Chaotic Dynamics via Invariant Measures

We study an optimal transportation approach for recovering parameters in dynamical systems with a single smoothly varying attractor. We assume that the data is not sufficient for estimating time derivatives of state variables but enough to approximate the long-time behavior of the system through an approximation of its physical measure. Thus, we fit physical measures by taking the Wasserstein distance from optimal transportation as a misfit function between two probability distributions. In particular, we analyze the regularity of the resulting loss function for general transportation costs and derive gradient formulas. Physical measures are approximated as fixed points of suitable PDE-based Perron--Frobenius operators. Test cases discussed in the paper include common low-dimensional dynamical systems.

math.DS↗

arXiv2022

A Generalized Weighted Optimization Method for Computational Learning and Inversion

The generalization capacity of various machine learning models exhibits different phenomena in the under- and over-parameterized regimes. In this paper, we focus on regression models such as feature regression and kernel regression and analyze a generalized weighted least-squares optimization method for computational learning and inversion with noisy data. The highlight of the proposed framework is that we allow weighting in both the parameter space and the data space. The weighting scheme encodes both a priori knowledge on the object to be learned and a strategy to weight the contribution of different data points in the loss function. Here, we characterize the impact of the weighting scheme on the generalization error of the learning method, where we derive explicit generalization errors for the random Fourier feature model in both the under- and over-parameterized regimes. For more general feature maps, error bounds are provided based on the singular values of the feature matrix. We demonstrate that appropriate weighting from prior knowledge can improve the generalization capability of the learned model.

cs.LG↗

arXiv2022

Implicit Regularization Effects of the Sobolev Norms in Image Processing

In this paper, we propose to use the general $L^2$-based Sobolev norms, i.e., $H^s$ norms where $s\in \mathbb{R}$, to measure the data discrepancy due to noise in image processing tasks that are formulated as optimization problems. As opposed to a popular trend of developing regularization methods, we emphasize that an implicit regularization effect can be achieved through the class of Sobolev norms as the data-fitting term. Specifically, we analyze that the implicit regularization comes from the weights that the $H^s$ norm imposes on different frequency contents of an underlying image. We further analyze the underlying noise assumption of using the Sobolev norm as the data-fitting term from a Bayesian perspective, build the connections with the Sobolev gradient-based methods and discuss the preconditioning effects on the convergence rate of the gradient descent algorithm, leading to a better understanding of functional spaces/metrics and the optimization process involved in image processing. Numerical results in full waveform inversion, image denoising and deblurring demonstrate the implicit regularization effects.

math.NA↗

arXiv2021

Anderson Acceleration Based on the $\mathcal{H}^{-s}$ Sobolev Norm for Contractive and Noncontractive Fixed-Point Operators

Anderson acceleration (AA) is a technique for accelerating the convergence of fixed-point iterations. In this paper, we apply AA to a sequence of functions and modify the norm in its internal optimization problem to the $\mathcal{H}^{-s}$ norm, for some positive integer $s$, to bias it towards low-frequency spectral content in the residual. We analyze the convergence of AA by quantifying its improvement over Picard iteration. We find that AA based on the $\mathcal{H}^{-2}$ norm is well-suited to solve fixed-point operators derived from second-order elliptic differential operators, including the Helmholtz equation.

math.NA↗

arXiv2021

Adjoint DSMC for nonlinear Boltzmann equation constrained optimization

Applications for kinetic equations such as optimal design and inverse problems often involve finding unknown parameters through gradient-based optimization algorithms. Based on the adjoint-state method, we derive two different frameworks for approximating the gradient of an objective functional constrained by the nonlinear Boltzmann equation. While the forward problem can be solved by the DSMC method, it is difficult to efficiently solve the high-dimensional continuous adjoint equation obtained by the "optimize-then-discretize" approach. This challenge motivates us to propose an adjoint DSMC method following the "discretize-then-optimize" approach for Boltzmann-constrained optimization. We also analyze the properties of the two frameworks and their connections. Several numerical examples are presented to demonstrate their accuracy and efficiency.

math.NA↗

arXiv2021

Stability of Gibbs Posteriors from the Wasserstein Loss for Bayesian Full Waveform Inversion

Recently, the Wasserstein loss function has been proven to be effective when applied to deterministic full-waveform inversion (FWI) problems. We consider the application of this loss function in Bayesian FWI so that the uncertainty can be captured in the solution. Other loss functions that are commonly used in practice are also considered for comparison. Existence and stability of the resulting Gibbs posteriors are shown on function space under weak assumptions on the prior and model. In particular, the distribution arising from the Wasserstein loss is shown to be quite stable with respect to high-frequency noise in the data. We then illustrate the difference between the resulting distributions numerically, using Laplace approximations to estimate the unknown velocity field and uncertainty associated with the estimates.

math.ST↗

arXiv2021

Optimal Transport Based Seismic Inversion: Beyond Cycle Skipping

Full-waveform inversion (FWI) is today a standard process for the inverse problem of seismic imaging. PDE-constrained optimization is used to determine unknown parameters in a wave equation that represent geophysical properties. The objective function measures the misfit between the observed data and the calculated synthetic data, and it has traditionally been the least-squares norm. In a sequence of papers, we introduced the Wasserstein metric from optimal transport as an alternative misfit function for mitigating the so-called cycle skipping, which is the trapping of the optimization process in local minima. In this paper, we first give a sharper theorem regarding the convexity of the Wasserstein metric as the objective function. We then focus on two new issues. One is the necessary normalization of turning seismic signals into probability measures such that the theory of optimal transport applies. The other, which is beyond cycle skipping, is the inversion for parameters below reflecting interfaces. For the first, we propose a class of normalizations and prove several favorable properties for this class. For the latter, we demonstrate that FWI using optimal transport can recover geophysical properties from domains where no seismic waves travel through. We finally illustrate these properties by the realistic application of imaging salt inclusions, which has been a significant challenge in exploration geophysics.

math.NA↗

arXiv2020

Anderson Acceleration for Seismic Inversion

The state-of-art seismic imaging techniques treat inversion tasks such as FWI and LSRTM as PDE-constrained optimization problems. Due to the large-scale nature, gradient-based optimization algorithms are preferred in practice to update the model iteratively. Higher-order methods converge in fewer iterations but often require higher computational costs, more line search steps, and bigger memory storage. A balance among these aspects has to be considered. We propose using Anderson acceleration (AA), a popular strategy to speed up the convergence of fixed-point iterations, to accelerate the steepest descent algorithm, which we innovatively treat as a fixed-point iteration. Independent of the dimensionality of the unknown parameters, the computational cost of implementing the method can be reduced to an extremely low-dimensional least-squares problem. The cost can be further reduced by a low-rank update. We discuss the theoretical connections and the differences between AA and other well-known optimization methods such as L-BFGS and the restarted GMRES and compare their computational cost and memory demand. Numerical examples of FWI and LSRTM applied to the Marmousi benchmark demonstrate the acceleration effects of AA. Compared with the steepest descent method, AA can achieve fast convergence and provide competitive results with some quasi-Newton methods, making it an attractive optimization strategy for seismic inversion.

math.NA↗

arXiv2020

The quadratic Wasserstein metric for inverse data matching

This work characterizes, analytically and numerically, two major effects of the quadratic Wasserstein ($W_2$) distance as the measure of data discrepancy in computational solutions of inverse problems. First, we show, in the infinite-dimensional setup, that the $W_2$ distance has a smoothing effect on the inversion process, making it robust against high-frequency noise in the data but leading to a reduced resolution for the reconstructed objects at a given noise level. Second, we demonstrate that for some finite-dimensional problems, the $W_2$ distance leads to optimization problems that have better convexity than the classical $L^2$ and $H^{-1}$ distances, making it a more preferred distance to use when solving such inverse matching problems.

math.NA↗

arXiv2019

Analysis and Application of Optimal Transport For Challenging Seismic Inverse Problems

In seismic exploration, sources and measurements of seismic waves on the surface are used to determine model parameters representing geophysical properties of the earth. Full-waveform inversion (FWI) is a nonlinear seismic inverse technique that inverts the model parameters by minimizing the difference between the synthetic data from the forward wave propagation and the observed true data in PDE-constrained optimization. The traditional least-squares method of measuring this difference suffers from three main drawbacks including local minima trapping, sensitivity to noise, and difficulties in reconstruction below reflecting layers. Unlike the local amplitude comparison in the least-squares method, the quadratic Wasserstein distance from the optimal transport theory considers both the amplitude differences and the phase mismatches when measuring data misfit. We will briefly review our earlier development and analysis of optimal transport-based inversion and include improvements, for example, a stronger convexity proof. The main focus will be on the third "challenge" with new results on sub-reflection recovery.

math.NA↗

arXiv2018

Seismic Inversion and the Data Normalization for Optimal Transport

Full waveform inversion (FWI) has recently become a favorite technique for the inverse problem of finding properties in the earth from measurements of vibrations of seismic waves on the surface. Mathematically, FWI is PDE constrained optimization where model parameters in a wave equation are adjusted such that the misfit between the computed and the measured dataset is minimized. In a sequence of papers, we have shown that the quadratic Wasserstein distance from optimal transport is to prefer as misfit functional over the standard $L^2$ norm. Datasets need however first to be normalized since seismic signals do not satisfy the requirements of optimal transport. There has been a puzzling contradiction in the results. Normalization methods that satisfy theorems pointing to ideal properties for FWI have not performed well in practical computations, and other scaling methods that do not satisfy these theorems have performed much better in practice. In this paper, we will shed light on this issue and resolve this contradiction.

math.NA↗

arXiv2018

Seismic Imaging and Optimal Transport

Seismology has been an active science for a long time. It changed character about 50 years ago when the earth's vibrations could be measured on the surface more accurately and more frequently in space and time. The full wave field could be determined, and partial differential equations (PDE) started to be used in the inverse process of finding properties of the interior of the earth. We will briefly review earlier techniques but mainly focus on Full Waveform Inversion (FWI) for the acoustic formulation. FWI is a PDE constrained optimization in which the variable velocity in a forward wave equation is adjusted such that the solution matches measured data on the surface. The minimization of the mismatch is usually coupled with the adjoint state method, which also includes the solution to an adjoint wave equation. The least-squares norm is the conventional objective function measuring the difference between simulated and measured data, but it often results in the minimization trapped in local minima. One way to mitigate this is by selecting another misfit function with better convexity properties. Here we propose using the quadratic Wasserstein metric as a new misfit function in FWI. The optimal map defining the quadratic Wasserstein metric can be computed by solving a Monge-Ampere equation. Theorems pointing to the advantages of using optimal transport over the least-squares norm will be discussed, and a number of large-scale computational examples will be presented.

math.NA↗

arXiv2017

Application of Optimal Transport and the Quadratic Wasserstein Metric to Full-Waveform Inversion

Conventional full-waveform inversion (FWI) using the least-squares norm ($L^2$) as a misfit function is known to suffer from cycle skipping. This increases the risk of computing a local rather than the global minimum of the misfit. In our previous work, we proposed the quadratic Wasserstein metric ($W_2$) as a new misfit function for FWI. The $W_2$ metric has been proved to have many ideal properties with regards to convexity and insensitivity to noise. When the observed and predicted seismic data are regarded as two density functions, the quadratic Wasserstein metric corresponds to the optimal cost of rearranging one density into the other, where the transportation cost is quadratic in distance. The difficulty of transforming seismic signals into nonnegative density functions is discussed. Unlike the $L^2$ norm, $W_2$ measures not only amplitude differences, but also global phase shifts, which helps to avoid cycle skipping issues. In this work, we build on our earlier method to cover more realistic high-resolution applications by embedding the $W_2$ technique into the framework of the adjoint-state method and applying it to seismic relevant 2D examples: the Camembert, the Marmousi, and the 2004 BP models. We propose a new way of using the $W_2$ metric trace-by-trace in FWI and compare it to global $W_2$ via the solution of the Monge-Ampère equation. With corresponding adjoint source, the velocity model can be updated using the l-BFGS method. Numerical results show the effectiveness of $W_2$ for alleviating cycle skipping issues and sensitivity to noise. Both mathematical theory and numerical examples demonstrate that the quadratic Wasserstein metric is a good candidate for a misfit function in seismic inversion.

physics.geo-ph↗