arXiv subjects

Zecheng Zhang

Publications and source records attributed to Zecheng Zhang.

53 records · Page 3Linked to original sources

arXiv2022

BelNet: Basis enhanced learning, a mesh-free neural operator

Operator learning trains a neural network to map functions to functions. An ideal operator learning framework should be mesh-free in the sense that the training does not require a particular choice of discretization for the input functions, allows for the input and output functions to be on different domains, and is able to have different grids between samples. We propose a mesh-free neural operator for solving parametric partial differential equations. The basis enhanced learning network (BelNet) projects the input function into a latent space and reconstructs the output functions. In particular, we construct part of the network to learn the ``basis'' functions in the training process. This generalized the networks proposed in Chen and Chen's universal approximation theory for the nonlinear operators to account for differences in input and output meshes. Through several challenging high-contrast and multiscale problems, we show that our approach outperforms other operator learning methods for these tasks and allows for more freedom in the sampling and/or discretization process.

math.NA↗

arXiv2022

A replica exchange preconditioned Crank-Nicolson Langevin dynamic MCMC method for Bayesian inverse problems

This paper proposes a replica exchange preconditioned Langevin diffusion discretized by the Crank-Nicolson scheme (repCNLD) to handle high-dimensional and multi-modal distribution problems. Sampling from high-dimensional and multi-modal distributions is a challenging question. The performance of many standard MCMC chains deteriorates as the dimension of parameters increases, and many MCMC algorithms cannot capture all modes if the energy function is not convex. The proposed repCNLD can accelerate the convergence of the single-chain pCNLD, and can capture all modes of the multi-modal distributions. We proposed the Crank-Nicolson discretization, which is robust. Moreover, the discretization error grows linearly with respect to the time step size. We extend repCNLD to the multi-variance setting to further accelerate the convergence and save computation costs. Additionally, we derive an unbiased estimator of the swapping rate for the multi-variance repCNLD method, providing a guide for the choice of the low-fidelity model used in the second chain. We test our methods with high-dimensional Gaussian mixture models and high-dimensional nonlinear PDE inverse problems. Particularly, we employ the discrete adjoint method to efficiently calculate gradients for nonlinear PDE inverse problems.

math.NA↗

arXiv2022

Hybrid explicit-implicit learning for multiscale problems with time dependent source

The splitting method is a powerful method for solving partial differential equations. Various splitting methods have been designed to separate different physics, nonlinearities, and so on. Recently, a new splitting approach has been proposed where some degrees of freedom are handled implicitly while other degrees of freedom are handled explicitly. As a result, the scheme contains two equations, one implicit and the other explicit. The stability of this approach has been studied. It was shown that the time step scales as the coarse spatial mesh size, which can provide a significant computational advantage. However, the implicit solution part can still be expensive, especially for nonlinear problems. In this paper, we introduce modified partial machine learning algorithms to replace the implicit solution part of the algorithm. These algorithms are first introduced in arXiv:2109.02147, where a homogeneous source term is considered along with the Transformer, which is a neural network that can predict future dynamics. In this paper, we consider time-dependent source terms which is a generalization of the previous work. Moreover, we use the whole history of the solution to train the network. As the implicit part of the equations is more complicated to solve, we design a neural network to predict it based on training. Furthermore, we compute the explicit part of the solution using our splitting strategy. In addition, we use Proper Orthogonal Decomposition based model reduction in machine learning. The machine learning algorithms provide computational saving without sacrificing accuracy. We present three numerical examples which show that our machine learning scheme is stable and accurate.

math.NA↗

arXiv2022

SAIS: Supervising and Augmenting Intermediate Steps for Document-Level Relation Extraction

Stepping from sentence-level to document-level, the research on relation extraction (RE) confronts increasing text length and more complicated entity interactions. Consequently, it is more challenging to encode the key information sources--relevant contexts and entity types. However, existing methods only implicitly learn to model these critical information sources while being trained for RE. As a result, they suffer the problems of ineffective supervision and uninterpretable model predictions. In contrast, we propose to explicitly teach the model to capture relevant contexts and entity types by supervising and augmenting intermediate steps (SAIS) for RE. Based on a broad spectrum of carefully designed tasks, our proposed SAIS method not only extracts relations of better quality due to more effective supervision, but also retrieves the corresponding supporting evidence more accurately so as to enhance interpretability. By assessing model uncertainty, SAIS further boosts the performance via evidence-based data augmentation and ensemble inference while reducing the computational cost. Eventually, SAIS delivers state-of-the-art RE results on three benchmarks (DocRED, CDR, and GDA) and outperforms the runner-up by 5.04% relatively in F1 score in evidence retrieval on DocRED.

cs.CL↗

arXiv2021

Accelerated replica exchange stochastic gradient Langevin diffusion enhanced Bayesian DeepONet for solving noisy parametric PDEs

The Deep Operator Networks~(DeepONet) is a fundamentally different class of neural networks that we train to approximate nonlinear operators, including the solution operator of parametric partial differential equations (PDE). DeepONets have shown remarkable approximation and generalization capabilities even when trained with relatively small datasets. However, the performance of DeepONets deteriorates when the training data is polluted with noise, a scenario that occurs very often in practice. To enable DeepONets training with noisy data, we propose using the Bayesian framework of replica-exchange Langevin diffusion. Such a framework uses two particles, one for exploring and another for exploiting the loss function landscape of DeepONets. We show that the proposed framework's exploration and exploitation capabilities enable (1) improved training convergence for DeepONets in noisy scenarios and (2) attaching an uncertainty estimate for the predicted solutions of parametric PDEs. In addition, we show that replica-exchange Langeving Diffusion (remarkably) also improves the DeepONet's mean prediction accuracy in noisy scenarios compared with vanilla DeepONets trained with state-of-the-art gradient-based optimization algorithms (e.g. Adam). To reduce the potentially high computational cost of replica, in this work, we propose an accelerated training framework for replica-exchange Langevin diffusion that exploits the neural network architecture of DeepONets to reduce its computational cost up to 25% without compromising the proposed framework's performance. Finally, we illustrate the effectiveness of the proposed Bayesian framework using a series of experiments on four parametric PDE problems.

math.NA↗

arXiv2021

Theoretical and numerical studies of inverse source problem for the linear parabolic equation with sparse boundary measurements

We consider the inverse source problem in the parabolic equation, where the unknown source possesses the semi-discrete formulation. Theoretically, we prove that the flux data from any nonempty open subset of the boundary can uniquely determine the semi-discrete source. This means the observed area can be extremely small, and that is why we call the data as sparse boundary data. For the numerical reconstruction, we formulate the problem from the Bayesian sequential prediction perspective and conduct the numerical examples which estimate the space-time-dependent source state by state. To better demonstrate the performance of the method, we solve two common multiscale problems from two models with a long sequence of the source. The numerical results illustrate that the inversion is accurate and efficient.

math.NA↗

arXiv2021

HEI: hybrid explicit-implicit learning for multiscale problems

Splitting is a method to handle application problems by splitting physics, scales, domain, and so on. Many splitting algorithms have been designed for efficient temporal discretization. In this paper, our goal is to use temporal splitting concepts in designing machine learning algorithms and, at the same time, help splitting algorithms by incorporating data and speeding them up. Since the spitting solution usually has an explicit and implicit part, we will call our method hybrid explicit-implict (HEI) learning. We will consider a recently introduced multiscale splitting algorithms. To approximate the dynamics, only a few degrees of freedom are solved implicitly, while others explicitly. In this paper, we use this splitting concept in machine learning and propose several strategies. First, the implicit part of the solution can be learned as it is more difficult to solve, while the explicit part can be computed. This provides a speed-up and data incorporation for splitting approaches. Secondly, one can design a hybrid neural network architecture because handling explicit parts requires much fewer communications among neurons and can be done efficiently. Thirdly, one can solve the coarse grid component via PDEs or other approximation methods and construct simpler neural networks for the explicit part of the solutions. We discuss these options and implement one of them by interpreting it as a machine translation task. This interpretation successfully enables us using the Transformer since it can perform model reduction for multiple time series and learn the connection. We also find that the splitting scheme is a great platform to predict the coarse solution with insufficient information of the target model: the target problem is partially given and we need to solve it through a known problem. We conduct four numerical examples and the results show that our method is stable and accurate.

math.NA↗

arXiv2021

NH-PINN: Neural homogenization based physics-informed neural network for multiscale problems

Physics-informed neural network (PINN) is a data-driven approach to solve equations. It is successful in many applications; however, the accuracy of the PINN is not satisfactory when it is used to solve multiscale equations. Homogenization is a way of approximating a multiscale equation by a homogenized equation without multiscale property; it includes solving cell problems and the homogenized equation. The cell problems are periodic; and we propose an oversampling strategy which greatly improves the PINN accuracy on periodic problems. The homogenized equation has constant or slow dependency coefficient and can also be solved by PINN accurately. We hence proposed a 3-step method to improve the PINN accuracy for solving multiscale problems with the help of the homogenization. We apply our method to solve three equations which represent three different homogenization. The results show that the proposed method greatly improves the PINN accuracy. Besides, we also find that the PINN aided homogenization may achieve better accuracy than the numerical methods driven homogenization; PINN hence is a potential alternative to implementing the homogenization.

math.NA↗

arXiv2021

Multi-variance replica exchange stochastic gradient MCMC for inverse and forward Bayesian physics-informed neural network

Physics-informed neural network (PINN) has been successfully applied in solving a variety of nonlinear non-convex forward and inverse problems. However, the training is challenging because of the non-convex loss functions and the multiple optima in the Bayesian inverse problem. In this work, we propose a multi-variance replica exchange stochastic gradient Langevin diffusion method to tackle the challenge of the multiple local optima in the optimization and the challenge of the multiple modal posterior distribution in the inverse problem. Replica exchange methods are capable of escaping from the local traps and accelerating the convergence. However, it may not be efficient to solve mathematical inversion problems by using the vanilla replica method directly since the method doubles the computational cost in evaluating the forward solvers (likelihood functions) in the two chains. To address this issue, we propose to make different assumptions on the energy function estimation and this facilities one to use solvers of different fidelities in the likelihood function evaluation. We give an unbiased estimate of the swapping rate and give an estimation of the discretization error of the scheme. To verify our idea, we design and solve four inverse problems which have multiple modes. The proposed method is also employed to train the Bayesian PINN to solve the forward and inverse problems; faster and more accurate convergence has been observed when compared to the stochastic gradient Langevin diffusion (SGLD) method and vanila replica exchange methods.

math.NA↗

arXiv2021

Distributed-Memory Tensor Completion for Generalized Loss Functions in Python using New Sparse Tensor Kernels

Tensor computations are increasingly prevalent numerical techniques in data science, but pose unique challenges for high-performance implementation. We provide novel algorithms and systems infrastructure which enable efficient parallel implementation of algorithms for tensor completion with generalized loss functions. Specifically, we consider alternating minimization, coordinate minimization, and a quasi-Newton (generalized Gauss-Newton) method. By extending the Cyclops library, we implement all of these methods in high-level Python syntax. To make possible tensor completion for very sparse tensors, we introduce new multi-tensor primitives, for which we provide specialized parallel implementations. We compare these routines to pairwise contraction of sparse tensors by reduction to hypersparse matrix formats, and find that the multi-tensor routines are more efficient in theoretical cost and execution time in experiments. We provide microbenchmarking results on the Stampede2 supercomputer to demonstrate the efficiency of the new primitives and Cyclops functionality. We then study the performance of the tensor completion methods for a synthetic tensor with 10 billion nonzeros and the Netflix dataset, considering both least squares and Poisson loss functions.

cs.DC↗

arXiv2021

Computational multiscale methods for parabolic wave approximations in heterogeneous media

In this paper, we develop a computational multiscale to solve the parabolic wave approximation with heterogeneous and variable media. Parabolic wave approximation is a technique to approximate the full wave equation. One benefit of the method is that: one wave propagation direction can be taken as an evolution direction, and we then can discretize it using a classical scheme like Backward Euler. Consequently, we obtain a set of quasi-gas-dynamic (QGD) models with different heterogeneous permeability fields. Then, we employ constraint energy minimization generalized multiscale finite element method (CEM-GMsFEM) to perform spatial discretization for the problem. The resulting system can be solved by combining the central difference in time evolution. Due to the variable media, we apply the technique of proper orthogonal decomposition (POD) to further the dimension of the problem and solve the corresponding model problem in the POD space instead of in the whole multiscale space spanned by all possible multiscale basis functions. We prove the stability of the full discretization scheme and give the convergence analysis of the proposed approximation scheme. Numerical results verify the effectiveness of the proposed method.

math.NA↗

arXiv2021

A deep neural network approach on solving the linear transport model under diffusive scaling

In this work, we propose a learning method for solving the linear transport equation under the diffusive scaling. Due to the multiscale nature of our model equation, the model is challenging to solve by using conventional methods. We employ the physical informed neural network (PINN) framework, a mesh-free learning method that can numerically solve partial differential equations. Compared to conventional methods (such as finite difference or finite element type), our proposed learning method is able to obtain the solution at any given points in the chosen domain accurately and efficiently, which enables us to better understand the physics underlying the model. In our framework, the solution is approximated by a neural network that satisfies both the governing equation and other constraints. The network is then trained with a combination of different loss terms. Using the approximation theory and energy estimates for kinetic models, we prove theoretically that the total loss vanishes as the neural network converges, upon which the neural network approximated solution converges pointwisely to the analytic solution of the linear transport model. Numerical experiments for two benchmark examples are conducted to verify the effectiveness and accuracy of our proposed method.

math.NA↗

arXiv2020

Multi-agent Reinforcement Learning Accelerated MCMC on Multiscale Inversion Problem

In this work, we propose a multi-agent actor-critic reinforcement learning (RL) algorithm to accelerate the multi-level Monte Carlo Markov Chain (MCMC) sampling algorithms. The policies (actors) of the agents are used to generate the proposal in the MCMC steps; and the critic, which is centralized, is in charge of estimating the long term reward. We verify our proposed algorithm by solving an inverse problem with multiple scales. There are several difficulties in the implementation of this problem by using traditional MCMC sampling. Firstly, the computation of the posterior distribution involves evaluating the forward solver, which is very time consuming for a problem with heterogeneous. We hence propose to use the multi-level algorithm. More precisely, we use the generalized multiscale finite element method (GMsFEM) as the forward solver in evaluating a posterior distribution in the multi-level rejection procedure. Secondly, it is hard to find a function which can generate samplings which are meaningful. To solve this issue, we learn an RL policy as the proposal generator. Our experiments show that the proposed method significantly improves the sampling process

cs.LG↗

arXiv2020

A multi-stage deep learning based algorithm for multiscale modelreduction

In this work, we propose a multi-stage training strategy for the development of deep learning algorithms applied to problems with multiscale features. Each stage of the pro-posed strategy shares an (almost) identical network structure and predicts the same reduced order model of the multiscale problem. The output of the previous stage will be combined with an intermediate layer for the current stage. We numerically show that using different reduced order models as inputs of each stage can improve the training and we propose several ways of adding different information into the systems. These methods include mathematical multiscale model reductions and network approaches; but we found that the mathematical approach is a systematical way of decoupling information and gives the best result. We finally verified our training methodology on a time dependent nonlinear problem and a steady state model

math.NA↗

arXiv2020

Computational multiscale methods for quasi-gas dynamic equations

In this paper, we consider the quasi-gas-dynamic (QGD) model in a multiscale environment. The model equations can be regarded as a hyperbolic regularization and are derived from kinetic equations. So far, the research on QGD models has been focused on problems with constant coefficients. In this paper, we investigate the QGD model in multiscale media, which can be used in porous media applications. This multiscale problem is interesting from a multiscale methodology point of view as the model problem has a hyperbolic multiscale term, and designing multiscale methods for hyperbolic equations is challenging. In the paper, we apply the constraint energy minimizing generalized multiscale finite element method (CEM-GMsFEM) combined with the leapfrog scheme in time to solve this problem. The CEM-GMsFEM provides a flexible and systematical framework to construct crucial multiscale basis functions for approximating the solution to the problem with reduced computational cost. With this approach of spatial discretization, we establish the stability of the fully discretized scheme under a relaxed version of the so-called CFL condition. Complete convergence analysis of the proposed method is presented. Numerical results are provided to illustrate and verify the theoretical findings.

math.NA↗

arXiv2020

paper2repo: GitHub Repository Recommendation for Academic Papers

GitHub has become a popular social application platform, where a large number of users post their open source projects. In particular, an increasing number of researchers release repositories of source code related to their research papers in order to attract more people to follow their work. Motivated by this trend, we describe a novel item-item cross-platform recommender system, $\textit{paper2repo}$, that recommends relevant repositories on GitHub that match a given paper in an academic search system such as Microsoft Academic. The key challenge is to identify the similarity between an input paper and its related repositories across the two platforms, $\textit{without the benefit of human labeling}$. Towards that end, paper2repo integrates text encoding and constrained graph convolutional networks (GCN) to automatically learn and map the embeddings of papers and repositories into the same space, where proximity offers the basis for recommendation. To make our method more practical in real life systems, labels used for model training are computed automatically from features of user actions on GitHub. In machine learning, such automatic labeling is often called {\em distant supervision\/}. To the authors' knowledge, this is the first distant-supervised cross-platform (paper to repository) matching system. We evaluate the performance of paper2repo on real-world data sets collected from GitHub and Microsoft Academic. Results demonstrate that it outperforms other state of the art recommendation methods.

cs.SI↗

arXiv2020

Learning Algorithms for Coarsening Uncertainty Space and Applications to Multiscale Simulations

In this paper, we investigate and design multiscale simulations for stochastic multiscale PDEs. As for the space, we consider a coarse grid and a known multiscale method, the Generalized Multiscale Finite Element Method (GMsFEM). In order to obtain a small dimensional representation of the solution in each coarse block, the uncertainty space needs to be partitioned (coarsened). This coarsening collects realizations that provide similar multiscale features as outlined in GMsFEM (or other method of choice). This step is known to be computationally demanding as it requires many local solves and clustering based on them. In this paper, we take a different approach and learn coarsening the uncertainty space. Our methods use deep learning techniques in identifying clusters(coarsening) in the uncertainty space. We use convolutional neural networks combined with some techniques in adversary neural networks. We define appropriate loss functions in the proposed neural networks, where the loss function is composed of several parts that includes terms related to clusters and reconstruction of basis functions. We present numerical results for channelized permeability fields in the examples of flows in porous media.

math.NA↗