arXiv subjects

Siddhartha Mishra

Publications and source records attributed to Siddhartha Mishra.

At least 73 records · Page 4Linked to original sources

arXiv2022

Graph-Coupled Oscillator Networks

We propose Graph-Coupled Oscillator Networks (GraphCON), a novel framework for deep learning on graphs. It is based on discretizations of a second-order system of ordinary differential equations (ODEs), which model a network of nonlinear controlled and damped oscillators, coupled via the adjacency structure of the underlying graph. The flexibility of our framework permits any basic GNN layer (e.g. convolutional or attentional) as the coupling function, from which a multi-layer deep neural network is built up via the dynamics of the proposed ODEs. We relate the oversmoothing problem, commonly encountered in GNNs, to the stability of steady states of the underlying ODE and show that zero-Dirichlet energy steady states are not stable for our proposed ODEs. This demonstrates that the proposed framework mitigates the oversmoothing problem. Moreover, we prove that GraphCON mitigates the exploding and vanishing gradients problem to facilitate training of deep multi-layer GNNs. Finally, we show that our approach offers competitive performance with respect to the state-of-the-art on a variety of graph-based learning tasks.

cs.LG↗

arXiv2022

Word2Box: Capturing Set-Theoretic Semantics of Words using Box Embeddings

Learning representations of words in a continuous space is perhaps the most fundamental task in NLP, however words interact in ways much richer than vector dot product similarity can provide. Many relationships between words can be expressed set-theoretically, for example, adjective-noun compounds (eg. "red cars"$\subseteq$"cars") and homographs (eg. "tongue"$\cap$"body" should be similar to "mouth", while "tongue"$\cap$"language" should be similar to "dialect") have natural set-theoretic interpretations. Box embeddings are a novel region-based representation which provide the capability to perform these set-theoretic operations. In this work, we provide a fuzzy-set interpretation of box embeddings, and learn box representations of words using a set-theoretic training objective. We demonstrate improved performance on various word similarity tasks, particularly on less common words, and perform a quantitative and qualitative analysis exploring the additional unique expressivity provided by Word2Box.

cs.CL↗

arXiv2022

Agnostic Physics-Driven Deep Learning

This work establishes that a physical system can perform statistical learning without gradient computations, via an Agnostic Equilibrium Propagation (Aeqprop) procedure that combines energy minimization, homeostatic control, and nudging towards the correct response. In Aeqprop, the specifics of the system do not have to be known: the procedure is based only on external manipulations, and produces a stochastic gradient descent without explicit gradient computations. Thanks to nudging, the system performs a true, order-one gradient step for each training sample, in contrast with order-zero methods like reinforcement or evolutionary strategies, which rely on trial and error. This procedure considerably widens the range of potential hardware for statistical learning to any system with enough controllable parameters, even if the details of the system are poorly known. Aeqprop also establishes that in natural (bio)physical systems, genuine gradient-based statistical learning may result from generic, relatively simple mechanisms, without backpropagation and its requirement for analytic knowledge of partial derivatives.

cs.LG↗

arXiv2022

Variable-Input Deep Operator Networks

Existing architectures for operator learning require that the number and locations of sensors (where the input functions are evaluated) remain the same across all training and test samples, significantly restricting the range of their applicability. We address this issue by proposing a novel operator learning framework, termed Variable-Input Deep Operator Network (VIDON), which allows for random sensors whose number and locations can vary across samples. VIDON is invariant to permutations of sensor locations and is proved to be universal in approximating a class of continuous operators. We also prove that VIDON can efficiently approximate operators arising in PDEs. Numerical experiments with a diverse set of PDEs are presented to illustrate the robust performance of VIDON in learning operators.

cs.LG↗

arXiv2022

Physics Informed Neural Networks (PINNs)for approximating nonlinear dispersive PDEs

We propose a novel algorithm, based on physics-informed neural networks (PINNs) to efficiently approximate solutions of nonlinear dispersive PDEs such as the KdV-Kawahara, Camassa-Holm and Benjamin-Ono equations. The stability of solutions of these dispersive PDEs is leveraged to prove rigorous bounds on the resulting error. We present several numerical experiments to demonstrate that PINNs can approximate solutions of these dispersive PDEs very accurately

math.NA↗

arXiv2022

On the discrete equation model for compressible multiphase fluid flows

The modeling of multi-phase flow is very challenging, given the range of scales as well as the diversity of flow regimes that one encounters in this context. We revisit the discrete equation method (DEM) for two-phase flow in the absence of heat conduction and mass transfer. We analyze the resulting probability coefficients and prove their local convexity, rigorously establishing that our version of DEM can model different flow regimes ranging from the disperse to stratified (or separated) flow. Moreover, we reformulate the underlying mesoscopic model in terms of an one-parameter family of PDEs that interpolates between different flow regimes. We also propose two sets of procedures to enforce relaxation to equilibrium. We perform several numerical tests to show the flexibility of the proposed formulation, as well as to interpret different model components. The one-parameter family of PDEs provides an unified framework for modeling mean quantities for a multiphase flow, while at the same time identifying two key parameters that model the inherent uncertainty in terms of the underlying microstructure.

math.NA↗

arXiv2022

On the vanishing viscosity limit of statistical solutions of the incompressible Navier-Stokes equations

We study statistical solutions of the incompressible Navier-Stokes equation and their vanishing viscosity limit. We show that a formulation using correlation measures, which are probability measures accounting for spatial correlations, and moment equations is equivalent to statistical solutions in the Foias-Prodi sense. Under the assumption of weak scaling, a weaker version of Kolmogorov's self-similarity at small scales hypothesis that allows for intermittency corrections, we show that the limit is a statistical solution of the incompressible Euler equations. To pass to the limit, we derive a Karman-Howarth-Monin relation for statistical solutions and combine it with the weak scaling assumption and a compactness theorem for correlation measures.

math.AP↗

arXiv2022

Long Expressive Memory for Sequence Modeling

We propose a novel method called Long Expressive Memory (LEM) for learning long-term sequential dependencies. LEM is gradient-based, it can efficiently process sequential tasks with very long-term dependencies, and it is sufficiently expressive to be able to learn complicated input-output maps. To derive LEM, we consider a system of multiscale ordinary differential equations, as well as a suitable time-discretization of this system. For LEM, we derive rigorous bounds to show the mitigation of the exploding and vanishing gradients problem, a well-known challenge for gradient-based recurrent sequential learning methods. We also prove that LEM can approximate a large class of dynamical systems to high accuracy. Our empirical results, ranging from image and time-series classification through dynamical systems prediction to speech recognition and language modeling, demonstrate that LEM outperforms state-of-the-art recurrent neural networks, gated recurrent units, and long short-term memory models.

cs.LG↗

arXiv2022

On Bayesian data assimilation for PDEs with ill-posed forward problems

We study Bayesian data assimilation (filtering) for time-evolution PDEs, for which the underlying forward problem may be very unstable or ill-posed. Such PDEs, which include the Navier-Stokes equations of fluid dynamics, are characterized by a high sensitivity of solutions to perturbations of the initial data, a lack of rigorous global well-posedness results as well as possible non-convergence of numerical approximations. Under very mild and readily verifiable general hypotheses on the forward solution operator of such PDEs, we prove that the posterior measure expressing the solution of the Bayesian filtering problem is stable with respect to perturbations of the noisy measurements, and we provide quantitative estimates on the convergence of approximate Bayesian filtering distributions computed from numerical approximations. For the Navier-Stokes equations, our results imply uniform stability of the filtering problem even at arbitrarily small viscosity, when the underlying forward problem may become ill-posed, as well as the compactness of numerical approximants in a suitable metric on time-parametrized probability measures.

math.AP↗

arXiv2022

Error estimates for DeepOnets: A deep learning framework in infinite dimensions

DeepONets have recently been proposed as a framework for learning nonlinear operators mapping between infinite dimensional Banach spaces. We analyze DeepONets and prove estimates on the resulting approximation and generalization errors. In particular, we extend the universal approximation property of DeepONets to include measurable mappings in non-compact spaces. By a decomposition of the error into encoding, approximation and reconstruction errors, we prove both lower and upper bounds on the total error, relating it to the spectral decay properties of the covariance operators, associated with the underlying measures. We derive almost optimal error bounds with very general affine reconstructors and with random sensor locations as well as bounds on the generalization error, using covering number arguments. We illustrate our general framework with four prototypical examples of nonlinear operators, namely those arising in a nonlinear forced ODE, an elliptic PDE with variable coefficients and nonlinear parabolic and hyperbolic PDEs. While the approximation of arbitrary Lipschitz operators by DeepONets to accuracy $ε$ is argued to suffer from a "curse of dimensionality" (requiring a neural networks of exponential size in $1/ε$), in contrast, for all the above concrete examples of interest, we rigorously prove that DeepONets can break this curse of dimensionality (achieving accuracy $ε$ with neural networks of size that can grow algebraically in $1/ε$). Thus, we demonstrate the efficient approximation of a potentially large class of operators with this machine learning framework.

math.NA↗

arXiv2021

A novel fourth-order WENO interpolation technique. A possible new tool designed for radiative transfer

Context. Several numerical problems require the interpolation of discrete data that present various types of discontinuities. The radiative transfer is a typical example of such a problem. This calls for high-order well-behaved techniques to interpolate both smooth and discontinuous data. Aims. The final aim is to propose new techniques suitable for applications in the context of numerical radiative transfer. Methods. We have proposed and tested two different techniques. Essentially non-oscillatory (ENO) techniques generate several candidate interpolations based on different substencils. The smoothest candidate interpolation is determined from a measure for the local smoothness, thereby enabling the essential non-oscillatory property. Weighted ENO (WENO) techniques use a convex combination of all candidate substencils to obtain high-order accuracy in smooth regions while keeping the essentially non-oscillatory property. In particular, we have outlined and tested a novel well-performing fourth-order WENO interpolation technique for both uniform and nonuniform grids. Results. Numerical tests prove that the fourth-order WENO interpolation guarantees fourth-order accuracy in smooth regions of the interpolated functions. In the presence of discontinuities, the fourth-order WENO interpolation enables the non-oscillatory property, avoiding oscillations. Unlike Bézier and monotonic high-order Hermite interpolations, it does not degenerate to a linear interpolation near smooth extrema of the interpolated function. Conclusions. The novel fourth-order WENO interpolation guarantees high accuracy in smooth regions, while effectively handling discontinuities. This interpolation technique might be particularly suitable for several problems, including a number of radiative transfer applications such as multidimensional problems, multigrid methods, and formal solutions.

math.NA↗

arXiv2021

On the approximation of functions by tanh neural networks

We derive bounds on the error, in high-order Sobolev norms, incurred in the approximation of Sobolev-regular as well as analytic functions by neural networks with the hyperbolic tangent activation function. These bounds provide explicit estimates on the approximation error with respect to the size of the neural networks. We show that tanh neural networks with only two hidden layers suffice to approximate functions at comparable or better rates than much deeper ReLU neural networks.

math.NA↗

arXiv2021

Well-posedness of Bayesian inverse problems for hyperbolic conservation laws

We study the well-posedness of the Bayesian inverse problem for scalar hyperbolic conservation laws where the statistical information about inputs such as the initial datum and (possibly discontinuous) flux function are inferred from noisy measurements. In particular, the Lipschitz continuity of the measurement to posterior map as well as the stability of the posterior to approximations, are established with respect to the Wasserstein distance. Numerical experiments are presented to illustrate the derived estimates.

math.NA↗

arXiv2021

On universal approximation and error bounds for Fourier Neural Operators

Fourier neural operators (FNOs) have recently been proposed as an effective framework for learning operators that map between infinite-dimensional spaces. We prove that FNOs are universal, in the sense that they can approximate any continuous operator to desired accuracy. Moreover, we suggest a mechanism by which FNOs can approximate operators associated with PDEs efficiently. Explicit error bounds are derived to show that the size of the FNO, approximating operators associated with a Darcy type elliptic PDE and with the incompressible Navier-Stokes equations of fluid dynamics, only increases sub (log)-linearly in terms of the reciprocal of the error. Thus, FNOs are shown to efficiently approximate operators arising in a large class of PDEs.

math.NA↗

arXiv2021

Error analysis for physics informed neural networks (PINNs) approximating Kolmogorov PDEs

Physics informed neural networks approximate solutions of PDEs by minimizing pointwise residuals. We derive rigorous bounds on the error, incurred by PINNs in approximating the solutions of a large class of linear parabolic PDEs, namely Kolmogorov equations that include the heat equation and Black-Scholes equation of option pricing, as examples. We construct neural networks, whose PINN residual (generalization error) can be made as small as desired. We also prove that the total $L^2$-error can be bounded by the generalization error, which in turn is bounded in terms of the training error, provided that a sufficient number of randomly chosen training (collocation) points is used. Moreover, we prove that the size of the PINNs and the number of training samples only grow polynomially with the underlying dimension, enabling PINNs to overcome the curse of dimensionality in this context. These results enable us to provide a comprehensive error analysis for PINNs in approximating Kolmogorov PDEs.

math.NA↗

arXiv2021

UnICORNN: A recurrent model for learning very long time dependencies

The design of recurrent neural networks (RNNs) to accurately process sequential inputs with long-time dependencies is very challenging on account of the exploding and vanishing gradient problem. To overcome this, we propose a novel RNN architecture which is based on a structure preserving discretization of a Hamiltonian system of second-order ordinary differential equations that models networks of oscillators. The resulting RNN is fast, invertible (in time), memory efficient and we derive rigorous bounds on the hidden state gradients to prove the mitigation of the exploding and vanishing gradient problem. A suite of experiments are presented to demonstrate that the proposed RNN provides state of the art performance on a variety of learning tasks with (very) long-time dependencies.

cs.LG↗

arXiv2021

Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies

Circuits of biological neurons, such as in the functional parts of the brain can be modeled as networks of coupled oscillators. Inspired by the ability of these systems to express a rich set of outputs while keeping (gradients of) state variables bounded, we propose a novel architecture for recurrent neural networks. Our proposed RNN is based on a time-discretization of a system of second-order ordinary differential equations, modeling networks of controlled nonlinear oscillators. We prove precise bounds on the gradients of the hidden states, leading to the mitigation of the exploding and vanishing gradient problem for this RNN. Experiments show that the proposed RNN is comparable in performance to the state of the art on a variety of benchmarks, demonstrating the potential of this architecture to provide stable and accurate RNNs for processing complex sequential data.

cs.LG↗

arXiv2020

Arbitrarily high-order (weighted) essentially non-oscillatory finite difference schemes for anelastic flows on staggered meshes

We propose a WENO finite difference scheme to approximate anelastic flows, and scalars advected by them, on staggered grids. In contrast to existing WENO schemes on staggered grids, the proposed scheme is designed to be arbitrarily high-order accurate as it judiciously combines ENO interpolations of velocities with WENO reconstructions of spatial derivatives. A set of numerical experiments are presented to demonstrate the increase in accuracy and robustness with the proposed scheme, when compared to existing WENO schemes and state-of-the-art central finite difference schemes.

math.NA↗