arXiv subjects

Takashi Furuya

Publications and source records attributed to Takashi Furuya.

35 records · Page 2Linked to original sources

arXiv2024

Quantitative Approximation for Neural Operators in Nonlinear Parabolic Equations

Neural operators serve as universal approximators for general continuous operators. In this paper, we derive the approximation rate of solution operators for the nonlinear parabolic partial differential equations (PDEs), contributing to the quantitative approximation theorem for solution operators of nonlinear PDEs. Our results show that neural operators can efficiently approximate these solution operators without the exponential growth in model complexity, thus strengthening the theoretical foundation of neural operators. A key insight in our proof is to transfer PDEs into the corresponding integral equations via Duahamel's principle, and to leverage the similarity between neural operators and Picard's iteration, a classical algorithm for solving PDEs. This approach is potentially generalizable beyond parabolic PDEs to a range of other equations, including the Navier-Stokes equation, nonlinear Schrödinger equations and nonlinear wave equations, which can be solved by Picard's iteration.

cs.LG↗

arXiv2024

Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Experts

Mixture-of-Experts (MoEs) can scale up beyond traditional deep learning models by employing a routing strategy in which each input is processed by a single "expert" deep learning model. This strategy allows us to scale up the number of parameters defining the MoE while maintaining sparse activation, i.e., MoEs only load a small number of their total parameters into GPU VRAM for the forward pass depending on the input. In this paper, we provide an approximation and learning-theoretic analysis of mixtures of expert MLPs with (P)ReLU activation functions. We first prove that for every error level $\varepsilon>0$ and every Lipschitz function $f:[0,1]^n\to \mathbb{R}$, one can construct a MoMLP model (a Mixture-of-Experts comprising of (P)ReLU MLPs) which uniformly approximates $f$ to $\varepsilon$ accuracy over $[0,1]^n$, while only requiring networks of $\mathcal{O}(\varepsilon^{-1})$ parameters to be loaded in memory. Additionally, we show that MoMLPs can generalize since the entire MoMLP model has a (finite) VC dimension of $\tilde{O}(L\max\{nL,JW\})$, if there are $L$ experts and each expert has a depth and width of $J$ and $W$, respectively.

stat.ML↗

arXiv2023

Convergences for Minimax Optimization Problems over Infinite-Dimensional Spaces Towards Stability in Adversarial Training

Training neural networks that require adversarial optimization, such as generative adversarial networks (GANs) and unsupervised domain adaptations (UDAs), suffers from instability. This instability problem comes from the difficulty of the minimax optimization, and there have been various approaches in GANs and UDAs to overcome this problem. In this study, we tackle this problem theoretically through a functional analysis. Specifically, we show the convergence property of the minimax problem by the gradient descent over the infinite-dimensional spaces of continuous functions and probability measures under certain conditions. Using this setting, we can discuss GANs and UDAs comprehensively, which have been studied independently. In addition, we show that the conditions necessary for the convergence property are interpreted as stabilization techniques of adversarial training such as the spectral normalization and the gradient penalty.

stat.ML↗

arXiv2023

Out-of-distributional risk bounds for neural operators with applications to the Helmholtz equation

Despite their remarkable success in approximating a wide range of operators defined by PDEs, existing neural operators (NOs) do not necessarily perform well for all physics problems. We focus here on high-frequency waves to highlight possible shortcomings. To resolve these, we propose a subfamily of NOs enabling an enhanced empirical approximation of the nonlinear operator mapping wave speed to solution, or boundary values for the Helmholtz equation on a bounded domain. The latter operator is commonly referred to as the ''forward'' operator in the study of inverse problems. Our methodology draws inspiration from transformers and techniques such as stochastic depth. Our experiments reveal certain surprises in the generalization and the relevance of introducing stochastic depth. Our NOs show superior performance as compared with standard NOs, not only for testing within the training distribution but also for out-of-distribution scenarios. To delve into this observation, we offer an in-depth analysis of the Rademacher complexity associated with our modified models and prove an upper bound tied to their stochastic depth that existing NOs do not satisfy. Furthermore, we obtain a novel out-of-distribution risk bound tailored to Gaussian measures on Banach spaces, again relating stochastic depth with the bound. We conclude by proposing a hypernetwork version of the subfamily of NOs as a surrogate model for the mentioned forward operator.

cs.LG↗

arXiv2023

Globally injective and bijective neural operators

Recently there has been great interest in operator learning, where networks learn operators between function spaces from an essentially infinite-dimensional perspective. In this work we present results for when the operators learned by these networks are injective and surjective. As a warmup, we combine prior work in both the finite-dimensional ReLU and operator learning setting by giving sharp conditions under which ReLU layers with linear neural operators are injective. We then consider the case the case when the activation function is pointwise bijective and obtain sufficient conditions for the layer to be injective. We remark that this question, while trivial in the finite-rank case, is subtler in the infinite-rank case and is proved using tools from Fredholm theory. Next, we prove that our supplied injective neural operators are universal approximators and that their implementation, with finite-rank neural networks, are still injective. This ensures that injectivity is not `lost' in the transcription from analytical operators to their finite-rank implementation with networks. Finally, we conclude with an increase in abstraction and consider general conditions when subnetworks, which may be many layers deep, are injective and surjective and provide an exact inversion from a `linearization.' This section uses general arguments from Fredholm theory and Leray-Schauder degree theory for non-linear integral equations to analyze the mapping properties of neural operators in function spaces. These results apply to subnetworks formed from the layers considered in this work, under natural conditions. We believe that our work has applications in Bayesian UQ where injectivity enables likelihood estimation and in inverse problems where surjectivity and injectivity corresponds to existence and uniqueness, respectively.

cs.LG↗

arXiv2022

Local recovery of a piecewise constant anisotropic conductivity in EIT on domains with exposed corners

We study the local recovery of an unknown piecewise constant anisotropic conductivity in EIT (electric impedance tomography) on certain bounded Lipschitz domains $Ω$ in $\mathbb{R}^2$ with corners. The measurement is conducted on a connected open subset of the boundary $\partialΩ$ of $Ω$ containing corners and is given as a localized Neumann-to-Dirichlet map. The above unknown conductivity is defined via a decomposition of $Ω$ into polygonal cells. Specifically, we consider a parallelogram-based decomposition and a trapezoid-based decomposition. We assume that the decomposition is known, but the conductivity on each cell is unknown. We prove that the local recovery is almost surely true near a known piecewise constant anisotropic conductivity $γ_0$. We do so by proving that the injectivity of the Fréchet derivative $F'(γ_0)$ of the forward map $F$, say, at $γ_0$ is almost surely true. The proof presented, here, involves defining different classes of decompositions for $γ_0$ and a perturbation or contrast $H$ in a proper way so that we can find in the interior of a cell for $γ_0$ exposed single or double corners of a cell of $\mbox{supp}H$ for the former decomposition and latter decomposition, respectively. Then, by adapting the usual proof near such corners, we establish the aforementioned injectivity.

math.AP↗

arXiv2022

Inverse medium scattering problems with Kalman filter techniques

We study the inverse medium scattering problem to reconstruct the unknown inhomogeneous medium from the far-field patterns of scattered waves. The inverse scattering problem is generally ill-posed and nonlinear, and the iterative optimization method is often adapted. A natural iterative approach to this problem is to place all available measurements and mappings into one long vector and mapping, respectively, and to iteratively solve the linearized large system equation using the Tikhonov regularization method, which is called the Levenberg-Marquardt scheme. However, this is computationally expensive because we must construct the larger system equations when the number of available measurements increases. In this paper, we propose two reconstruction algorithms based on the Kalman filter. One is the algorithm equivalent to the Levenberg-Marquardt scheme, and the other is inspired by the Extended Kalman Filter. For the algorithm derivation, we iteratively apply the Kalman filter to the linearized equation for our nonlinear equation. Our proposed algorithms sequentially update the state and the weight of the norm for the state space, which avoids the construction of a large system equation and retains the information of past updates. Finally, we provide numerical examples to demonstrate our proposed algorithms.

math.AP↗

arXiv2021

Inverse medium scattering problems with Kalman filter techniques I. Linear case

In this paper, we study the inverse acoustic medium scattering problem to reconstruct the unknown inhomogeneous medium from far field patterns of scattered waves. We propose the reconstruction scheme based on the Kalman filter, which becomes possible to sequentially estimate the inhomogeneous medium. We also show that in the linear inverse problem, the estimation for the Kalman filter is equivalent to that for the Tikhonov regularization. Finally, we give numerical examples to demonstrate our proposed method.

math.AP↗

arXiv2021

Inverse medium scattering problems with Kalman filter techniques II. Nonlinear case

In this paper, we study the inverse medium scattering problem to reconstruct unknown inhomogeneous medium from far field patterns of scattered waves. In the first part of our work, the linear inverse scattering problem was discussed, while in the second part, we deal with the nonlinear problem. The main idea is to apply the linear Kalman filter to the linearized problem. There are several ways to linearize, which introduce two reconstruction algorithms. Finally, we give numerical examples to demonstrate our proposed method.

math.AP↗

arXiv2021

Spectral Pruning for Recurrent Neural Networks

Recurrent neural networks (RNNs) are a class of neural networks used in sequential tasks. However, in general, RNNs have a large number of parameters and involve enormous computational costs by repeating the recurrent structures in many time steps. As a method to overcome this difficulty, RNN pruning has attracted increasing attention in recent years, and it brings us benefits in terms of the reduction of computational cost as the time step progresses. However, most existing methods of RNN pruning are heuristic. The purpose of this paper is to study the theoretical scheme for RNN pruning method. We propose an appropriate pruning algorithm for RNNs inspired by "spectral pruning", and provide the generalization error bounds for compressed RNNs. We also provide numerical experiments to demonstrate our theoretical results and show the effectiveness of our pruning method compared with existing methods.

stat.ML↗

arXiv2021

Remarks on the factorization and monotonicity method for inverse acoustic scatterings

We study the factorization and monotonicity method for inverse acoustic scattering problems. Firstly, we give a new general functional analysis theorem for the monotonicity method. Comparing with the factorization method, the general theorem of the monotonicity generates reconstruction schemes under weaker {\it a priori} assumptions for unknown targets, and can directly deal with mixed problems so that the unknown targets have several different boundary conditions. Using the general theorem, we give the reconstruction scheme for the mixed crack that the Dirichlet boundary condition is imposed on one side of the crack and the Neumann boundary condition on the other side, which is a new extension of monotonicity method.

math.AP↗

arXiv2020

The factorization and monotonicity method for the defect in an open periodic waveguide

In this paper, we consider the inverse scattering problem to reconstruct the defect in an open periodic waveguide from near field data. Our first aim is to mention that there is a mistake in the factorization method of Lechleiter 2008. By this we can not apply it to solve this inverse problem. Our second aim is to give ways to understand the defect from inside (Theorem 1.1) and outside (Theorem 1.2) based on the monotonicity method.

math.AP↗

arXiv2019

The direct and inverse scattering problem for the semilinear Schrödinger equation

We study the direct and inverse scattering problem for the semilinear Schrödinger equation $Δu+a(x,u)+k^2u=0$ in $\mathbb{R}^d$. We show well-posedness in the direct problem for small solutions based on the Banach fixed point theorem, and the solution has the certain asymptotic behavior at infinity. We also show the inverse problem that the semilinear function $a(x,z)$ is uniquely determined from the scattering data. The idea is certain linearization that by using sources with several parameters we differentiate the nonlinear equation with respect to these parameter in order to get the linear one.

math.AP↗

arXiv2019

The monotonicity method for the inverse crack scattering problem

The monotonicity method for the inverse acoustic scattering problem is to understand the inclusion relation between an unknown object and artificial one by comparing the far field operator with artificial operator. This paper introduces the development of this method to the inverse crack scattering problem. Our aim is to give the following two indicators: One (Theorem 1.1) is to determine whether an artificial small arc is contained in the unknown arc. The other one (Theorem 1.2) is whether an artificial large domain contain the unknown one. Finally, numerical examples based on Theorem 1.1 are given.

math.AP↗

arXiv2019

Scattering by the local perturbation of an open periodic waveguide in the half plane

We consider the scattering problem of the local perturbation of an open periodic waveguide in the half plane. Recently by Kirsch and Lechleiter, a new radiation condition was introduced in order to solve the unperturbed case. In this paper, with the same radiation condition and an additional assumption we show the well-posedness of the perturbed scattering problem.

math.AP↗

arXiv2019

The factorization method for the inverse acoustic scattering problems with limited aperture data

We study the factorization method for the inverse acoustic scattering problems in the case of limited aperture data. In this case, the factorization of the far field operator is not symmetric. So, we can not apply the original factorization method which requires the symmetricity of the factorization. In this paper, we provide a new functional analytic theorem corresponding to the case of limited aperture data, and by applying it, we give the characterizations of the unknown domain.

math.AP↗

arXiv2017

Hermite expansions of some tempered distributions

We compute Hermite expansions of some tempered distributions by using the Bargmann transform. In other words, we calculate the Taylor expansions of the corresponding entire functions. Our method of computations seems to be superior to the direct computations in the shifts of singularities and the higher dimensional cases.

math.CA↗