Searcharxiv⌕ Search

arXiv subjects

Isao Ishikawa

Publications and source records attributed to Isao Ishikawa.

At least 37 records · Page 2Linked to original sources
arXiv2024

Koopman and Perron-Frobenius Operators on reproducing kernel Banach spaces

Koopman and Perron-Frobenius operators for dynamical systems have been getting popular in a number of fields in science these days. Properties of the Koopman operator essentially depend on the choice of function spaces where it acts. Particularly the case of reproducing kernel Hilbert spaces (RKHSs) draws more and more attention in data science. In this paper, we give a general framework for Koopman and Perron-Frobenius operators on reproducing kernel Banach spaces (RKBSs). More precisely, we extend basic known properties of these operators from RKHSs to RKBSs and state new results, including symmetry and sparsity concepts, on these operators on RKBS for discrete and continuous time systems.

math.DS↗
arXiv2023

Joint Group Invariant Functions on Data-Parameter Domain Induce Universal Neural Networks

The symmetry and geometry of input data are considered to be encoded in the internal data representation inside the neural network, but the specific encoding rule has been less investigated. In this study, we present a systematic method to induce a generalized neural network and its right inverse operator, called the ridgelet transform, from a joint group invariant function on the data-parameter domain. Since the ridgelet transform is an inverse, (1) it can describe the arrangement of parameters for the network to represent a target function, which is understood as the encoding rule, and (2) it implies the universality of the network. Based on the group representation theory, we present a new simple proof of the universality by using Schur's lemma in a unified manner covering a wide class of networks, for example, the original ridgelet transform, formal deep networks, and the dual voice transform. Since traditional universality theorems were demonstrated based on functional analysis, this study sheds light on the group theoretic aspect of the approximation theory, connecting geometric deep learning to abstract harmonic analysis.

cs.LG↗
arXiv2023

Koopman spectral analysis of skew-product dynamics on Hilbert $C^*$-modules

We introduce a linear operator on a Hilbert $C^*$-module for analyzing skew-product dynamical systems. The operator is defined by composition and multiplication. We show that it admits a decomposition in the Hilbert $C^*$-module, called eigenoperator decomposition, that generalizes the concept of the eigenvalue decomposition. This decomposition reconstructs the Koopman operator of the system in a manner that represents the continuous spectrum through eigenoperators. In addition, it is related to the notions of cocycle and Oseledets subspaces and it is useful for characterizing coherent structures under skew-product dynamics. We present numerical applications to simple systems on two-dimensional domains.

math.DS↗
arXiv2023

Boundedness of composition operators on higher order Besov spaces in one dimension

This paper aims to characterize boundedness of composition operators on Besov spaces $B^s_{p,q}$ of higher order derivatives $s>1+1/p$ on the one-dimensional Euclidean space. In contrast to the lower order case $0 1$. We prove a relation between the composition operators and pointwise multipliers of Besov spaces, and effectively use the characterizations of the pointwise multipliers. As a result, we obtain necessary and sufficient conditions for the boundedness of composition operators for general $p$, $q$, and $s$ such that $1 1+1/p$. In this paper, we treat, as a map that induces the composition operator, not only a homeomorphism on the real line but also a continuous map whose number of elements of inverse images at any one point is bounded above. We also show a similar characterization of the boundedness of composition operators on Sobolev spaces.

math.FA↗
arXiv2023

Bounded composition operators on functional quasi-Banach spaces and stability of dynamical systems

In this paper, we investigate the boundedness of composition operators defined on a quasi-Banach space continuously included in the space of smooth functions on a manifold. We prove that the boundedness of a composition operator strongly restricts the behavior of the original map, and it provides an effective method to investigate the properties of composition operators using the theory of dynamical system. Consequently, we prove that only affine maps can induce bounded composition operators on any quasi-Banach space continuously included in the space of entire functions of one variable if the function space contains a nonconstant function. We also prove that any polynomial automorphisms except affine transforms cannot induce bounded composition operators on a quasi-Banach space composed of entire functions in the two-dimensional complex affine space under several mild conditions.

math.FA↗
arXiv2022

Universality of Group Convolutional Neural Networks Based on Ridgelet Analysis on Groups

We show the universality of depth-2 group convolutional neural networks (GCNNs) in a unified and constructive manner based on the ridgelet theory. Despite widespread use in applications, the approximation property of (G)CNNs has not been well investigated. The universality of (G)CNNs has been shown since the late 2010s. Yet, our understanding on how (G)CNNs represent functions is incomplete because the past universality theorems have been shown in a case-by-case manner by manually/carefully assigning the network parameters depending on the variety of convolution layers, and in an indirect manner by converting/modifying the (G)CNNs into other universal approximators such as invariant polynomials and fully-connected networks. In this study, we formulate a versatile depth-2 continuous GCNN $S[γ]$ as a nonlinear mapping between group representations, and directly obtain an analysis operator, called the ridgelet trasform, that maps a given function $f$ to the network parameter $γ$ so that $S[γ]=f$. The proposed GCNN covers typical GCNNs such as the cyclic convolution on multi-channel images, networks on permutation-invariant inputs (Deep Sets), and $\mathrm{E}(n)$-equivariant networks. The closed-form expression of the ridgelet transform can describe how the network parameters are organized to represent a function. While it has been known only for fully-connected networks, this study is the first to obtain the ridgelet transform for GCNNs. By discretizing the closed-form expression, we can systematically generate a constructive proof of the $cc$-universality of finite GCNNs. In other words, our universality proofs are more unified and constructive than previous proofs.

cs.LG↗
arXiv2022

Fully-Connected Network on Noncompact Symmetric Space and Ridgelet Transform based on Helgason-Fourier Analysis

Neural network on Riemannian symmetric space such as hyperbolic space and the manifold of symmetric positive definite (SPD) matrices is an emerging subject of research in geometric deep learning. Based on the well-established framework of the Helgason-Fourier transform on the noncompact symmetric space, we present a fully-connected network and its associated ridgelet transform on the noncompact symmetric space, covering the hyperbolic neural network (HNN) and the SPDNet as special cases. The ridgelet transform is an analysis operator of a depth-2 continuous network spanned by neurons, namely, it maps an arbitrary given function to the weights of a network. Thanks to the coordinate-free reformulation, the role of nonlinear activation functions is revealed to be a wavelet function, and the reconstruction formula directly yields the universality of the proposed networks.

cs.LG↗
arXiv2022

Universal approximation property of invertible neural networks

Invertible neural networks (INNs) are neural network architectures with invertibility by design. Thanks to their invertibility and the tractability of Jacobian, INNs have various machine learning applications such as probabilistic modeling, generative modeling, and representation learning. However, their attractive properties often come at the cost of restricting the layer designs, which poses a question on their representation power: can we use these models to approximate sufficiently diverse functions? To answer this question, we have developed a general theoretical framework to investigate the representation power of INNs, building on a structure theorem of differential geometry. The framework simplifies the approximation problem of diffeomorphisms, which enables us to show the universal approximation properties of INNs. We apply the framework to two representative classes of INNs, namely Coupling-Flow-based INNs (CF-INNs) and Neural Ordinary Differential Equations (NODEs), and elucidate their high representation power despite the restrictions on their architectures.

cs.LG↗
arXiv2021

Reproducing kernel Hilbert C*-module and kernel mean embeddings

Kernel methods have been among the most popular techniques in machine learning, where learning tasks are solved using the property of reproducing kernel Hilbert space (RKHS). In this paper, we propose a novel data analysis framework with reproducing kernel Hilbert $C^*$-module (RKHM) and kernel mean embedding (KME) in RKHM. Since RKHM contains richer information than RKHS or vector-valued RKHS (vvRKHS), analysis with RKHM enables us to capture and extract structural properties in such as functional data. We show a branch of theories for RKHM to apply to data analysis, including the representer theorem, and the injectivity and universality of the proposed KME. We also show RKHM generalizes RKHS and vvRKHS. Then, we provide concrete procedures for employing RKHM and the proposed KME to data analysis.

stat.ML↗
arXiv2021

Generalized eigenvalues of the Perron-Frobenius operators of symbolic dynamical systems

The generalized spectral theory is an effective approach to analyze a linear operator on a Hilbert space $\mathcal{H}$ with a continuous spectrum. The generalized spectrum is computed via analytic continuations of the resolvent operators using a dense locally convex subspace $X$ of $\mathcal{H}$ and its dual space $X'$. The three topological spaces $X \subset \mathcal{H} \subset X'$ is called the rigged Hilbert space or the Gelfand triplet. In this paper, the generalized spectra of the Perron-Frobenius operators of the one-sided and two-sided shifts of finite types (symbolic dynamical systems) are determined. A one-sided subshift of finite type which is conjugate to the multiplication with the golden ration on $[0,1]$ modulo $1$ is also considered. A new construction of the Gelfand triplet for the generalized spectrum of symbolic dynamical systems is proposed by means of an algebraic procedure. The asymptotic formula of the iteration of Perron-Frobenius operators is also given. The iteration converges to the mixing state whose rate of convergence is determined by the generalized spectrum.

math.DS↗
arXiv2021

Composition operators on reproducing kernel Hilbert spaces with analytic positive definite functions

In this paper, we specify what functions induce the bounded composition operators on a reproducing kernel Hilbert space (RKHS) associated with an analytic positive definite function defined on $\mathbf{R}^d$. We prove that only affine transforms can do so in a pretty large class of RKHS. Our result covers not only the Paley-Wiener space on the real line, studied in previous works, but also much more general RKHSs corresponding to analytic positive definite functions where existing methods do not work. Our method only relies on an intrinsic properties of the RKHSs, and we establish a connection between the behavior of composition operators and the asymptotic properties of the greatest zeros of orthogonal polynomials on a weighted $L^2$-spaces on the real line. We also investigate the compactness of the composition operators and show that any bounded composition operators cannot be compact in our situation.

math.FA↗
arXiv2021

Ridge Regression with Over-Parametrized Two-Layer Networks Converge to Ridgelet Spectrum

Characterization of local minima draws much attention in theoretical studies of deep learning. In this study, we investigate the distribution of parameters in an over-parametrized finite neural network trained by ridge regularized empirical square risk minimization (RERM). We develop a new theory of ridgelet transform, a wavelet-like integral transform that provides a powerful and general framework for the theoretical study of neural networks involving not only the ReLU but general activation functions. We show that the distribution of the parameters converges to a spectrum of the ridgelet transform. This result provides a new insight into the characterization of the local minima of neural networks, and the theoretical background of an inductive bias theory based on lazy regimes. We confirm the visual resemblance between the parameter distribution trained by SGD, and the ridgelet spectrum calculated by numerical integration through numerical experiments with finite models.

cs.LG↗
arXiv2020

Universal Approximation Property of Neural Ordinary Differential Equations

Neural ordinary differential equations (NODEs) is an invertible neural network architecture promising for its free-form Jacobian and the availability of a tractable Jacobian determinant estimator. Recently, the representation power of NODEs has been partly uncovered: they form an $L^p$-universal approximator for continuous maps under certain conditions. However, the $L^p$-universality may fail to guarantee an approximation for the entire input domain as it may still hold even if the approximator largely differs from the target function on a small region of the input space. To further uncover the potential of NODEs, we show their stronger approximation property, namely the $\sup$-universality for approximating a large class of diffeomorphisms. It is shown by leveraging a structure theorem of the diffeomorphism group, and the result complements the existing literature by establishing a fairly large set of mappings that NODEs can approximate with a stronger guarantee.

cs.LG↗
arXiv2020

A global universality of two-layer neural networks with ReLU activations

In the present study, we investigate a universality of neural networks, which concerns a density of the set of two-layer neural networks in a function spaces. There are many works that handle the convergence over compact sets. In the present paper, we consider a global convergence by introducing a norm suitably, so that our results will be uniform over any compact set.

cs.LG↗
arXiv2020

Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators

Invertible neural networks based on coupling flows (CF-INNs) have various machine learning applications such as image synthesis and representation learning. However, their desirable characteristics such as analytic invertibility come at the cost of restricting the functional forms. This poses a question on their representation power: are CF-INNs universal approximators for invertible functions? Without a universality, there could be a well-behaved invertible transformation that the CF-INN can never approximate, hence it would render the model class unreliable. We answer this question by showing a convenient criterion: a CF-INN is universal if its layers contain affine coupling and invertible linear functions as special cases. As its corollary, we can affirmatively resolve a previously unsolved problem: whether normalizing flow models based on affine coupling can be universal distributional approximators. In the course of proving the universality, we prove a general theorem to show the equivalence of the universality for certain diffeomorphism classes, a theoretical insight that is of interest by itself.

cs.LG↗
arXiv2020

Krylov Subspace Method for Nonlinear Dynamical Systems with Random Noise

Operator-theoretic analysis of nonlinear dynamical systems has attracted much attention in a variety of engineering and scientific fields, endowed with practical estimation methods using data such as dynamic mode decomposition. In this paper, we address a lifted representation of nonlinear dynamical systems with random noise based on transfer operators, and develop a novel Krylov subspace method for estimating the operators using finite data, with consideration of the unboundedness of operators. For this purpose, we first consider Perron-Frobenius operators with kernel-mean embeddings for such systems. We then extend the Arnoldi method, which is the most classical type of Kryov subspace method, so that it can be applied to the current case. Meanwhile, the Arnoldi method requires the assumption that the operator is bounded, which is not necessarily satisfied for transfer operators on nonlinear systems. We accordingly develop the shift-invert Arnoldi method for Perron-Frobenius operators to avoid this problem. Also, we describe an approach of evaluating predictive accuracy by estimated operators on the basis of the maximum mean discrepancy, which is applicable, for example, to anomaly detection in complex systems. The empirical performance of our methods is investigated using synthetic and real-world healthcare data.

cs.LG↗
arXiv2020

Boundedness of composition operators on Morrey spaces and weak Morrey spaces

In this study, we investigate the boundedness of composition operators acting on Morrey spaces and weak Morrey spaces. The primary aim of this study is to investigate a necessary and sufficient condition on the boundedness of the composition operator induced by a diffeomorphism on Morrey spaces. In particular, detailed information is derived from the boundedness, i.e., the bi-Lipschitz continuity of the mapping that induces the composition operator follows from the continuity of the composition mapping. The idea of the proof is to determine the Morrey norm of the characteristic functions, and employ a specific function composed of a characteristic function. As the specific function belongs to Morrey spaces but not to Lebesgue spaces, the result reveals a new phenomenon not observed in Lebesgue spaces. Subsequently, we prove the boundedness of the composition operator induced by a mapping that satisfies a suitable volume estimate on general weak-type spaces generated by normed spaces. As a corollary, a necessary and sufficient condition for the boundedness of the composition operator on weak Morrey spaces is provided.

math.FA↗