arXiv subjects

Felix Voigtlaender

Publications and source records attributed to Felix Voigtlaender.

49 records · Page 3Linked to original sources

arXiv2020

Topological properties of the set of functions generated by neural networks of fixed size

We analyze the topological properties of the set of functions that can be implemented by neural networks of a fixed size. Surprisingly, this set has many undesirable properties. It is highly non-convex, except possibly for a few exotic activation functions. Moreover, the set is not closed with respect to $L^p$-norms, $0 < p < \infty$, for all practically-used activation functions, and also not closed with respect to the $L^\infty$-norm for all practically-used activation functions except for the ReLU and the parametric ReLU. Finally, the function that maps a family of weights to the function computed by the associated network is not inverse stable for every practically used activation function. In other words, if $f_1, f_2$ are two functions realized by neural networks and if $f_1, f_2$ are close in the sense that $\|f_1 - f_2\|_{L^\infty} \leq \varepsilon$ for $\varepsilon > 0$, it is, regardless of the size of $\varepsilon$, usually not possible to find weights $w_1, w_2$ close together such that each $f_i$ is realized by a neural network with weights $w_i$. Overall, our findings identify potential causes for issues in the training procedure of deep learning such as no guaranteed convergence, explosion of parameters, and slow convergence.

math.GN↗

arXiv2019

Embeddings of decomposition spaces

Many smoothness spaces in harmonic analysis are decomposition spaces. In this paper we ask: Given two decomposition spaces, is there an embedding between the two? A decomposition space $\mathcal{D}(\mathcal{Q}, L^p, Y)$ can be described using : a covering $\mathcal{Q}=(Q_{i})_{i\in I}$ of the frequency domain, an exponent $p$ and a sequence space $Y\subset\mathbb{C}^{I}$. Given these, the decomp. space norm of a distribution $g$ is $\| g\| _{\mathcal{D}(\mathcal{Q}, L^p, Y)}=\left\| \left(\left\| \mathcal{F}^{-1}\left(φ_{i}\widehat{g}\right)\right\| _{L^{p}}\right)_{i\in I}\right\| _{Y}$, where $(φ_{i})_{i\in I}$ is a suitable partition of unity for $\mathcal{Q}$. We establish readily verifiable criteria which ensure an embedding $\mathcal{D}(\mathcal{Q}, L^{p_1}, Y)\hookrightarrow\mathcal{D}(\mathcal{P}, L^{p_2}, Z)$, mostly concentrating on the case, $Y=\ell_{w}^{q_{1}}(I)$ and $Z=\ell_{v}^{q_{2}}(J)$. The relevant sufficient conditions are $p_{1}\leq p_{2}$, and finiteness of a norm of the form \[ \left\| \left(\left\| (α_{i}\,β_j \cdot v_{j}/w_{i})_{i\in I_{j}}\right\| _{\ell^{t}}\right)_{j\in J}\right\| _{\ell^{s}}<\infty, \] where the \[ I_{j}=\{ i\in I : Q_{i}\cap P_{j}\neq\emptyset\} \qquad\text{ for }j\in J \] are defined in terms of the two coverings $\mathcal{Q}=(Q_{i})_{i\in I}$ and $\mathcal{P}=(P_{j})_{j\in J}$. We also show that these criteria are sharp: For almost arbitrary coverings and certain ranges of $p_{1},p_{2}$, our criteria yield a complete characterization. The same holds for arbitrary values of $p_{1},p_{2}$ under more strict assumptions on the coverings. We illustrate the resulting theory by applications to $α$-modulation and Besov spaces. All known embedding results for these spaces are special cases of our approach; often, we improve considerably upon the state of the art.

math.FA↗

arXiv2019

Design and properties of wave packet smoothness spaces

We introduce a family of quasi-Banach spaces - which we call wave packet smoothness spaces - that includes those function spaces which can be characterised by the sparsity of their expansions in Gabor frames, wave atoms, and many other frame constructions. We construct Banach frames for and atomic decompositions of the wave packet smoothness spaces and study their embeddings in each other and in a few more classical function spaces such as Besov and Sobolev spaces.

math.FA↗

arXiv2019

Approximation in $L^p(μ)$ with deep ReLU neural networks

We discuss the expressive power of neural networks which use the non-smooth ReLU activation function $\varrho(x) = \max\{0,x\}$ by analyzing the approximation theoretic properties of such networks. The existing results mainly fall into two categories: approximation using ReLU networks with a fixed depth, or using ReLU networks whose depth increases with the approximation accuracy. After reviewing these findings, we show that the results concerning networks with fixed depth--- which up to now only consider approximation in $L^p(λ)$ for the Lebesgue measure $λ$--- can be generalized to approximation in $L^p(μ)$, for any finite Borel measure $μ$. In particular, the generalized results apply in the usual setting of statistical learning theory, where one is interested in approximation in $L^2(\mathbb{P})$, with the probability measure $\mathbb{P}$ describing the distribution of the data.

math.FA↗

arXiv2018

Learning sparse representations on the sphere

Many representation systems on the sphere have been proposed in the past, such as spherical harmonics, wavelets, or curvelets. Each of these data representations is designed to extract a specific set of features, and choosing the best fixed representation system for a given scientific application is challenging. In this paper, we show that we can learn directly a representation system from given data on the sphere. We propose two new adaptive approaches: the first is a (potentially multi-scale) patch-based dictionary learning approach, and the second consists in selecting a representation among a parametrized family of representations, the α-shearlets. We investigate their relative performance to represent and denoise complex structures on different astrophysical data sets on the sphere.

astro-ph.IM↗

arXiv2018

On the Atomic Decomposition of Coorbit Spaces with Non-Integrable Kernel

This paper ist concerned with recent progress in the context of coorbit space theory. Based on a square integrable group representation, the coorbit theory provides new families of associated smoothness spaces, where the smoothness of a function is measured by the decay of the associated voice transform. Moreover, by discretizing the representation, atomic decomposi- tions and Banach frames can be constructed. Usually, the whole machinery works well if the associated reproducing kernel is integrable with respect to a weighted Haar measure on the group. In recent studies, it has turned out that to some extent coorbit spaces can still be established if this condition is violated. In this paper, we clarify in which sense atomic decompositions and Banach frames for these generalized coorbit spaces can be obtained.

math.FA↗

arXiv2018

Optimal approximation of piecewise smooth functions using deep ReLU neural networks

We study the necessary and sufficient complexity of ReLU neural networks---in terms of depth and number of weights---which is required for approximating classifier functions in $L^2$. As a model class, we consider the set $\mathcal{E}^β(\mathbb R^d)$ of possibly discontinuous piecewise $C^β$ functions $f : [-1/2, 1/2]^d \to \mathbb R$, where the different smooth regions of $f$ are separated by $C^β$ hypersurfaces. For dimension $d \geq 2$, regularity $β> 0$, and accuracy $\varepsilon > 0$, we construct artificial neural networks with ReLU activation function that approximate functions from $\mathcal{E}^β(\mathbb R^d)$ up to $L^2$ error of $\varepsilon$. The constructed networks have a fixed number of layers, depending only on $d$ and $β$, and they have $O(\varepsilon^{-2(d-1)/β})$ many nonzero weights, which we prove to be optimal. In addition to the optimality in terms of the number of weights, we show that in order to achieve the optimal approximation rate, one needs ReLU networks of a certain depth. Precisely, for piecewise $C^β(\mathbb R^d)$ functions, this minimal depth is given---up to a multiplicative constant---by $β/d$. Up to a log factor, our constructed networks match this bound. This partly explains the benefits of depth for ReLU networks by showing that deep networks are necessary to achieve efficient approximation of (piecewise) smooth functions. Finally, we analyze approximation in high-dimensional spaces where the function $f$ to be approximated can be factorized into a smooth dimension reducing feature map $τ$ and classifier function $g$---defined on a low-dimensional feature space---as $f = g \circ τ$. We show that in this case the approximation rate depends only on the dimension of the feature space and not the input dimension.

math.FA↗

arXiv2017

Analysis vs. synthesis sparsity for $α$-shearlets

There are two notions of sparsity associated to a frame $Ψ=(ψ_i)_{i\in I}$: Analysis sparsity of $f$ means that the analysis coefficients $(\langle f,ψ_i\rangle)_i$ are sparse, while synthesis sparsity means that $f=\sum_i c_iψ_i$ with sparse coefficients $(c_i)_i$. Here, sparsity of $c=(c_i)_i$ means $c\in\ell^p(I)$ for a given $p<2$. We show that both notions of sparsity coincide if $Ψ={\rm SH}(φ,ψ;δ)$ is a discrete (cone-adapted) shearlet frame with 'nice' generators $φ,ψ$ and fine enough sampling density $δ>0$. The required 'niceness' is explicitly quantified in terms of Fourier-decay and vanishing moment conditions. Precisely, we show that suitable shearlet systems simultaneously provide Banach frames and atomic decompositions for the shearlet smoothness spaces $\mathscr{S}_s^{p,q}$ introduced by Labate et al. Hence, membership in $\mathscr{S}_s^{p,q}$ is simultaneously equivalent to analysis sparsity and to synthesis sparsity w.r.t. the shearlet frame. As an application, we prove that shearlets yield (almost) optimal approximation rates for cartoon-like functions $f$: If $ε>0$, then $\Vert f-f_N\Vert_{L^2}\lesssim N^{-(1-ε)}$, where $f_N$ is a linear combination of N shearlets. This might appear to be well-known, but the existing proofs only establish this approximation rate w.r.t. the dual $\tildeΨ$ of $Ψ$, not w.r.t. $Ψ$ itself. This is not completely satisfying, since the properties of $\tildeΨ$ (decay, smoothness, etc.) are largely unknown. We also consider $α$-shearlet systems. For these, the shearlet smoothness spaces have to be replaced by $α$-shearlet smoothness spaces. We completely characterize the embeddings between these spaces, allowing us to decide whether sparsity w.r.t. $α_1$-shearlets implies sparsity w.r.t. $α_2$-shearlets.

math.FA↗

arXiv2016

Structured, compactly supported Banach frame decompositions of decomposition spaces

$\newcommand{mc}[1]{\mathcal{#1}}$ $\newcommand{D}{\mc{D}(\mc{Q},L^p,\ell_w^q)}$ We present a framework for the construction of structured, possibly compactly supported Banach frames and atomic decompositions for decomposition spaces. Such a space $\D$ is defined using a frequency covering $\mc{Q}=(Q_i)_{i\in I}$: If $(φ_i)_{i}$ is a suitable partition of unity subordinate to $\mc{Q}$, then $\Vert g\Vert_{\D}:=\left\Vert\left(\Vert\mc{F}^{-1}(φ_i\hat{g})\Vert_{L^p}\right)_{i}\right\Vert_{\ell_w^q}$. We assume $\mc{Q}=(T_iQ+b_i)_{i}$, with $T_i\in{\rm GL}(\Bbb{R}^d),b_i\in\Bbb{R}^d$. Given a prototype $γ$, we consider the system \[Ψ_{c}=(L_{c\cdot T_i^{-T}k}γ^{[i]})_{i\in I,k\in\Bbb{Z}^d}\text{ with }γ^{[i]}=|\det T_i|^{1/2}\, M_{b_i}(γ\circ T_i^T),\] with translation $L_x$ and modulation $M_ξ$. We provide verifiable conditions on $γ$ under which $Ψ_c$ forms a Banach frame or an atomic decomposition for $\D$, for small enough sampling density $c>0$. Our theory allows compactly supported prototypes and applies for arbitrary $p,q\in(0,\infty]$. Often, $Ψ_c$ is both a Banach frame and an atomic decomposition, so that analysis sparsity is equivalent to synthesis sparsity, i.e. the analysis coefficients $(\langle f,L_{c\cdot T_i^{-T}k}γ^{[i]}\rangle)_{i,k}$ lie in $\ell^p$ iff $f$ belongs to a certain decomposition space, iff $f=\sum_{i,k}c_k^{(i)}\cdot L_{c\cdot T_i^{-T}k}γ^{[i]}$ with $(c_k^{(i)})_{i,k}\in\ell^p$. This is convenient if only analysis sparsity is known to hold: Generally, this only yields synthesis sparsity w.r.t. the dual frame, about which often only little is known. But our theory yields synthesis sparsity w.r.t. the well-understood primal frame. In particular, our theory applies to $α$-modulation spaces and inhom. Besov spaces. It also applies to shearlet frames, as we show in a companion paper.

math.FA↗

arXiv2016

From Frazier-Jawerth characterizations of Besov spaces to Wavelets and Decomposition spaces

This article describes how the ideas promoted by the fundamental papers published by M. Frazier and B. Jawerth in the eighties have influenced subsequent developments related to the theory of atomic decompositions and Banach frames for function spaces such as the modulation spaces and Besov-Triebel-Lizorkin spaces. Both of these classes of spaces arise as special cases of two different, general constructions of function spaces: coorbit spaces and decomposition spaces. Coorbit spaces are defined by imposing certain decay conditions on the so-called voice transform of the function/distribution under consideration. As a concrete example, one might think of the wavelet transform, leading to the theory of Besov-Triebel-Lizorkin spaces. Decomposition spaces, on the other hand, are defined using certain decompositions in the Fourier domain. For Besov-Triebel-Lizorkin spaces, one uses a dyadic decomposition, while a uniform decomposition yields modulation spaces. Only recently, the second author has established a fruitful connection between modern variants of wavelet theory with respect to general dilation groups (which can be treated in the context of coorbit theory) and a particular family of decomposition spaces. In this way, optimal inclusion results and invariance properties for a variety of smoothness spaces can be established. We will present an outline of these connections and comment on the basic results arising in this context.

math.FA↗

arXiv2016

Embeddings of Decomposition Spaces into Sobolev and BV Spaces

In the present paper, we investigate whether an embedding of a decomposition space $\mathcal{D}\left(\mathcal{Q},L^{p},Y\right)$ into a given Sobolev space $W^{k,q}(\mathbb{R}^{d})$ exists. As special cases, this includes embeddings into Sobolev spaces of (homogeneous and inhomogeneous) Besov spaces, ($α$)-modulation spaces, shearlet smoothness spaces and also of a large class of wavelet coorbit spaces, in particular of shearlet-type coorbit spaces. Precisely, we will show that under extremely mild assumptions on the covering $\mathcal{Q}=\left(Q_{i}\right)_{i\in I}$, we have $\mathcal{D}\left(\mathcal{Q},L^{p},Y\right)\hookrightarrow W^{k,q}(\mathbb{R}^{d})$ as soon as $p\leq q$ and $Y\hookrightarrow\ell_{u^{\left(k,p,q\right)}}^{q^{\triangledown}}\left(I\right)$ hold. Here, $q^{\triangledown}=\min\left\{ q,q'\right\} $ and the weight $u^{\left(k,p,q\right)}$ can be easily computed, only based on the covering $\mathcal{Q}$ and on the parameters $k,p,q$. Conversely, a necessary condition for existence of the embedding is that $p\leq q$ and $Y\cap\ell_{0}\left(I\right)\hookrightarrow\ell_{u^{\left(k,p,q\right)}}^{q}\left(I\right)$ hold, where $\ell_{0}\left(I\right)$ denotes the space of finitely supported sequences on $I$. All in all, for the range $q \in (0,2]\cup\{\infty\}$, we obtain a complete characterization of existence of the embedding in terms of readily verifiable criteria. We can also completely characterize existence of an embedding of a decomposition space into a BV space.

math.FA↗

arXiv2014

Resolution of the wavefront set using general continuous wavelet transforms

We consider the problem of characterizing the wavefront set of a tempered distribution $u\in\mathcal{S}'(\mathbb{R}^{d})$ in terms of its continuous wavelet transform, where the latter is defined with respect to a suitably chosen dilation group $H\subset{\rm GL}(\mathbb{R}^{d})$. In this paper we develop a comprehensive and unified approach that allows to establish characterizations of the wavefront set in terms of rapid coefficient decay, for a large variety of dilation groups. For this purpose, we introduce two technical conditions on the dual action of the group $H$, called microlocal admissibilty and (weak) cone approximation property. Essentially, microlocal admissibilty sets up a systematical relationship between the scales in a wavelet dilated by $h\in H$ on one side, and the matrix norm of $h$ on the other side. The (weak) cone approximation property describes the ability of the wavelet system to adapt its frequency-side localization to arbitrary frequency cones. Together, microlocal admissibility and the weak cone approximation property allow the characterization of points in the wavefront set using multiple wavelets. Replacing the weak cone approximation by its stronger counterpart gives access to single wavelet characterizations. We illustrate the scope of our results by discussing -- in any dimension $d\ge2$ -- the similitude, diagonal and shearlet dilation groups, for which we verify the pertinent conditions. As a result, similitude and diagonal groups can be employed for multiple wavelet characterizations, whereas for the shearlet groups a single wavelet suffices. In particular, the shearlet characterization (previously only established for $d=2$) holds in arbitrary dimensions.

math.FA↗

arXiv2014

Wavelet Coorbit Spaces viewed as Decomposition Spaces

In this paper we show that the Fourier transform induces an isomorphism between the coorbit spaces defined by Feichtinger and Gröchenig of the mixed, weighted Lebesgue spaces $L_{v}^{p,q}$ with respect to the quasi-regular representation of a semi-direct product $\mathbb{R}^{d}\rtimes H$ with suitably chosen dilation group $H$, and certain decomposition spaces $\mathcal{D}\left(\mathcal{Q},L^{p},\ell_{u}^{q}\right)$ (essentially as introduced by Feichtinger and Gröbner), where the localized ,,parts`` of a function are measured in the $\mathcal{F}L^{p}$-norm. This equivalence is useful in several ways: It provides access to a Fourier-analytic understanding of wavelet coorbit spaces, and it allows to discuss coorbit spaces associated to different dilation groups in a common framework. As an illustration of these points, we include a short discussion of dilation invariance properties of coorbit spaces associated to different types of dilation groups.

math.FA↗