Searcharxiv⌕ Search

arXiv subjects

Stephan Wojtowytsch

Publications and source records attributed to Stephan Wojtowytsch.

At least 37 records · Page 2Linked to original sources
arXiv2021

Some observations on high-dimensional partial differential equations with Barron data

We use explicit representation formulas to show that solutions to certain partial differential equations lie in Barron spaces or multilayer spaces if the PDE data lie in such function spaces. Consequently, these solutions can be represented efficiently using artificial neural networks, even in high dimension. Conversely, we present examples in which the solution fails to lie in the function space associated to a neural network under consideration.

math.AP↗
arXiv2021

On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers

A recent numerical study observed that neural network classifiers enjoy a large degree of symmetry in the penultimate layer. Namely, if $h(x) = Af(x) +b$ where $A$ is a linear map and $f$ is the output of the penultimate layer of the network (after activation), then all data points $x_{i, 1}, \dots, x_{i, N_i}$ in a class $C_i$ are mapped to a single point $y_i$ by $f$ and the points $y_i$ are located at the vertices of a regular $k-1$-dimensional standard simplex in a high-dimensional Euclidean space. We explain this observation analytically in toy models for highly expressive deep neural networks. In complementary examples, we demonstrate rigorously that even the final output of the classifier $h$ is not uniform over data samples from a class $C_i$ if $h$ is a shallow network (or if the deeper layers do not bring the data samples into a convenient geometric configuration).

cs.LG↗
arXiv2020

Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't

The purpose of this article is to review the achievements made in the last few years towards the understanding of the reasons behind the success and subtleties of neural network-based machine learning. In the tradition of good old applied mathematics, we will not only give attention to rigorous mathematical results, but also the insight we have gained from careful numerical experiments as well as the analysis of simplified models. Along the way, we also list the open problems which we believe to be the most important topics for further study. This is not a complete overview over this quickly moving field, but we hope to provide a perspective which may be helpful especially to new researchers in the area.

cs.LG↗
arXiv2020

Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernels

We establish a scale separation of Kolmogorov width type between subspaces of a given Banach space under the condition that a sequence of linear maps converges much faster on one of the subspaces. The general technique is then applied to show that reproducing kernel Hilbert spaces are poor $L^2$-approximators for the class of two-layer neural networks in high dimension, and that multi-layer networks with small path norm are poor approximators for certain Lipschitz functions, also in the $L^2$-topology.

math.FA↗
arXiv2020

Confined elasticae and the buckling of cylindrical shells

For curves of prescribed length embedded into the unit disc in two dimensions, we obtain scaling results for the minimal elastic energy as the length just exceeds $2π$ and in the large length limit. In the small excess length case, we prove convergence to a fourth order obstacle type problem with integral constraint on the real line which we then solve. From the solution, we obtain the first order coefficient $Θ\approx 37$ in the energy expansion $2π+ Θδ^{1/3} + o(δ^{1/3})$ when a curve has length $2π+ δ$. We present an application of the scaling result to buckling in two-layer cylindrical shells where we can determine an explicit bifurcation point between compression and buckling in terms of universal constants and material parameters scaling with the thickness of the inner shell.

math.DG↗
arXiv2020

A priori estimates for classification problems using neural networks

We consider binary and multi-class classification problems using hypothesis classes of neural networks. For a given hypothesis class, we use Rademacher complexity estimates and direct approximation theorems to obtain a priori error estimates for regularized loss functionals.

stat.ML↗
arXiv2020

On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics

We develop Banach spaces for ReLU neural networks of finite depth $L$ and infinite width. The spaces contain all finite fully connected $L$-layer networks and their $L^2$-limiting objects under bounds on the natural path-norm. Under this norm, the unit ball in the space for $L$-layer networks has low Rademacher complexity and thus favorable generalization properties. Functions in these spaces can be approximated by multi-layer neural networks with dimension-independent convergence rates. The key to this work is a new way of representing functions in some form of expectations, motivated by multi-layer neural networks. This representation allows us to define a new class of continuous models for machine learning. We show that the gradient flow defined this way is the natural continuous analog of the gradient descent dynamics for the associated multi-layer neural networks. We show that the path-norm increases at most polynomially under this continuous gradient flow dynamics.

stat.ML↗
arXiv2020

Connected Coulomb Columns: Analysis and Numerics

We consider a version of Gamow's liquid drop model with a short range attractive perimeter-penalizing potential and a long-range Coulomb interaction of a uniformly charged mass in $\R^3$. Here we constrain ourselves to minimizing among the class of shapes that are columnar, i.e., constant in one spatial direction. Using the standard perimeter in the energy would lead to non-existence for any prescribed cross-sectional area due to the infinite mass in the constant spatial direction. In order to heal this defect we use a connected perimeter instead. We prove existence of minimizers for this connected isoperimetric problem with long-range interaction and study the shapes of minimizers in the small and large cross section regimes. For an intermediate regime we use an Ohta-Kawasaki phase field model with connectedness constraint to study the shapes of minimizers numerically.

math.AP↗
arXiv2020

On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime

We describe a necessary and sufficient condition for the convergence to minimum Bayes risk when training two-layer ReLU-networks by gradient descent in the mean field regime with omni-directional initial parameter distribution. This article extends recent results of Chizat and Bach to ReLU-activated networks and to the situation in which there are no parameters which exactly achieve MBR. The condition does not depend on the initalization of parameters and concerns only the weak convergence of the realization of the neural network, not its parameter distribution.

math.AP↗
arXiv2020

Can Shallow Neural Networks Beat the Curse of Dimensionality? A mean field training perspective

We prove that the gradient descent training of a two-layer neural network on empirical or population risk may not decrease population risk at an order faster than $t^{-4/(d-2)}$ under mean field scaling. Thus gradient descent training for fitting reasonably smooth, but truly high-dimensional data may be subject to the curse of dimensionality. We present numerical evidence that gradient descent training with general Lipschitz target functions becomes slower and slower as the dimension increases, but converges at approximately the same rate in all dimensions when the target function lies in the natural function space for two-layer ReLU networks.

cs.LG↗
arXiv2020

On the motion of curved dislocations in three dimensions: Simplified linearized elasticity

It is shown that in core-radius cutoff regularized simplified elasticity (where the elastic energy depends quadratically on the full displacement gradient rather than its symmetrized version), the force on a dislocation curve by the negative gradient of the elastic energy asymptotically approaches the mean curvature of the curve as the cutoff radius converges to zero. Rigorous error bounds in Hölder spaces are provided. As an application, convergence of dislocations moving by the gradient flow of the elastic energy to dislocations moving by the gradient flow of the arclength functional, when the motion law is given by an $H^1$-type dissipation, and convergence to curve shortening flow in co-dimension $2$ for the usual $L^2$-dissipation is established. In the second scenario, existence and regularity are assumed while the $H^1$-gradient flow is treated in full generality (for short time). The methods developed here are a blueprint for the more physical setting of linearized isotropic elasticity.

math.AP↗
arXiv2018

Keeping it together: a phase field version of path-connectedness and its implementation

We describe the implementation of a topological constraint in finite element simulations of phase field models which ensures path-connectedness of preimages of intervals in the phase field variable. Two main applications of our method are presented. First, a discrete steepest decent of a phase field version of a bending energy with spontaneous curvature and additional surface area penalty is shown, which leads to disconnected surfaces without our topological constraint but connected surfaces with the constraint. The second application is the segmentation of an image into a connected component and its exterior. Numerically, our constraint is treated using a suitable geodesic distance function which is computed using Dijkstra's algorithm.

math.NA↗
arXiv2017

The Effect of Forest Dislocations on the Evolution of a Phase-Field Model for Plastic Slip

We consider the gradient flow evolution of a phase-field model for crystal dislocations in a single slip system in the presence of forest dislocations. The model consists of a Peierls-Nabarro type energy penalizing non-integer slip and elastic stress. Forest dislocations are introduced as a perforation of the domain by small disks where slip is prohibited. The $Γ$-limit of this energy was deduced by Garroni and Müller (2005 and 2006). Our main result shows that the gradient flows of these $Γ$-convergent energy functionals do not approach the gradient flow of the limiting energy. Indeed, the gradient flow dynamics remains a physically reasonable model in the case of non-monotone loading. Our proofs rely on the construction of explicit sub- and super-solutions to a fractional Allen-Cahn equation on a flat torus or in the plane, with Dirichlet data on a union of small discs. The presence of these obstacles leads to an additional friction in the viscous evolution which appears as a stored energy in the $Γ$-limit, but it does not act as a driving force. Extensions to related models with soft pinning and non-viscous evolutions are also discussed. In terms of physics, our results explain how in this phase field model the presence of forest dislocations still allows for plastic as opposed to only elastic deformation.

math.AP↗
arXiv2017

Uniform Regularity and Convergence of Phase-Fields for Willmore's Energy

We investigate the convergence of phase fields for the Willmore problem away from the support of a limiting measure $μ$. For this purpose, we introduce a suitable notion of essentially uniform convergence. This mode of convergence is a natural generalisation of uniform convergence that precisely describes the convergence of phase fields in three dimensions. More in detail, we show that, in three space dimensions, points close to which the phase fields stay bounded away from a pure phase lie either in the support of the limiting mass measure $μ$ or contribute a positive amount to the limiting Willmore energy. Thus there can only be finitely many such points. As an application, we investigate the Hausdorff limit of level sets of sequences of phase fields with bounded energy. We also obtain results on boundedness and $L^p$-convergence of phase fields and convergence from outside the interval between the wells of a double-well potential. For minimisers of suitable energy functionals, we deduce uniform convergence of the phase fields from essentially uniform convergence.

math.AP↗
arXiv2017

On the Boundary Regularity of Phase-Fields for Willmore's Energy

We demonstrate that Radon measures which arise as the limit of the Modica-Mortola measures associated to phase-fields with uniformly bounded diffuse area and Willmore energy may be singular at the boundary of a domain and discuss implications for practical applications. We furthermore give partial regularity results for the phase-fields $u_ε$ at the boundary in terms of boundary conditions and counterexamples without boundary conditions.

math.AP↗
arXiv2016

Phase field models for thin elastic structures with topological constraint

This article is concerned with the problem of minimising the Willmore energy in the class of \emph{connected} surfaces with prescribed area which are confined to a small container. We propose a phase field approximation based on De Giorgi's diffuse Willmore functional to this variational problem. Our main contribution is a penalisation term which ensures connectedness in the sharp interface limit. The penalisation of disconnectedness is based on a geodesic distance chosen to be small between two points that lie on the same connected component of the transition layer of the phase field. We prove that in two dimensions, sequences of phase fields with uniformly bounded diffuse Willmore energy and diffuse area converge uniformly to the zeros of a double-well potential away from the support of a limiting measure. In three dimensions, we show that they converge $\mathcal{H}^1$-almost everywhere on curves. This enables us to show $Γ$-convergence to a sharp interface problem that only allows for connected structures. The results also imply Hausdorff convergence of the level sets in two dimensions and a similar result in three dimensions. We furthermore present numerical evidence of the effectiveness of our model. The implementation relies on a coupling of Dijkstra's algorithm in order to compute the topological penalty to a finite element approach for the Willmore term.

math.AP↗
arXiv2016

Helfrich's Energy and Constrained Minimisation

For every $g\in\mathbb{N}_0$ and $ε>0$, we construct a smooth genus $g$ surface embedded into the unit ball with area $8π$ and Willmore energy smaller than $8π+ ε$. From this we deduce that a minimising sequence for Willmore's energy in the class of genus $g$ surfaces embedded in the unit ball with area $8π$ converges to a doubly covered sphere for all $g\in\mathbb{N}_0$. We obtain the same result for certain Canham-Helfrich energies with $χ_K\leq 0$ without genus constraint and show that Canham-Helfrich energies with $χ_K>0$ are not bounded from below in the class of smooth surfaces with area $S$ embedded into a domain $Ω\Subset \mathbb{R}^3$. Furthermore, we prove that the class of connected surfaces embedded in a domain $Ω\Subset\mathbb{R}^3$ with uniformly bounded Willmore energy and area is compact under varifold convergence.

math.DG↗