arXiv subjects

Len Spek

Publications and source records attributed to Len Spek.

5 recordsLinked to original sources

arXiv2025

Deep Networks are Reproducing Kernel Chains

Identifying an appropriate function space for deep neural networks remains a key open question. While shallow neural networks are naturally associated with Reproducing Kernel Banach Spaces (RKBS), deep networks present unique challenges. In this work, we extend RKBS to chain RKBS (cRKBS), a new framework that composes kernels rather than functions, preserving the desirable properties of RKBS. We prove that any deep neural network function is a neural cRKBS function, and conversely, any neural cRKBS function defined on a finite dataset corresponds to a deep neural network. This approach provides a sparse solution to the empirical risk minimization problem, requiring no more than $N$ neurons per layer, where $N$ is the number of data points.

cs.LG↗

arXiv2024

Embeddings between Barron spaces with higher order activation functions

The approximation properties of infinitely wide shallow neural networks heavily depend on the choice of the activation function. To understand this influence, we study embeddings between Barron spaces with different activation functions. These embeddings are proven by providing push-forward maps on the measures $μ$ used to represent functions $f$. An activation function of particular interest is the rectified power unit ($\operatorname{RePU}$) given by $\operatorname{RePU}_s(x)=\max(0,x)^s$. For many commonly used activation functions, the well-known Taylor remainder theorem can be used to construct a push-forward map, which allows us to prove the embedding of the associated Barron space into a Barron space with a $\operatorname{RePU}$ as activation function. Moreover, the Barron spaces associated with the $\operatorname{RePU}_s$ have a hierarchical structure similar to the Sobolev spaces $H^m$.

stat.ML↗

arXiv2023

Duality for Neural Networks through Reproducing Kernel Banach Spaces

Reproducing Kernel Hilbert spaces (RKHS) have been a very successful tool in various areas of machine learning. Recently, Barron spaces have been used to prove bounds on the generalisation error for neural networks. Unfortunately, Barron spaces cannot be understood in terms of RKHS due to the strong nonlinear coupling of the weights. This can be solved by using the more general Reproducing Kernel Banach spaces (RKBS). We show that these Barron spaces belong to a class of integral RKBS. This class can also be understood as an infinite union of RKHS spaces. Furthermore, we show that the dual space of such RKBSs, is again an RKBS where the roles of the data and parameters are interchanged, forming an adjoint pair of RKBSs including a reproducing kernel. This allows us to construct the saddle point problem for neural networks, which can be used in the whole field of primal-dual optimisation.

math.FA↗

arXiv2023

Bifurcations of Neural Fields on the Sphere

A common model to study pattern formation in large groups of neurons is the neural field. We investigate a neural field with excitatory and inhibitory neurons, like Wilson and Cowan (1972), with transmission delays and gap junctions. We build on the work of Visser et al. (2017) by investigating pattern formation in these models on the sphere. Specifically, we investigate how different amounts of gap-junctions, modelled by a diffusion term, influence the behaviour of the neural field. We look in detail at the periodic orbits which are generated by Hopf bifurcation in the presence of spherical symmetry. For this end, we derive general formulas to compute the normal form coefficients of these bifurcations up to third order and predict the stability of the resulting branches, and formulate a novel numerical method to solve delay equations on the sphere.

math.DS↗

arXiv2021

Neural Field Models with Transmission Delays and Diffusion

A neural field models the large scale behaviour of large groups of neurons. We extend results of van Gils et al. [2013] and Dijkstra et al. [2015] by including a diffusion term into the neural field, which models direct, electrical connections. We extend known and prove new sun-star calculus results for delay equations to be able to include diffusion and explicitly characterise the essential spectrum. For a certain class of connectivity functions in the neural field model, we are able to compute its spectral properties and the first Lyapunov coefficient of a Hopf bifurcation. By examining a numerical example, we find that the addition of diffusion suppresses non-synchronised steady-states, while favouring synchronised oscillatory modes.

math.DS↗