arXiv subjects

José Mourão

Publications and source records attributed to José Mourão.

14 recordsLinked to original sources

arXiv2025

A Teacher-Student Perspective on the Dynamics of Learning Near the Optimal Point

Near an optimal learning point of a neural network, the learning performance of gradient descent dynamics is dictated by the Hessian matrix of the loss function with respect to the network parameters. We characterize the Hessian eigenspectrum for some classes of teacher-student problems, when the teacher and student networks have matching weights, showing that the smaller eigenvalues of the Hessian determine long-time learning performance. For linear networks, we analytically establish that for large networks the spectrum asymptotically follows a convolution of a scaled chi-square distribution with a scaled Marchenko-Pastur distribution. We numerically analyse the Hessian spectrum for polynomial and other non-linear networks. Furthermore, we show that the rank of the Hessian matrix can be seen as an effective number of parameters for networks using polynomial activation functions. For a generic non-linear activation function, such as the error function, we empirically observe that the Hessian matrix is always full rank.

stat.ML↗

arXiv2025

A ZeNN architecture to avoid the Gaussian trap

We propose a new simple architecture, Zeta Neural Networks (ZeNNs), in order to overcome several shortcomings of standard multi-layer perceptrons (MLPs). Namely, in the large width limit, MLPs are non-parametric, they do not have a well-defined pointwise limit, they lose non-Gaussian attributes and become unable to perform feature learning; moreover, finite width MLPs perform poorly in learning high frequencies. The new ZeNN architecture is inspired by three simple principles from harmonic analysis: i) Enumerate the perceptons and introduce a non-learnable weight to enforce convergence; ii) Introduce a scaling (or frequency) factor; iii) Choose activation functions that lead to near orthogonal systems. We will show that these ideas allow us to fix the referred shortcomings of MLPs. In fact, in the infinite width limit, ZeNNs converge pointwise, they exhibit a rich asymptotic structure beyond Gaussianity, and perform feature learning. Moreover, when appropriate activation functions are chosen, (finite width) ZeNNs excel at learning high-frequency features of functions with low dimensional domains.

cs.LG↗

arXiv2024

The Positivity of the Neural Tangent Kernel

The Neural Tangent Kernel (NTK) has emerged as a fundamental concept in the study of wide Neural Networks. In particular, it is known that the positivity of the NTK is directly related to the memorization capacity of sufficiently wide networks, i.e., to the possibility of reaching zero loss in training, via gradient descent. Here we will improve on previous works and obtain a sharp result concerning the positivity of the NTK of feedforward networks of any depth. More precisely, we will show that, for any non-polynomial activation function, the NTK is strictly positive definite. Our results are based on a novel characterization of polynomial functions which is of independent interest.

cs.LG↗

arXiv2023

Wide neural networks: From non-gaussian random fields at initialization to the NTK geometry of training

Recent developments in applications of artificial neural networks with over $n=10^{14}$ parameters make it extremely important to study the large $n$ behaviour of such networks. Most works studying wide neural networks have focused on the infinite width $n \to +\infty$ limit of such networks and have shown that, at initialization, they correspond to Gaussian processes. In this work we will study their behavior for large, but finite $n$. Our main contributions are the following: (1) The computation of the corrections to Gaussianity in terms of an asymptotic series in $n^{-\frac{1}{2}}$. The coefficients in this expansion are determined by the statistics of parameter initialization and by the activation function. (2) Controlling the evolution of the outputs of finite width $n$ networks, during training, by computing deviations from the limiting infinite width case (in which the network evolves through a linear flow). This improves previous estimates and yields sharper decay rates for the (finite width) NTK in terms of $n$, valid during the entire training procedure. As a corollary, we also prove that, with arbitrarily high probability, the training of sufficiently wide neural networks converges to a global minimum of the corresponding quadratic loss function. (3) Estimating how the deviations from Gaussianity evolve with training in terms of $n$. In particular, using a certain metric in the space of measures we find that, along training, the resulting measure is within $n^{-\frac{1}{2}}(\log n)^{1+}$ of the time dependent Gaussian process corresponding to the infinite width network (which is explicitly given by precomposing the initial Gaussian process with the linear flow corresponding to training in the infinite width limit).

cs.LG↗

arXiv2019

A new approximation method for geodesics on the space of Kähler metrics using complexified symplectomorphisms and Gröbner Lie series

It has been shown that the Cauchy problem for geodesics in the space of Kähler metrics with a fixed cohomology class on a compact complex manifold $M$ can be effectively reduced to the problem of finding the flow of a related hamiltonian vector field $X_H$, followed by analytic continuation of the time to complex time. This opens the possibility of expressing the geodesic $ω_t$ in terms of Gröbner Lie series of the form $\exp(\sqrt{-1} \, tX_H)(f)$, for local holomorphic functions $f$. The main goal of this paper is to use truncated Lie series as a new way of constructing approximate solutions to the geodesic equation. For the case of an elliptic curve and $H$ a certain Morse function squared, we approximate the relevant Lie series by their first twelve terms, calculated with the help of Mathematica. This leads to approximate geodesics which hit the boundary of the space of Kähler metrics in finite geodesic time. For quantum mechanical applications, one is interested also on the non-Kähler polarizations that one obtains by crossing the boundary of the space of Kähler structures. Properties of the approximate geodesics and its extensions are also studied using Mathematica.

math.SG↗

arXiv2019

Holomorphic fractional Fourier transforms

The Fractional Fourier Transform (FrFT) has widespread applications in areas like signal analysis, Fourier optics, diffraction theory, etc. The Holomorphic Fractional Fourier Transform (HFrFT) proposed in the present paper may be used in the same wide range of applications with improved properties. The HFrFT of signals spans a one-parameter family of (essentially) holomorphic functions, where the parameter takes values in the bounded interval $t\in (0,π/2)$. At the boundary values of the parameter, one obtains the original signal at $t=0$ and its Fourier transform at the other end of the interval $t=π/2$. If the initial signal is $L^2 $, then, for an appropriate choice of inner product that will be detailed below, the transform is unitary for all values of the parameter in the interval. This transform provides a heat kernel smoothening of the signals while preserving unitarity for $L^2$-signals and continuously interpolating between the original signal and its Fourier transform.

math-ph↗

arXiv2019

Segal-Bargmann transforms from hyperbolic Hamiltonians

We consider the imaginary time flow of a quadratic hyperbolic Hamiltonian on the symplectic plane, apply it to the Schrödinger polarization and study the corresponding evolution of polarized sections. The flow is periodic in imaginary time and the evolution of polarized sections has interesting features. On the time intervals for which the polarization is real or Kähler, the half--form corrected time evolution of polarized sections is given by unitary operators which turn out to be equivalent to the classical Segal-Bargmann transforms (which are usually associated to the quadratic elliptic Hamiltonian $H=\frac12 p^2$ and to the heat operator). At the right endpoint of these intervals, the evolution of polarized sections is given by the Fourier transform from the Schrödinger to the momentum representation. In the complementary intervals of imaginary time, the polarizations are anti--Kähler and the Hilbert space of polarized sections collapses to ${\mathcal H}= \{0\}$. Hyperbolic quadratic Hamiltonians thus give rise to a new factorization of the Segal-Bargmann transform, which is very different from the usual one, where one first applies a bounded contraction operator (the heat kernel operator), mapping $L^2$--states to real analytic functions with unique analytic continuation, and then one applies analytic continuation. In the factorization induced by an hyperbolic complexifier, both factors are unbounded operators but their composition is, in the Kähler or real sectors, unitary. In another paper [KMNT], we explore the application of the above family of unitary transforms to the definition of new holomorphic fractional Fourier transforms.

math-ph↗

arXiv2016

Clifford Coherent State Transforms on Spheres

We introduce a one-parameter family of transforms, $U^t_{(m)}$, $t>0$, from the Hilbert space of Clifford algebra valued square integrable functions on the $m$--dimensional sphere, $L^2(S^{m},dσ_{m})\otimes \mathbb{C}_{m+1}$, to the Hilbert spaces, ${\mathcal M}L^2(\mathbb{R}^{m+1} \setminus \{0\},dμ_t)$, of monogenic functions on $\mathbb{R}^{m+1}\setminus \{0\}$ which are square integrable with respect to appropriate measures, $dμ_t$. We prove that these transforms are unitary isomorphisms of the Hilbert spaces and are extensions of the Segal-Bargman coherent state transform, $U_{(1)} : L^2(S^{1},dσ_{1}) \longrightarrow {\mathcal H}L^2({\mathbb{C} \setminus \{0\}},dμ)$, to higher dimensional spheres in the context of Clifford analysis. In Clifford analysis it is natural to replace the analytic continuation from $S^m$ to $S^m_{\mathbb{C}}$ as in \cite{Ha1, St, HM} by the Cauchy--Kowalewski extension from $S^m$ to $\mathbb{R}^{m+1}\setminus \{0\}$. One then obtains a unitary isomorphism from an $L^2$--Hilbert space to an Hilbert space of solutions of the Dirac equation, that is to a Hilbert space of monogenic functions.

math.FA↗

arXiv2016

Extending coherent state transforms to Clifford analysis

We introduce two extensions of the Segal-Bargmann coherent state transform from $L^2({\mathbb R},dx)$ to Hilbert spaces of slice monogenic and axial monogenic functions and study their properties. These two transforms are related by the dual Radon transform. Representation theoretic and quantum mechanical aspects of the new representations are studied.

math-ph↗

arXiv2016

Coherent State Transforms and the Weyl Equation in Clifford Analysis

We study a transform, inspired by coherent state transforms, from the Hilbert space of Clifford algebra valued square integrable functions $L^2({\mathbb R}^m,dx)\otimes {\mathbb C}_{m}$ to a Hilbert space of solutions of the Weyl equation on ${\mathbb R}^{m+1}= {\mathbb R} \times {\mathbb R}^m$, namely to the Hilbert space ${\mathcal M}L^2({\mathbb R}^{m+1},dμ)$ of ${\mathbb C}_m$-valued monogenic functions on ${\mathbb R}^{m+1}$ which are $L^2$ with respect to an appropriate measure $dμ$. We prove that this transform is a unitary isomorphism of Hilbert spaces and that it is therefore an analog of the Segal-Bargmann transform for Clifford analysis. As a corollary we obtain an orthonormal basis of monogenic functions on ${\mathbb R}^{m+1}$. We also study the case when ${\mathbb R}^m$ is replaced by the $m$-torus ${\mathbb T}^m.$ Quantum mechanically, this extension establishes the unitary equivalence of the Schrödinger representation on $M$, for $M={\mathbb R}^m$ and $M={\mathbb T}^m$, with a representation on the Hilbert space ${\mathcal M}L^2({\mathbb R} \times M,dμ)$ of solutions of the Weyl equation on the space-time ${\mathbb R}\times M$.

math.FA↗

arXiv2011

Quantization of some moduli spaces of parabolic vector bundles on CP^1

We address quantization of the natural symplectic structure on a moduli space of parabolic vector bundles of parabolic degree zero over $\mathbf{CP}^1$ with four parabolic points and parabolic weights in {0,1/2}. Identifying such parabolic bundles as vector bundles on an elliptic curve, we obtain explicit expressions for the corresponding non-abelian theta functions. These non-abelian theta functions are described in terms of certain naturally defined distributions on the compact group SU(2).

math.AG↗

arXiv1997

The Status of Diffeomorphism Superselection in Euclidean 2+1 Gravity

This work addresses a specific technical question of relevance to canonical quantization of gravity using the so-called new variables and loop-based techniques of Ashtekar, Rovelli, and Smolin. In particular, certain `superselection laws' that arise in current applications of these techniques to solving the diffeomorphism constraint are considered. Their status is elucidated by studying an analogous system: 2+1 Euclidean gravity. For that system, these superselection laws are shown to be spurious. This, however, is only a technical difficulty. The usual quantum theory may still be obtained from a loop representation and the technique known as `Refined Algebraic Quantization.'

gr-qc↗

arXiv1996

Closed formula for Wilson loops for $SU(N)$ Quantum Yang-Mills Theory in two dimensions

A closed expression of the Euclidean Wilson-loop functionals is derived for pure Yang-Mills continuum theories with gauge groups $SU(N)$ and $U(1)$ and space-time topologies $\Rl^1\times\Rl^1$ and $\Rl^1\times S^1$. (For the $U(1)$ theory, we also consider the $S^1\times S^1$ topology.) The treatment is rigorous, manifestly gauge invariant, manifestly invariant under area preserving diffeomorphisms and handles all (piecewise analytic) loops in one stroke. Equivalence between the resulting Euclidean theory and and the Hamiltonian framework is then established. Finally, an extension of the Osterwalder-Schrader axioms for gauge theories is proposed. These axioms are satisfied for the present model.

hep-th↗

arXiv1994

Coherent State Transforms for Spaces of Connections

The Segal-Bargmann transform plays an important role in quantum theories of linear fields. Recently, Hall obtained a non-linear analog of this transform for quantum mechanics on Lie groups. Given a compact, connected Lie group $G$ with its normalized Haar measure $μ_H$, the Hall transform is an isometric isomorphism from $L^2(G, μ_H)$ to ${\cal H}(G^{\Co})\cap L^2(G^{\Co}, ν)$, where $G^{\Co}$ the complexification of $G$, ${\cal H}(G^{\Co})$ the space of holomorphic functions on $G^{\Co}$, and $ν$ an appropriate heat-kernel measure on $G^{\Co}$. We extend the Hall transform to the infinite dimensional context of non-Abelian gauge theories by replacing the Lie group $G$ by (a certain extension of) the space ${\cal A}/{\cal G}$ of connections modulo gauge transformations. The resulting ``coherent state transform'' provides a holomorphic representation of the holonomy $C^\star$ algebra of real gauge fields. This representation is expected to play a key role in a non-perturbative, canonical approach to quantum gravity in 4-dimensions.

gr-qc↗