arXiv subjects

Fabrizio Pittorino

Publications and source records attributed to Fabrizio Pittorino.

23 records · Page 2Linked to original sources

arXiv2021

Entropic gradient descent algorithms and wide flat minima

The properties of flat minima in the empirical risk landscape of neural networks have been debated for some time. Increasing evidence suggests they possess better generalization capabilities with respect to sharp ones. First, we discuss Gaussian mixture classification models and show analytically that there exist Bayes optimal pointwise estimators which correspond to minimizers belonging to wide flat regions. These estimators can be found by applying maximum flatness algorithms either directly on the classifier (which is norm independent) or on the differentiable loss function used in learning. Next, we extend the analysis to the deep learning scenario by extensive numerical validations. Using two algorithms, Entropy-SGD and Replicated-SGD, that explicitly include in the optimization objective a non-local flatness measure known as local entropy, we consistently improve the generalization error for common architectures (e.g. ResNet, EfficientNet). An easy to compute flatness measure shows a clear correlation with test accuracy.

cs.LG↗

arXiv2021

Order symmetry breaking and broad distribution of events in spiking neural networks with continuous membrane potential

We introduce an exactly integrable version of the well-known leaky integrate-and-fire (LIF) model, with continuous membrane potential at the spiking event, the c-LIF. We investigate the dynamical regimes of a fully connected network of excitatory c-LIF neurons in the presence of short-term synaptic plasticity. By varying the coupling strength among neurons, we show that a complex chaotic dynamics arises, characterized by scale free avalanches. The origin of this phenomenon in the c-LIF can be related to the order symmetry breaking in neurons spike-times, which corresponds to the onset of a broad activity distribution. Our analysis uncovers a general mechanism through which networks of simple neurons can be attracted to a complex basin in the phase space.

q-bio.NC↗

arXiv2020

Shaping the learning landscape in neural networks around wide flat minima

Learning in Deep Neural Networks (DNN) takes place by minimizing a non-convex high-dimensional loss function, typically by a stochastic gradient descent (SGD) strategy. The learning process is observed to be able to find good minimizers without getting stuck in local critical points, and that such minimizers are often satisfactory at avoiding overfitting. How these two features can be kept under control in nonlinear devices composed of millions of tunable connections is a profound and far reaching open question. In this paper we study basic non-convex one- and two-layer neural network models which learn random patterns, and derive a number of basic geometrical and algorithmic features which suggest some answers. We first show that the error loss function presents few extremely wide flat minima (WFM) which coexist with narrower minima and critical points. We then show that the minimizers of the cross-entropy loss function overlap with the WFM of the error loss. We also show examples of learning devices for which WFM do not exist. From the algorithmic perspective we derive entropy driven greedy and message passing algorithms which focus their search on wide flat regions of minimizers. In the case of SGD and cross-entropy loss, we show that a slow reduction of the norm of the weights along the learning process also leads to WFM. We corroborate the results by a numerical study of the correlations between the volumes of the minimizers, their Hessian and their generalization performance on real data.

cs.LG↗

arXiv2017

Chaos and correlated avalanches in excitatory neural networks with synaptic plasticity

A collective chaotic phase with power law scaling of activity events is observed in a disordered mean field network of purely excitatory leaky integrate-and-fire neurons with short-term synaptic plasticity. The dynamical phase diagram exhibits two transitions from quasi-synchronous and asynchronous regimes to the nontrivial, collective, bursty regime with avalanches. In the homogeneous case without disorder, the system synchronizes and the bursty behavior is reflected into a doubling-period transition to chaos for a two dimensional discrete map. Numerical simulations show that the bursty chaotic phase with avalanches exhibits a spontaneous emergence of time correlations and enhanced Kolmogorov complexity. Our analysis reveals a mechanism for the generation of irregular avalanches that emerges from the combination of disorder and deterministic underlying chaotic dynamics.

cond-mat.dis-nn↗

arXiv2014

Nonsingular cosmology from evolutionary quantum gravity

We provide a cosmological implementation of the evolutionary quantum gravity, describing an isotropic Universe, in the presence of a negative cosmological constant and a massive (preinflationary) scalar field. We demonstrate that the considered Universe has a nonsingular quantum behavior, associated to a primordial bounce, whose ground state has a high occupation number. Furthermore, in such a vacuum state, the super-Hamiltonian eigenvalue is negative, corresponding to a positive emerging dust energy density. The regularization of the model is performed via a polymer quantum approach to the Universe scale factor and the proper classical limit is then recovered, in agreement with a preinflationary state of the Universe. Since the dust energy density is redshifted by the Universe deSitter phase and the cosmological constant does not enter the ground state eigenvalue, we get a late-time cosmology, compatible with the present observations, endowed with a turning point in the far future.

gr-qc↗