arXiv subjects

Stefano Peluchetti

Publications and source records attributed to Stefano Peluchetti.

22 records · Page 2Linked to original sources

arXiv2020

Infinitely deep neural networks as diffusion processes

When the parameters are independently and identically distributed (initialized) neural networks exhibit undesirable properties that emerge as the number of layers increases, e.g. a vanishing dependency on the input and a concentration on restrictive families of functions including constant functions. We consider parameter distributions that shrink as the number of layers increases in order to recover well-behaved stochastic processes in the limit of infinite depth. This leads to set forth a link between infinitely deep residual networks and solutions to stochastic differential equations, i.e. diffusion processes. We show that these limiting processes do not suffer from the aforementioned issues and investigate their properties.

stat.ML↗

arXiv2020

Stable behaviour of infinitely wide deep neural networks

We consider fully connected feed-forward deep neural networks (NNs) where weights and biases are independent and identically distributed as symmetric centered stable distributions. Then, we show that the infinite wide limit of the NN, under suitable scaling on the weights, is a stochastic process whose finite-dimensional distributions are multivariate stable distributions. The limiting process is referred to as the stable process, and it generalizes the class of Gaussian processes recently obtained as infinite wide limits of NNs (Matthews at al., 2018b). Parameters of the stable process can be computed via an explicit recursion over the layers of the network. Our result contributes to the theory of fully connected feed-forward deep NNs, and it paves the way to expand recent lines of research that rely on Gaussian infinite wide limits.

stat.ML↗

arXiv2019

An empirical study of pretrained representations for few-shot classification

Recent algorithms with state-of-the-art few-shot classification results start their procedure by computing data features output by a large pretrained model. In this paper we systematically investigate which models provide the best representations for a few-shot image classification task when pretrained on the Imagenet dataset. We test their representations when used as the starting point for different few-shot classification algorithms. We observe that models trained on a supervised classification task have higher performance than models trained in an unsupervised manner even when transferred to out-of-distribution datasets. Models trained with adversarial robustness transfer better, while having slightly lower accuracy than supervised models.

cs.LG↗

arXiv2012

ε-Strong simulation of the Brownian path

We present an iterative sampling method which delivers upper and lower bounding processes for the Brownian path. We develop such processes with particular emphasis on being able to unbiasedly simulate them on a personal computer. The dominating processes converge almost surely in the supremum and $L_1$ norms. In particular, the rate of converge in $L_1$ is of the order $\mathcal {O}(\mathcal{K}^{-1/2})$, $\mathcal{K}$ denoting the computing cost. The a.s. enfolding of the Brownian path can be exploited in Monte Carlo applications involving Brownian paths whence our algorithm (termed the $\varepsilon$-strong algorithm) can deliver unbiased Monte Carlo estimators over path expectations, overcoming discretisation errors characterising standard approaches. We will show analytical results from applications of the $\varepsilon$-strong algorithm for estimating expectations arising in option pricing. We will also illustrate that individual steps of the algorithm can be of separate interest, giving new simulation methods for interesting Brownian distributions.

stat.CO↗