arXiv subjects

Yann Ollivier

Publications and source records attributed to Yann Ollivier.

48 records · Page 3Linked to original sources

arXiv2015

Riemannian metrics for neural networks II: recurrent networks and learning symbolic data sequences

Recurrent neural networks are powerful models for sequential data, able to represent complex dependencies in the sequence that simpler models such as hidden Markov models cannot handle. Yet they are notoriously hard to train. Here we introduce a training procedure using a gradient ascent in a Riemannian metric: this produces an algorithm independent from design choices such as the encoding of parameters and unit activities. This metric gradient ascent is designed to have an algorithmic cost close to backpropagation through time for sparsely connected networks. We use this procedure on gated leaky neural networks (GLNNs), a variant of recurrent neural networks with an architecture inspired by finite automata and an evolution equation inspired by continuous-time networks. GLNNs trained with a Riemannian gradient are demonstrated to effectively capture a variety of structures in synthetic problems: basic block nesting as in context-free grammars (an important feature of natural languages, but difficult to learn), intersections of multiple independent Markov-type relations, or long-distance relationships such as the distant-XOR problem. This method does not require adjusting the network structure or initial parameters: the network used is a sparse random graph and the initialization is identical for all problems considered.

cs.NE↗

arXiv2015

Auto-encoders: reconstruction versus compression

We discuss the similarities and differences between training an auto-encoder to minimize the reconstruction error, and training the same auto-encoder to compress the data via a generative model. Minimizing a codelength for the data using an auto-encoder is equivalent to minimizing the reconstruction error plus some correcting terms which have an interpretation as either a denoising or contractive property of the decoding function. These terms are related but not identical to those used in denoising or contractive auto-encoders [Vincent et al. 2010, Rifai et al. 2011]. In particular, the codelength viewpoint fully determines an optimal noise level for the denoising criterion.

cs.NE↗

arXiv2013

Objective Improvement in Information-Geometric Optimization

Information-Geometric Optimization (IGO) is a unified framework of stochastic algorithms for optimization problems. Given a family of probability distributions, IGO turns the original optimization problem into a new maximization problem on the parameter space of the probability distributions. IGO updates the parameter of the probability distribution along the natural gradient, taken with respect to the Fisher metric on the parameter manifold, aiming at maximizing an adaptive transform of the objective function. IGO recovers several known algorithms as particular instances: for the family of Bernoulli distributions IGO recovers PBIL, for the family of Gaussian distributions the pure rank-mu CMA-ES update is recovered, and for exponential families in expectation parametrization the cross-entropy/ML method is recovered. This article provides a theoretical justification for the IGO framework, by proving that any step size not greater than 1 guarantees monotone improvement over the course of optimization, in terms of q-quantile values of the objective function f. The range of admissible step sizes is independent of f and its domain. We extend the result to cover the case of different step sizes for blocks of the parameters in the IGO algorithm. Moreover, we prove that expected fitness improves over time when fitness-proportional selection is applied, in which case the RPP algorithm is recovered.

cs.LG↗

arXiv2013

Layer-wise learning of deep generative models

When using deep, multi-layered architectures to build generative models of data, it is difficult to train all layers at once. We propose a layer-wise training procedure admitting a performance guarantee compared to the global optimum. It is based on an optimistic proxy of future performance, the best latent marginal. We interpret auto-encoders in this setting as generative models, by showing that they train a lower bound of this criterion. We test the new learning procedure against a state of the art method (stacked RBMs), and find it to improve performance. Both theory and experiments highlight the importance, when training deep architectures, of using an inference model (from data to hidden variables) richer than the generative model (from hidden variables to data).

cs.NE↗

arXiv2011

A curved Brunn--Minkowski inequality on the discrete hypercube

We compare two approaches to Ricci curvature on non-smooth spaces, in the case of the discrete hypercube $\{0,1\}^N$. While the coarse Ricci curvature of the first author readily yields a positive value for curvature, the displacement convexity property of Lott, Sturm and the second author could not be fully implemented. Yet along the way we get new results of a combinatorial and probabilistic nature, including a curved Brunn--Minkowski inequality on the discrete hypercube.

math.PR↗

arXiv2010

Curvature, concentration and error estimates for Markov chain Monte Carlo

We provide explicit nonasymptotic estimates for the rate of convergence of empirical means of Markov chains, together with a Gaussian or exponential control on the deviations of empirical means. These estimates hold under a "positive curvature" assumption expressing a kind of metric ergodicity, which generalizes the Ricci curvature from differential geometry and, on finite graphs, amounts to contraction under path coupling.

math.PR↗

arXiv2007

Ricci curvature of Markov chains on metric spaces

We define the Ricci curvature of Markov chains on metric spaces as a local contraction coefficient of the random walk acting on the space of probability measures equipped with a Wasserstein transportation distance. For Brownian motion on a Riemannian manifold this gives back the value of Ricci curvature of a tangent vector. Examples of positively curved spaces for this definition include the discrete cube and discrete versions of the Ornstein--Uhlenbeck process. Moreover this generalization is consistent with the Bakry--Émery Ricci curvature for Brownian motion with a drift on a Riemannian manifold. Positive Ricci curvature is easily shown to imply a spectral gap, a Lévy--Gromov-like Gaussian concentration theorem and a kind of modified logarithmic Sobolev inequality. These bounds are sharp in several interesting examples.

math.PR↗

arXiv2004

Some small cancellation properties of random groups

We work in the density model of random groups. We prove that they satisfy an isoperimetric inequality with sharp constant $1-2d$ depending upon the density parameter $d$. This implies in particular a property generalizing the ordinary $C'$ small cancellation condition, which could be termed ``macroscopic small cancellation''. This also sharpens the evaluation of the hyperbolicity constant $δ$. As a consequence we get that the standard presentation of a random group at density $d<1/5$ satisfies the Dehn algorithm and Greendlinger's Lemma, and that it does not for $d>1/5$.

math.GR↗

arXiv2004

Kazhdan groups with infinite outer automorphism group

For each countable group $Q$ we produce a short exact sequence $1\to N \to G \to Q\to 1$ where $G$ is f.g. and has a graphical $\frac16$ presentation and $N$ is f.g. and satisfies property $T$. As a consequence we produce a group $N$ with property $T$ such that $\Out(N)$ is infinite. Using the tools developed we are also able to produce examples of nonHopfian and non-coHopfian groups with property $T$. One of our main tools is the use of random groups to achieve certain properties.

math.GR↗

arXiv2004

Growth exponent of generic groups

Grigorchuk and de la Harpe asked if there are many groups with growth exponent close to that of the free group with the same number of generators. We prove that this is in fact the case for a generic group (in the density model of random groups). More generally, we prove that the growth exponent does not change much through a random quotient of a torsion-free hyperbolic group.

math.GR↗

arXiv2004

Cogrowth and spectral gap of generic groups

We prove that that for all $\eps$, having cogrowth exponent at most $1/2+\eps$ (in base $2m-1$ with $m$ the number of generators) is a generic property of groups in the density model of random groups. This generalizes a theorem of Grigorchuk and Champetier. More generally we show that the cogrowth of a random quotient of a torsion-free hyperbolic group stays close to that of this group. This proves in particular that the spectral gap of a generic group is as large as it can be.

math.GR↗

arXiv2004

Sharp phase transition theorems for hyperbolicity of random groups

We prove that in various natural models of a random quotient of a group, depending on a density parameter, for each hyperbolic group there is some critical density under which a random quotient is still hyperbolic with high probability, whereas above this critical value a random quotient is very probably trivial. We give explicit characterizations of these critical densities for the various models.

math.GR↗