Resolution of similar patterns in a solvable model of unsupervised deep learning with structured data

Abstract

Empirical data, on which deep learning relies, has substantial internal structure, yet prevailing theories often disregard this aspect. Recent research has led to the definition of structured data ensembles, aimed at equipping established theoretical frameworks with interpretable structural elements, a pursuit that aligns with the broader objectives of spin glass theory. We consider a one-parameter structured ensemble where data consists of correlated pairs of patterns, and a simplified model of unsupervised learning, whereby the internal representation of the training set is fixed at each layer. A mean field solution of the model identifies a set of layer-wise recurrence equations for the overlaps between the internal representations of an unseen input and of the training set. The bifurcation diagram of this discrete-time dynamics is topologically inequivalent to the unstructured one, and displays transitions between different phases, selected by varying the load (the number of training pairs divided by the width of the network). The network's ability to resolve different patterns undergoes a discontinuous transition to a phase where signal processing along the layers dissipates differential information about an input's proximity to the different patterns in a pair. A critical value of the parameter tuning the correlations separates regimes where data structure improves or hampers the identification of a given pair of patterns.