arXiv subjects

Alessandro Laio

Publications and source records attributed to Alessandro Laio.

At least 37 records · Page 2Linked to original sources

arXiv2025

Towards a robust approach to infer causality in molecular systems satisfying detailed balance

The ability to distinguish between correlation and causation of variables in molecular systems remains an interesting and open area of investigation. In this work, we probe causality in a molecular system using two independent computational methods that infer the causal direction through the language of information transfer. Specifically, we demonstrate that a molecular dynamics simulation involving a single Tryptophan in liquid water displays asymmetric information transfer between specific collective variables, such as solute and solvent coordinates. Analyzing a discrete Markov-state and Langevin dynamics on a 2D free energy surface, we show that the same kind of asymmetries can emerge even in extremely simple systems, undergoing equilibrium and time-reversible dynamics. We use these model systems to rationalize the unidirectional information transfer in the molecular system in terms of asymmetries in the underlying free energy landscape and/or relaxation dynamics of the relevant coordinates. Finally, we propose a computational experiment that allows one to decide if an asymmetric information transfer between two variables corresponds to a genuine causal link.

physics.chem-ph↗

arXiv2025

A distributional simplicity bias in the learning dynamics of transformers

The remarkable capability of over-parameterised neural networks to generalise effectively has been explained by invoking a ``simplicity bias'': neural networks prevent overfitting by initially learning simple classifiers before progressing to more complex, non-linear functions. While simplicity biases have been described theoretically and experimentally in feed-forward networks for supervised learning, the extent to which they also explain the remarkable success of transformers trained with self-supervised techniques remains unclear. In our study, we demonstrate that transformers, trained on natural language data, also display a simplicity bias. Specifically, they sequentially learn many-body interactions among input tokens, reaching a saturation point in the prediction error for low-degree interactions while continuing to learn high-degree interactions. To conduct this analysis, we develop a procedure to generate \textit{clones} of a given natural language data set, which rigorously capture the interactions between tokens up to a specified order. This approach opens up the possibilities of studying how interactions of different orders in the data affect learning, in natural language processing and beyond.

cs.CL↗

arXiv2024

Automatic feature selection and weighting in molecular systems using Differentiable Information Imbalance

Feature selection is essential in the analysis of molecular systems and many other fields, but several uncertainties remain: What is the optimal number of features for a simplified, interpretable model that retains essential information? How should features with different units be aligned, and how should their relative importance be weighted? Here, we introduce the Differentiable Information Imbalance (DII), an automated method to rank information content between sets of features. Using distances in a ground truth feature space, DII identifies a low-dimensional subset of features that best preserves these relationships. Each feature is scaled by a weight, which is optimized by minimizing the DII through gradient descent. This allows simultaneously performing unit alignment and relative importance scaling, while preserving interpretability. DII can also produce sparse solutions and determine the optimal size of the reduced feature space. We demonstrate the usefulness of this approach on two benchmark molecular problems: (1) identifying collective variables that describe conformations of a biomolecule, and (2) selecting features for training a machine-learning force field. These results show the potential of DII in addressing feature selection challenges and optimizing dimensionality in various applications. The method is available in the Python library DADApy.

cs.LG↗

arXiv2024

Coarse Grained Molecular Dynamics with Normalizing Flows

We propose a sampling algorithm relying on a collective variable (CV) of mid-size dimension modelled by a normalizing flow and using non-equilibrium dynamics to propose full configurational moves from the proposition of a refreshed value of the CV made by the flow. The algorithm takes the form of a Markov chain with non-local updates, allowing jumps through energy barriers across metastable states. The flow is trained throughout the algorithm to reproduce the free energy landscape of the CV. The output of the algorithm are a sample of thermalized configurations and the trained network that can be used to efficiently produce more configurations. We show the functioning of the algorithm first on a test case with a mixture of Gaussians. Then we successfully test it on a higher dimensional system consisting in a polymer in solution with a compact and an extended stable state separated by a high free energy barrier.

cond-mat.stat-mech↗

arXiv2024

Agnostic detection of large-scale weather patterns in the northern hemisphere: from blockings to teleconnections

Detecting recurrent weather patterns and understanding the transitions between such regimes are key to advancing our knowledge on the low-frequency variability of the atmosphere and have important implications in terms of weather and climate-related risks. We adapt an analysis pipeline inspired by Markov State Modelling and detect in an unsupervised manner the dominant winter mid-latitude Northern Hemisphere weather patterns in the Atlantic and Pacific sectors. The daily 500 hPa geopotential height fields are first classified in $\sim 200$ microstates. The weather dynamics is then represented in the basis of these microstates and the slowest decaying modes are identified from the spectral properties of the transition probability matrix. These modes are defined on the basis of the nonlinear dynamical processes of the system and not as tentative metastable states as often done in Markov state analysis. In the Atlantic and Pacific sectors slow relaxation processes are mainly related to transitions between blocked regimes and zonal flow. We also find strong evidence of a dynamical regime associated with the simultaneous Atlantic-Pacific blocking. When the analysis is performed in a broader geographical region of the Atlantic sector, we discover that the slowest relaxation modes of the system are associated with transitions between dynamical regimes that resemble teleconnection patterns like the North Atlantic Oscillation and weather regimes like the Scandinavian and Greenland blocking, yet have a much stronger dynamical foundation than classical methods based e.g. on EOF analysis. Our method clarifies that, as a result of the lack of a time-scale separation in the atmospheric variability of the mid-latitudes, there is no clear-cut way to represent the atmospheric dynamics in terms of few, well-defined modes of variability.

physics.ao-ph↗

arXiv2024

Robust inference of causality in high-dimensional dynamical processes from the Information Imbalance of distance ranks

We introduce an approach which allows detecting causal relationships between variables for which the time evolution is available. Causality is assessed by a variational scheme based on the Information Imbalance of distance ranks, a statistical test capable of inferring the relative information content of different distance measures. We test whether the predictability of a putative driven system Y can be improved by incorporating information from a potential driver system X, without explicitly modeling the underlying dynamics and without the need to compute probability densities of the dynamic variables. This framework makes causality detection possible even between high-dimensional systems where only few of the variables are known or measured. Benchmark tests on coupled chaotic dynamical systems demonstrate that our approach outperforms other model-free causality detection methods, successfully handling both unidirectional and bidirectional couplings. We also show that the method can be used to robustly detect causality in human electroencephalography data.

stat.ME↗

arXiv2024

Maximally informative feature selection using Information Imbalance: Application to COVID-19 severity prediction

Clinical databases typically include, for each patient, many heterogeneous features, for example blood exams, the clinical history before the onset of the disease, the evolution of the symptoms, the results of imaging exams, and many others. We here propose to exploit a recently developed statistical approach, the Information Imbalance, to compare different subsets of patient features, and automatically select the set of features which is maximally informative for a given clinical purpose, especially in minority classes. We adapt the Information Imbalance approach to work in a clinical framework, where patient features are often categorical and are generally available only for a fraction of the patients. We apply this algorithm to a data set of ~ 1,300 patients treated for COVID-19 in Udine hospital before October 2021. Using this approach, we find combinations of features which, if used in combination, are maximally informative of the clinical fate and of the severity of the disease. The optimal number of features, which is determined automatically, turns out to be between 10 and 15. These features can be measured at admission. The approach can be used also if the features are available only for a fraction of the patients, does not require imputation and, importantly, is able to automatically select features with small inter-feature correlation. Clinical insights deriving from this study are also discussed.

stat.ME↗

arXiv2024

Mapping of attention mechanisms to a generalized Potts model

Transformers are neural networks that revolutionized natural language processing and machine learning. They process sequences of inputs, like words, using a mechanism called self-attention, which is trained via masked language modeling (MLM). In MLM, a word is randomly masked in an input sequence, and the network is trained to predict the missing word. Despite the practical success of transformers, it remains unclear what type of data distribution self-attention can learn efficiently. Here, we show analytically that if one decouples the treatment of word positions and embeddings, a single layer of self-attention learns the conditionals of a generalized Potts model with interactions between sites and Potts colors. Moreover, we show that training this neural network is exactly equivalent to solving the inverse Potts problem by the so-called pseudo-likelihood method, well known in statistical physics. Using this mapping, we compute the generalization error of self-attention in a model scenario analytically using the replica method.

cond-mat.dis-nn↗

arXiv2023

The geometry of hidden representations of large transformer models

Large transformers are powerful architectures used for self-supervised data analysis across various data types, including protein sequences, images, and text. In these models, the semantic structure of the dataset emerges from a sequence of transformations between one representation and the next. We characterize the geometric and statistical properties of these representations and how they change as we move through the layers. By analyzing the intrinsic dimension (ID) and neighbor composition, we find that the representations evolve similarly in transformers trained on protein language tasks and image reconstruction tasks. In the first layers, the data manifold expands, becoming high-dimensional, and then contracts significantly in the intermediate layers. In the last part of the model, the ID remains approximately constant or forms a second shallow peak. We show that the semantic information of the dataset is better expressed at the end of the first peak, and this phenomenon can be observed across many models trained on diverse datasets. Based on our findings, we point out an explicit strategy to identify, without supervision, the layers that maximize semantic content: representations at intermediate layers corresponding to a relative minimum of the ID profile are more suitable for downstream learning tasks.

cs.LG↗

arXiv2023

Redundant representations help generalization in wide neural networks

Deep neural networks (DNNs) defy the classical bias-variance trade-off: adding parameters to a DNN that interpolates its training data will typically improve its generalization performance. Explaining the mechanism behind this ``benign overfitting'' in deep networks remains an outstanding challenge. Here, we study the last hidden layer representations of various state-of-the-art convolutional neural networks and find that if the last hidden representation is wide enough, its neurons tend to split into groups that carry identical information, and differ from each other only by statistically independent noise. The number of such groups increases linearly with the width of the layer, but only if the width is above a critical value. We show that redundant neurons appear only when the training process reaches interpolation and the training error is zero.

stat.ML↗

arXiv2023

Metadynamics for transition paths in irreversible dynamics

Stochastic systems often exhibit multiple viable metastable states that are long-lived. Over very long timescales, fluctuations may push the system to transition between them, drastically changing its macroscopic configuration. In realistic systems, these transitions can happen via multiple physical mechanisms, corresponding to multiple distinct transition channels for a pair of states. In this paper, we use the fact that the transition path ensemble is equivalent to the invariant measure of a gradient flow in pathspace, which can be efficiently sampled via metadynamics. We demonstrate how this pathspace metadynamics, previously restricted to reversible molecular dynamics, is in fact very generally applicable to metastable stochastic systems, including irreversible and time-dependent ones, and allows to estimate rigorously the relative probability of competing transition paths. We showcase this approach on the study of a stochastic partial differential equation describing magnetic field reversal in the presence of advection.

cond-mat.stat-mech↗

arXiv2023

Intrinsic dimension estimation for discrete metrics

Real world-datasets characterized by discrete features are ubiquitous: from categorical surveys to clinical questionnaires, from unweighted networks to DNA sequences. Nevertheless, the most common unsupervised dimensional reduction methods are designed for continuous spaces, and their use for discrete spaces can lead to errors and biases. In this letter we introduce an algorithm to infer the intrinsic dimension (ID) of datasets embedded in discrete spaces. We demonstrate its accuracy on benchmark datasets, and we apply it to analyze a metagenomic dataset for species fingerprinting, finding a surprisingly small ID, of order 2. This suggests that evolutive pressure acts on a low-dimensional manifold despite the high-dimensionality of sequences' space.

stat.ML↗

arXiv2023

Optimal transfer protocol by incremental layer defrosting

Transfer learning is a powerful tool enabling model training with limited amounts of data. This technique is particularly useful in real-world problems where data availability is often a serious limitation. The simplest transfer learning protocol is based on ``freezing" the feature-extractor layers of a network pre-trained on a data-rich source task, and then adapting only the last layers to a data-poor target task. This workflow is based on the assumption that the feature maps of the pre-trained model are qualitatively similar to the ones that would have been learned with enough data on the target task. In this work, we show that this protocol is often sub-optimal, and the largest performance gain may be achieved when smaller portions of the pre-trained network are kept frozen. In particular, we make use of a controlled framework to identify the optimal transfer depth, which turns out to depend non-trivially on the amount of available training data and on the degree of source-target task correlation. We then characterize transfer optimality by analyzing the internal representations of two networks trained from scratch on the source and the target task through multiple established similarity measures.

cs.LG↗

arXiv2022

Do Machine-Learning Atomic Descriptors and Order Parameters Tell the Same Story? The Case of Liquid Water

Machine-learning (ML) has become a key workhorse in molecular simulations. Building an ML model in this context, involves encoding the information of chemical environments using local atomic descriptors. In this work, we focus on the Smooth Overlap of Atomic Positions (SOAP) and their application in studying the properties of liquid water both in the bulk and at the hydrophobic air-water interface. By using a statistical test aimed at assessing the relative information content of different distance measures defined on the same data space, we investigate if these descriptors provide the same information as some of the common order parameters that are used to characterize local water structure such as hydrogen bonding, density or tetrahedrality to name a few. Our analysis suggests that the ML description and the standard order parameters of local water structure are not equivalent. In particular, a combination of these order parameters probing local water environments can predict SOAP similarity only approximately, and viceversa, the environments that are similar according to SOAP are not necessarily similar according to the standard order parameters. We also elucidate the role of some of the metaparameters entering in the SOAP definition in encoding chemical information.

cond-mat.soft↗

arXiv2022

DADApy: Distance-based Analysis of DAta-manifolds in Python

DADApy is a python software package for analysing and characterising high-dimensional data manifolds. It provides methods for estimating the intrinsic dimension and the probability density, for performing density-based clustering and for comparing different distance metrics. We review the main functionalities of the package and exemplify its usage in toy cases and in a real-world application. DADApy is freely available under the open-source Apache 2.0 license.

cs.LG↗

arXiv2022

A Monte Carlo approach to the conformal bootstrap

We introduce an approach to find approximate numerical solutions of truncated bootstrap equations for Conformal Field Theories (CFTs) in arbitrary dimensions. The method is based on a stochastic search via a Metropolis algorithm guided by an action $S$ which is the logarithm of the truncated bootstrap equations for a single scalar field correlator. While numerical conformal bootstrap methods based on semi-definite programming put rigorous exclusion bounds on CFTs, this method looks for approximate solutions, which correspond to local minima of $S$, when present, and can be even far from the extremality region. By this protocol we find that if no constraint on the operator scaling dimensions is imposed, $S$ has a single minimum, corresponding to the Free Theory. If we fix the external operator dimension, however, we encounter minima that can be studied with our approach. Imposing a conserved stress-tensor, a $\mathbf{Z}_2$ symmetry and one relevant scalar, we identify two regions where local minima of $S$ are present. When projected in the $(Δ_σ, Δ_ε)$-plane, $σ$ and $ε$ being the external and the lightest exchanged operators, one of these regions essentially coincides with the extremality line found in previous bootstrap studies. The other region is along the generalized free theories in $d = 2$ and below that in both $d = 3$ and $d = 4$. We empirically prove that some of the minima found are associated to known theories, including the $2d$ and $3d$ Ising theories and the $2d$ Yang-Lee model.

hep-th↗

arXiv2022

Prune and distill: similar reformatting of image information along rat visual cortex and deep neural networks

Visual object recognition has been extensively studied in both neuroscience and computer vision. Recently, the most popular class of artificial systems for this task, deep convolutional neural networks (CNNs), has been shown to provide excellent models for its functional analogue in the brain, the ventral stream in visual cortex. This has prompted questions on what, if any, are the common principles underlying the reformatting of visual information as it flows through a CNN or the ventral stream. Here we consider some prominent statistical patterns that are known to exist in the internal representations of either CNNs or the visual cortex and look for them in the other system. We show that intrinsic dimensionality (ID) of object representations along the rat homologue of the ventral stream presents two distinct expansion-contraction phases, as previously shown for CNNs. Conversely, in CNNs, we show that training results in both distillation and active pruning (mirroring the increase in ID) of low- to middle-level image information in single units, as representations gain the ability to support invariant discrimination, in agreement with previous observations in rat visual cortex. Taken together, our findings suggest that CNNs and visual cortex share a similarly tight relationship between dimensionality expansion/reduction of object representations and reformatting of image information.

q-bio.NC↗

arXiv2022

Ranking the information content of distance measures

Real-world data typically contain a large number of features that are often heterogeneous in nature, relevance, and also units of measure. When assessing the similarity between data points, one can build various distance measures using subsets of these features. Using the fewest features but still retaining sufficient information about the system is crucial in many statistical learning approaches, particularly when data are sparse. We introduce a statistical test that can assess the relative information retained when using two different distance measures, and determine if they are equivalent, independent, or if one is more informative than the other. This in turn allows finding the most informative distance measure out of a pool of candidates. The approach is applied to find the most relevant policy variables for controlling the Covid-19 epidemic and to find compact yet informative representations of atomic structures, but its potential applications are wide ranging in many branches of science.

stat.ML↗