arXiv subjects

Marco Baity-Jesi

Publications and source records attributed to Marco Baity-Jesi.

At least 19 recordsLinked to original sources

arXiv2026

Where You Place the Norm Matters: From Prejudiced to Neutral Initializations

Normalization layers were introduced to stabilize and accelerate training, yet their influence is critical already at initialization, where they shape signal propagation and output statistics before parameters adapt to data. In practice, both which normalization to use and where to place it are often chosen heuristically, despite the fact that these decisions can qualitatively alter a model's behavior. We provide a theoretical characterization of how normalization choice and placement (Pre-Norm vs. Post-Norm) determine the distribution of class predictions at initialization, ranging from unbiased (Neutral) to highly concentrated (Prejudiced) regimes. We show that these architectural decisions induce systematic shifts in the initial prediction regime, thereby modulating subsequent learning dynamics. By linking normalization design directly to prediction statistics at initialization, our results offer principled guidance for more controlled and interpretable network design, including clarifying how widely used choices such as BatchNorm vs. LayerNorm and Pre-Norm vs. Post-Norm shape behavior from the outset of training.

cs.LG↗

arXiv2026

When Bias Meets Trainability: Connecting Theories of Initialization

The statistical properties of deep neural networks (DNNs) at initialization play an important role to comprehend their trainability and the intrinsic architectural biases they possess before data exposure Well established mean field (MF) theories have uncovered that the distribution of parameters of randomly initialized networks strongly influences the behavior of the gradients, dictating whether they explode or vanish. Recent work has showed that untrained DNNs also manifest an initial guessing bias (IGB), in which large regions of the input space are assigned to a single class. In this work, we provide a theoretical proof that links IGB to previous MF theories for a vast class of DNNs, showing that efficient learning is tightly connected to a network prejudice towards a specific class. This connection leads to a counterintuitive conclusion: the initialization that optimizes trainability is systematically biased rather than neutral.

cs.LG↗

arXiv2024

Initial Guessing Bias: How Untrained Networks Favor Some Classes

Understanding and controlling biasing effects in neural networks is crucial for ensuring accurate and fair model performance. In the context of classification problems, we provide a theoretical analysis demonstrating that the structure of a deep neural network (DNN) can condition the model to assign all predictions to the same class, even before the beginning of training, and in the absence of explicit biases. We prove that, besides dataset properties, the presence of this phenomenon, which we call \textit{Initial Guessing Bias} (IGB), is influenced by model choices including dataset preprocessing methods, and architectural decisions, such as activation functions, max-pooling layers, and network depth. Our analysis of IGB provides information for architecture selection and model initialization. We also highlight theoretical consequences, such as the breakdown of node-permutation symmetry, the violation of self-averaging and the non-trivial effects that depth has on the phenomenon.

cs.LG↗

arXiv2024

A Theoretical Analysis of the Learning Dynamics under Class Imbalance

Data imbalance is a common problem in machine learning that can have a critical effect on the performance of a model. Various solutions exist but their impact on the convergence of the learning dynamics is not understood. Here, we elucidate the significant negative impact of data imbalance on learning, showing that the learning curves for minority and majority classes follow sub-optimal trajectories when training with a gradient-based optimizer. This slowdown is related to the imbalance ratio and can be traced back to a competition between the optimization of different classes. Our main contribution is the analysis of the convergence of full-batch (GD) and stochastic gradient descent (SGD), and of variants that renormalize the contribution of each per-class gradient. We find that GD is not guaranteed to decrease the loss for each class but that this problem can be addressed by performing a per-class normalization of the gradient. With SGD, class imbalance has an additional effect on the direction of the gradients: the minority class suffers from a higher directional noise, which reduces the effectiveness of the per-class gradient normalization. Our findings not only allow us to understand the potential and limitations of strategies involving the per-class gradients, but also the reason for the effectiveness of previously used solutions for class imbalance such as oversampling.

stat.ML↗

arXiv2024

Producing Plankton Classifiers that are Robust to Dataset Shift

Modern plankton high-throughput monitoring relies on deep learning classifiers for species recognition in water ecosystems. Despite satisfactory nominal performances, a significant challenge arises from Dataset Shift, which causes performances to drop during deployment. In our study, we integrate the ZooLake dataset with manually-annotated images from 10 independent days of deployment, serving as test cells to benchmark Out-Of-Dataset (OOD) performances. Our analysis reveals instances where classifiers, initially performing well in In-Dataset conditions, encounter notable failures in practical scenarios. For example, a MobileNet with a 92% nominal test accuracy shows a 77% OOD accuracy. We systematically investigate conditions leading to OOD performance drops and propose a preemptive assessment method to identify potential pitfalls when classifying new data, and pinpoint features in OOD images that adversely impact classification. We present a three-step pipeline: (i) identifying OOD degradation compared to nominal test performance, (ii) conducting a diagnostic analysis of degradation causes, and (iii) providing solutions. We find that ensembles of BEiT vision transformers, with targeted augmentations addressing OOD robustness, geometric ensembling, and rotation-based test-time augmentation, constitute the most robust model, which we call BEsT model. It achieves an 83% OOD accuracy, with errors concentrated on container classes. Moreover, it exhibits lower sensitivity to dataset shift, and reproduces well the plankton abundances. Our proposed pipeline is applicable to generic plankton classifiers, contingent on the availability of suitable test cells. By identifying critical shortcomings and offering practical procedures to fortify models against dataset shift, our study contributes to the development of more reliable plankton classification technologies.

cs.CV↗

arXiv2023

Differentiable modeling to unify machine learning and physical models and advance Geosciences

Process-Based Modeling (PBM) and Machine Learning (ML) are often perceived as distinct paradigms in the geosciences. Here we present differentiable geoscientific modeling as a powerful pathway toward dissolving the perceived barrier between them and ushering in a paradigm shift. For decades, PBM offered benefits in interpretability and physical consistency but struggled to efficiently leverage large datasets. ML methods, especially deep networks, presented strong predictive skills yet lacked the ability to answer specific scientific questions. While various methods have been proposed for ML-physics integration, an important underlying theme -- differentiable modeling -- is not sufficiently recognized. Here we outline the concepts, applicability, and significance of differentiable geoscientific modeling (DG). "Differentiable" refers to accurately and efficiently calculating gradients with respect to model variables, critically enabling the learning of high-dimensional unknown relationships. DG refers to a range of methods connecting varying amounts of prior knowledge to neural networks and training them together, capturing a different scope than physics-guided machine learning and emphasizing first principles. Preliminary evidence suggests DG offers better interpretability and causality than ML, improved generalizability and extrapolation capability, and strong potential for knowledge discovery, while approaching the performance of purely data-driven ML. DG models require less training data while scaling favorably in performance and efficiency with increasing amounts of data. With DG, geoscientists may be better able to frame and investigate questions, test hypotheses, and discover unrecognized linkages.

cs.LG↗

arXiv2023

An Introduction to the Theory of Spin Glasses

We review the main methods used to study spin glasses. In the first part, we focus on methods for fully connected models and systems defined on a tree, such as the replica method, the Thouless-Anderson-Palmer formalism, the cavity method, and the dynamical mean-field theory. In the second part, we deal with the description of low-dimensional systems, mostly in three spatial dimensions, which are mostly studied through numerical simulations. We conclude by mentioning some of the main open problems in the field.

cond-mat.dis-nn↗

arXiv2022

Modeling Node Exposure for Community Detection in Networks

In community detection, datasets often suffer a sampling bias for which nodes which would normally have a high affinity appear to have zero affinity. This happens for example when two affine users of a social network were not exposed to one another. Community detection on this kind of data suffers then from considering affine nodes as not affine. To solve this problem, we explicitly model the (non-)exposure mechanism in a Bayesian community detection framework, by introducing a set of additional hidden variables. Compared to approaches which do not model exposure, our method is able to better reconstruct the input graph, while maintaining a similar performance in recovering communities. Importantly, it allows to estimate the probability that two nodes have been exposed, a possibility not available with standard models.

cs.SI↗

arXiv2022

Predicting Chemical Hazard across Taxa through Machine Learning

We applied machine learning methods to predict chemical hazards focusing on fish acute toxicity across taxa. We analyzed the relevance of taxonomy and experimental setup, showing that taking them into account can lead to considerable improvements in the classification performance. We quantified the gain obtained throught the introduction of taxonomic and experimental information, compared to classification based on chemical information alone. We used our approach with standard machine learning models (K-nearest neighbors, random forests and deep neural networks), as well as the recently proposed Read-Across Structure Activity Relationship (RASAR) models, which were very successful in predicting chemical hazards to mammals based on chemical similarity. We were able to obtain accuracies of over 93% on datasets where, due to noise in the data, the maximum achievable accuracy was expected to be below 96%. The best performances were obtained by random forests and RASAR models. We analyzed metrics to compare our results with animal test reproducibility, and despite most of our models "outperform animal test reproducibility" as measured through recently proposed metrics, we showed that the comparison between machine learning performance and animal test reproducibility should be addressed with particular care. While we focused on fish mortality, our approach, provided that the right data is available, is valid for any combination of chemicals, effects and taxa.

q-bio.QM↗

arXiv2022

Competition between Barrier- and Entropy-Driven Activation in Glasses

In simplified models of glasses we clarify the existence of two different kinds of activated dynamics, which coexist, with one of the two dominating over the other. One is the energy barrier hopping that is typically used to picture activation, and the other one, which we call entropic activation, is driven by the scarcity of convenient directions. When entropic activation dominates, the height of the energy barriers is no longer the decisive to describe the system's slowdown. In our analysis, dominance of one mechanism over the other depends on the shape of the density of states and temperature. We also find that at low temperatures a phase transition between the two kinds of activation can occur. Our framework can be used to harmonize the facilitation and thermodynamic pictures of the slowdown of glasses.

cond-mat.dis-nn↗

arXiv2021

Revisiting the Concept of Activation in Supercooled Liquids

In this work we revisit the description of dynamics based on the concepts of metabasins and activation in mildly supercooled liquids via the analysis of the dynamics of a paradigmatic glass former between its onset temperature $T_{o}$ and mode-coupling temperature $T_{c}$. First, we provide measures that demonstrate that the onset of glassiness is indeed connected to the landscape, and that metabasin waiting time distributions are so broad that the system can remain stuck in a metabasin for times that exceed $τ_α$ by orders of magnitude. We then reanalyze the transitions between metabasins, providing several indications that the standard picture of activated dynamics in terms of traps does not hold in this regime. Instead, we propose that here activation is principally driven by entropic instead of energetic barriers. In particular, we illustrate that activation is not controlled by the hopping of high energetic barriers, and should more properly be interpreted as the entropic selection of nearly barrierless but rare pathways connecting metabasins on the landscape.

cond-mat.soft↗

arXiv2020

Effective Trap-like Activated Dynamics in a Continuous Landscape

We use a simple model to extend network models for activated dynamics to a continuous landscape with a well-defined notion of distance and a direct connection to many-body systems. The model consists of a tracer in a high-dimensional funnel landscape with no disorder. We find a non-equilibrium low-temperature phase with aging dynamics, that is effectively equivalent to that of models with built-in disorder, such as Trap Model, Step Model and REM.

cond-mat.dis-nn↗

arXiv2019

The jamming transition as a paradigm to understand the loss landscape of deep neural networks

Deep learning has been immensely successful at a variety of tasks, ranging from classification to AI. Learning corresponds to fitting training data, which is implemented by descending a very high-dimensional loss function. Understanding under which conditions neural networks do not get stuck in poor minima of the loss, and how the landscape of that loss evolves as depth is increased remains a challenge. Here we predict, and test empirically, an analogy between this landscape and the energy landscape of repulsive ellipses. We argue that in FC networks a phase transition delimits the over- and under-parametrized regimes where fitting can or cannot be achieved. In the vicinity of this transition, properties of the curvature of the minima of the loss are critical. This transition shares direct similarities with the jamming transition by which particles form a disordered solid as the density is increased, which also occurs in certain classes of computational optimization and learning problems such as the perceptron. Our analysis gives a simple explanation as to why poor minima of the loss cannot be encountered in the overparametrized regime, and puts forward the surprising result that the ability of fully connected networks to fit random data is independent of their depth. Our observations suggests that this independence also holds for real data. We also study a quantity $Δ$ which characterizes how well ($Δ<0$) or badly ($Δ>0$) a datum is learned. At the critical point it is power-law distributed, $P_+(Δ)\simΔ^θ$ for $Δ>0$ and $P_-(Δ)\sim(-Δ)^{-γ}$ for $Δ<0$, with $θ\approx0.3$ and $γ\approx0.2$. This observation suggests that near the transition the loss landscape has a hierarchical structure and that the learning dynamics is prone to avalanche-like dynamics, with abrupt changes in the set of patterns that are learned.

cond-mat.dis-nn↗

arXiv2019

On mean-field theories of dynamics in supercooled liquids

We develop a hybrid numerical approach to extract the exact memory function K(t) of a tagged particle in three-dimensional glass-forming liquids. We compare the behavior of the exact memory kernel to two mean-field approaches, namely the standard mode-coupling theory and a recently proposed ansatz for the memory function that forms the basis of a new derivation of the exact form of K(t) for a fluid with short-ranged interactions in infinite dimensions. Each of the mean-field functions qualitatively and quantitatively share traits with the exact K(t), although several important quantitative differences are manifest.

cond-mat.soft↗

arXiv2019

Maximum-energy records in glassy energy landscapes

We study the evolution of the maximum energy $E_\max(t)$ reached between time $0$ and time $t$ in the dynamics of simple models with glassy energy landscapes, in instant quenches from infinite temperature to a target temperature $T$. Through a detailed description of the activated dynamics, we are able to describe the evolution of $E_\max(t)$ from short times, through the aging regime, until after equilibrium is reached, thus providing a detailed description of the long-time dynamics. Finally, we compare our findings with numerical simulations of the $p$-spin glass and show how the maximum energy record can be used to identify the threshold energy in this model.

cond-mat.dis-nn↗

arXiv2019

Precursors of the Spin Glass Transition in Three Dimensions

We study energy landscape and dynamics of the three-dimensional Heisenberg Spin Glass model in the paramagnetic phase, i.e. for temperature $T$ larger than the critical temperature $T_\mathrm{c}$. The landscape is non-trivially related to the equilibrium states even in the high-temperature phase, and reveals an onset of non-trivial behavior at a temperature $T_\mathrm{o}$, which is also seen through the behavior of the thermoremanent magnetization. We also find a power-law growth of the relaxation times far from the spin-glass transition, indicating a dynamical crossover at a temperature $T_\mathrm{d}$, $T_\mathrm{c}<T_\mathrm{d}<T_\mathrm{o}$. The arising picture is reminiscent of the phenomenology of supercooled liquids, and poses questions on which mean-field models can describe qualitatively well the phenomenology in three dimensions. On the technical side, local energy minima are found with the Successive Overrelaxation algorithm, which reveals very efficient for energy minimization in this kind of models.

cond-mat.dis-nn↗

arXiv2018

Activated dynamics: an intermediate model between REM and p-spin

In order to study the activated dynamics of mean-field glasses, which takes place on times of order exp(N), where N is the system size, we introduce a new model, the Correlated Random Energy Model (CREM), that allows for a smooth interpolation between the REM and the p-spin models. We study numerically and analytically the CREM in the intermediate regime between REM and p-spin. We fully characterize its energy landscape, which is like a golf-course but, at variance with the REM, has metabasins (or holes) containing several configurations. We find that an effective trap-like description for the dynamics emerges, provided that one identifies metabasins in the CREM with configurations in the trap model.

cond-mat.dis-nn↗

arXiv2017

Activated Aging Dynamics and Effective Trap Model Description in the Random Energy Model

We study the out-of-equilibrium aging dynamics of the Random Energy Model (REM) ruled by a single spin-flip Metropolis dynamics. We focus on the dynamical evolution taking place on time-scales diverging with the system size. Our aim is to show to what extent the activated dynamics displayed by the REM can be described in terms of an effective trap model. We identify two time regimes: the first one corresponds to the process of escaping from a basin in the energy landscape and to the subsequent exploration of high energy configurations, whereas the second one corresponds to the evolution from a deep basin to the other. By combining numerical simulations with analytical arguments we show why the trap model description does not hold in the former but becomes exact in the second.

cond-mat.dis-nn↗