arXiv subjects

Gibbs Nwemadji

Publications and source records attributed to Gibbs Nwemadji.

3 recordsLinked to original sources

arXiv2026

When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index models

Pre-training on a source task is usually expected to facilitate fine-tuning on similar downstream problems. In this work, we mathematically show that this naive intuition is not always true: excessive pre-training can computationally slow down fine-tuning optimization. We study this phenomenon for low-rank adaptation (LoRA) fine-tuning on single-index models trained under one-pass SGD. Leveraging a summary statistics description of the fine-tuning dynamics, we precisely characterize how the convergence rate depends on the initial fine-tuning alignment and the degree of non-linearity of the target task. The key take away is that even when the pre-training and downstream tasks are well aligned, strong pre-training can induce a prolonged search phase and hinder convergence. Our theory thus provides a unified picture of how pre-training strength and task difficulty jointly shape the dynamics and limitations of LoRA fine-tuning in a nontrivial tractable model. On the practical side, we empirically show that our theoretical findings extend beyond our toy model and remain relevant in the context of a vision-transformer model trained on real data.

cs.LG↗

arXiv2026

Generalization performance of narrow one-hidden layer networks in the teacher-student setting

Understanding the generalization properties of neural networks on simple input-output distributions is key to explaining their performance on real datasets. The classical teacher-student setting, where a network is trained on data generated by a teacher model, provides a canonical theoretical test bed. In this context, a complete theoretical characterization of fully connected one-hidden-layer networks with generic activation functions remains missing. In this work, we develop a general framework for such networks with large width, yet much smaller than the input dimension. Using methods from statistical physics, we derive closed-form expressions for the typical performance of both finite-temperature (Bayesian) and empirical risk minimization estimators in terms of a small number of order parameters. We uncover a transition to a specialization phase, where hidden neurons align with teacher features once the number of samples becomes sufficiently large and proportional to the number of network parameters. Our theory accurately predicts the generalization error of networks trained on regression and classification tasks using either noisy full-batch gradient descent (Langevin dynamics) or deterministic full-batch gradient descent.

cond-mat.dis-nn↗

arXiv2023

On the capacity of a quantum perceptron for storing biased patterns

Although different architectures of quantum perceptrons have been recently put forward, the capabilities of such quantum devices versus their classical counterparts remain debated. Here, we consider random patterns and targets independently distributed with biased probabilities and investigate the storage capacity of a continuous quantum perceptron model that admits a classical limit, thus facilitating the comparison of performances. Such a more general context extends a previous study of the quantum storage capacity where using statistical mechanics techniques in the limit of a large number of inputs, it was proved that no quantum advantages are to be expected concerning the storage properties. This outcome is due to the fuzziness inevitably introduced by the intrinsic stochasticity of quantum devices. We strengthen such an indication by showing that the possibility of indefinitely enhancing the storage capacity for highly correlated patterns, as it occurs in a classical setting, is instead prevented at the quantum level.

quant-ph↗