arXiv subjects

Matteo Smerlak

Publications and source records attributed to Matteo Smerlak.

At least 19 recordsLinked to original sources

arXiv2026

Spectra of high-dimensional Spearman correlation matrices under scale-mixture dependence

We study the asymptotic spectral properties of high-dimensional Spearman correlation matrices for scale-mixture data. We consider observations of the form $x_t=σ_t ξ_t \in \mathbb{R}^N,$ where the coordinates of $ξ_t$ are i.i.d.\ and the scalar mixture variable $σ_t$ is shared by all coordinates. Under natural symmetry assumptions, the coordinates of $x_t$ are pairwise uncorrelated in both the Pearson and Spearman sense. Nevertheless, they are not independent when the mixture variable is non-degenerate. We show that this higher-order dependence survives the rank transformation and leaves a nontrivial spectral signature. In the proportional regime $N/T\to q\in(0,\infty),$ the empirical spectral distribution of the Spearman correlation matrix converges almost surely to a generalized Marčenko--Pastur law governed by the limiting distribution of an effective rank variance. We also formulate a broader latent-variable extension, which covers, in particular, some scale-mixture models with correlated directional components. We discuss solvable examples and numerical approximations, motivated in part by heavy-tailed data in robust multivariate statistics, econometrics, and finance.

math.ST↗

arXiv2026

Another Marcenko-Pastur law for Kendall's tau

Bandeira et al. (2017) show that the eigenvalues of the Kendall correlation matrix of $n$ i.i.d. random vectors in $\mathbb{R}^p$ are asymptotically distributed like $1/3 + (2/3)Y_q$, where $Y_q$ has a Marčenko-Pastur law with parameter $q=\lim(p/n)$ if $p, n\to\infty$ proportionately to one another. Here we show that another Marčenko-Pastur law emerges in the "ultra-high dimensional" scaling limit where $p\sim q'\, n^2/2$ for some $q'>0$: in this quadratic scaling regime, Kendall correlation eigenvalues converge weakly almost surely to $(1/3)Y_{q'}$.

math.PR↗

arXiv2024

Complexity-stability relationships in competitive disordered dynamical systems

Robert May famously used random matrix theory to predict that large, complex systems cannot admit stable fixed points. However, this general conclusion is not always supported by empirical observation: from cells to biomes, biological systems are large, complex, and often stable. In this paper, we revisit May's argument in light of recent developments in both ecology and random matrix theory. We focus on competitive systems, and, using a nonlinear generalization of the competitive Lotka-Volterra model, we show that there are, in fact, two kinds of complexity-stability relationships in disordered dynamical systems: if self-interactions grow faster with density than cross-interactions, complexity is destabilizing; but if cross-interactions grow faster than self-interactions, complexity is stabilizing.

cond-mat.stat-mech↗

arXiv2024

Kendall Correlation Coefficients for Portfolio Optimization

Markowitz's optimal portfolio relies on the accurate estimation of correlations between asset returns, a difficult problem when the number of observations is not much larger than the number of assets. Using powerful results from random matrix theory, several schemes have been developed to "clean" the eigenvalues of empirical correlation matrices. By contrast, the (in practice equally important) problem of correctly estimating the eigenvectors of the correlation matrix has received comparatively little attention. Here we discuss a class of correlation estimators generalizing Kendall's rank correlation coefficient which improve the estimation of both eigenvalues and eigenvectors in data-poor regimes. Using both synthetic and real financial data, we show that these generalized correlation coefficients yield Markowitz portfolios with lower out-of-sample risk than those obtained with rotationally invariant estimators. Central to these results is a property shared by all Kendall-like estimators but not with classical correlation coefficients: zero eigenvalues only appear when the number of assets becomes proportional to the square of the number of data points.

q-fin.ST↗

arXiv2024

Great year, bad Sharpe? A note on the joint distribution of performance and risk-adjusted return

Returns distributions are heavy-tailed across asset classes. In this note, I examine the implications of this well-known stylized fact for the joint statistics of performance (absolute return) and Sharpe ratio (risk-adjusted return). Using both synthetic and real data, I show that, all other things being equal, the investments with the best in-sample performance are never associated with the best in-sample Sharpe ratios (and vice versa). This counter-intuitive effect is unrelated to the risk-return tradeoff familiar from portfolio theory: it is, rather, a consequence of asymptotic correlations between the sample mean and sample standard deviation of heavy-tailed variables. In addition to its large sample noise, this non-monotonic association of the Sharpe ratio with performance puts into question its status as the gold standard metric of investment quality.

q-fin.ST↗

arXiv2022

Diversity enables the jump towards cooperation for the Traveler's Dilemma

Social dilemmas are situations in which collective welfare is at odds with individual gain. One widely studied example, due to the conflict it poses between human behaviour and game theoretic reasoning, is the Traveler's Dilemma. The dilemma relies on the players' incentive to undercut their opponent at the expense of losing a collective high payoff. Such individual incentive leads players to a systematic mutual undercutting until the lowest possible payoff is reached, which is the game's unique Nash equilibrium. However, if players were satisfied with a high payoff -- that is not necessarily higher than their opponent's -- they would both be better off individually and collectively. Here, we explain how it is possible to converge to this cooperative high payoff equilibrium. Our analysis focuses on decomposing the dilemma into a local and a global game. We show that players need to escape the local maximisation and jump to the global game, in order to reach the cooperative equilibrium. Using a dynamic approach, based on evolutionary game theory and learning theory models, we find that diversity, understood as the presence of suboptimal strategies, is the general mechanism that enables the jump towards cooperation.

cs.GT↗

arXiv2022

Perturbation theory without power series: iterative construction of non-analytic operator spectra

It is well known that quantum-mechanical perturbation theory often give rise to divergent series that require proper resummation. Here I discuss simple ways in which these divergences can be avoided in the first place. Using the elementary technique of relaxed fixed-point iteration, I obtain convergent expressions for various challenging ground states wavefunctions, including quartic, sextic and octic anharmonic oscillators, the hydrogenic Zeeman problem, and the Herbst-Simon Hamiltonian (with finite energy but vanishing Rayleigh-Schrödinger coefficients), all at arbitarily strong coupling. These results challenge the notion that non-analytic functions of coupling constants are intrinsically "non-perturbative". A possible application to exact diagonalization is briefly discussed.

quant-ph↗

arXiv2022

A fast iterative algorithm for near-diagonal eigenvalue problems

We introduce a novel eigenvalue algorithm for near-diagonal matrices inspired by Rayleigh-Schrödinger perturbation theory and termed Iterative Perturbative Theory (IPT). Contrary to standard eigenvalue algorithms, which are either 'direct' (to compute all eigenpairs) or 'iterative' (to compute just a few), IPT computes any number of eigenpairs with the same basic iterative procedure. Thanks to this perfect parallelism, IPT proves more efficient than classical methods (LAPACK or CUSOLVER for the full-spectrum problem, preconditioned Davidson solvers for extremal eigenvalues). We give sufficient conditions for linear convergence and demonstrate performance on dense and sparse test matrices, including one from quantum chemistry.

math.NA↗

arXiv2021

Diffusion approximations in population genetics and the rate of Muller's ratchet

Diffusion theory is a central tool of modern population genetics, yielding simple expressions for fixation probabilities and other quantities that are not easily derived from the underlying Wright-Fisher model. Unfortunately, the textbook derivation of diffusion equations as scaling limits requires evolutionary parameters (selection coefficients, mutation rates) to scale like the inverse population size -- a severe restriction that does not always reflect biological reality. Here we note that the Wright-Fisher model can be approximated by diffusion equations under more general conditions, including in regimes where selection and/or mutation are strong compared to genetic drift. As an illustration, we use a diffusion approximation of the Wright-Fisher model to improve estimates for the expected time to fixation of a strongly deleterious allele, i.e. the rate of Muller's ratchet.

q-bio.PE↗

arXiv2021

The cost of noise: stochastic punishment falls short of sustaining cooperation in social dilemma experiments

Identifying mechanisms able to sustain costly cooperation among self-interested agents is a central problem across social and biological sciences. One possible solution is peer punishment: when agents have an opportunity to sanction defectors, classical behavioral experiments suggest that cooperation can take root. Overlooked from standard experimental designs, however, is the fact that real-world human punishment -- the administration of justice -- is intrinsically noisy. Here we show that stochastic punishment falls short of sustaining cooperation in the repeated public good game. As punishment noise increases, we find that contributions decrease and punishment efforts intensify, resulting in a $45\%$ drop in gains compared to a noiseless control. Moreover, we observe that uncertainty causes a rise in antisocial punishment, a mutually harmful behavior previously associated with societies with a weak rule of law. Our approach brings to light challenges to cooperation that cannot be explained by economic rationality and strengthens the case for further investigations of the effect of noise -- and not just bias -- on human behavior.

physics.soc-ph↗

arXiv2020

Intermittency as metastability: a predictive approach to evolution in disordered environments

Many systems across the sciences evolve through a combination of multiplicative growth and diffusive transport. In the presence of disorder, these systems tend to form localized structures which alternate between long periods of relative stasis and short bursts of activity. This behaviour, known as intermittency in physics and punctuated equilibrium in evolutionary theory, is difficult to forecast; in particular there is no general principle to locate the regions where the system will settle, how long it will stay there, or where it will jump next. Here I introduce a predictive theory of linear intermittency that closes these gaps. I show that any positive linear system can be mapped onto a generalization of the "maximal entropy random walk", a Markov process on graphs with non-local transition rates. This construction reveals the localization islands as local minima of an effective potential, and intermittent jumps as barrier crossings in that potential. My results unify the concepts of intermittency in linear systems and Markovian metastability, and provide a generally applicable method to reduce, and predict, the dynamics of disordered linear systems. Applications span physics, evolutionary dynamics and epidemiology.

cond-mat.stat-mech↗

arXiv2020

Dynamical perturbation theory for eigenvalue problems

Many problems in physics, chemistry and other fields are perturbative in nature, i.e. differ only slightly from related problems with known solutions. Prominent among these is the eigenvalue perturbation problem, wherein one seeks the eigenvectors and eigenvalues of a matrix with small off-diagonal elements. Here we introduce a novel iterative algorithm to compute these eigenpairs based on fixed-point iteration for an algebraic equation in complex projective space. We show from explicit and numerical examples that our algorithm outperforms the usual Rayleigh-Schrödinger expansion on three counts. First, since it is not defined as a power series, its domain of convergence is not a priori confined to a disk in the complex plane; we find that it indeed usually extends beyond the standard perturbative radius of convergence. Second, it converges at a faster rate than the Rayleigh-Schrödinger expansion, i.e. fewer iterations are required to reach a given precision. Third, the (time- and space-) algorithmic complexity of each iteration does not increase with the order of the approximation, allowing for higher precision computations. Because this complexity is merely that of matrix multiplication, our dynamical scheme also scales better with the size of the matrix than general-purpose eigenvalue routines such as the shifted QR or divide-and-conquer algorithms. Whether they are dense, sparse, symmetric or unsymmetric, we confirm that dynamical diagonalization quickly outpaces LAPACK drivers as the size of matrices grows; for the computation of just the dominant eigenvector, our method converges order of magnitudes faster than the Arnoldi algorithm implemented in ARPACK.

math-ph↗

arXiv2019

Effective potential reveals evolutionary trajectories in complex fitness landscapes

Growing efforts to measure fitness landscapes in molecular and microbial systems are premised on a tight relationship between landscape topography and evolutionary trajectories. This relationship, however, is far from being straightforward: depending on their mutation rate, Darwinian populations can climb the closest fitness peak (survival of the fittest), settle in lower regions with higher mutational robustness (survival of the flattest), or fail to adapt altogether (error catastrophes). These bifurcations highlight that evolution does not necessarily drive populations "from lower peak to higher peak", as Wright imagined. The problem therefore remains: how exactly does a complex landscape topography constrain evolution, and can we predict where it will go next? Here I introduce a generalization of quasispecies theory which identifies metastable evolutionary states as minima of an effective potential. From this representation I derive a coarse-grained, Markov state model of evolution, which in turn forms a basis for evolutionary predictions across a wide range of mutation rates. Because the effective potential is related to the ground state of a quantum Hamiltonian, my approach could stimulate fruitful interactions between evolutionary dynamics and quantum many-body theory

q-bio.PE↗

arXiv2017

Natural selection as coarsening

Analogies between evolutionary dynamics and statistical mechanics, such as Fisher's second-law-like "fundamental theorem of natural selection" and Wright's "fitness landscapes", have had a deep and fruitful influence on the development of evolutionary theory. Here I discuss a new conceptual link between evolution and statistical physics. I argue that natural selection can be viewed as a coarsening phenomenon, similar to the growth of domain size in quenched magnets or to Ostwald ripening in alloys and emulsions. In particular, I show that the most remarkable features of coarsening---scaling and self-similarity---have strict equivalents in evolutionary dynamics. This analogy has three main virtues: it brings a set of well-developed mathematical tools to bear on evolutionary dynamics; it suggests new problems in theoretical evolution; and it provides coarsening physics with a new exactly soluble model.

q-bio.PE↗

arXiv2016

Universal statistics of selected values

Selection, the tendency of some traits to become more frequent than others in a population under the influence of some (natural or artificial) agency, is a key component of Darwinian evolution and countless other natural and social phenomena. Yet a general theory of selection, analogous to the Fisher-Tippett-Gnedenko theory of extreme events, is lacking. Here we introduce a probabilistic definition of selection and show that selected values are attracted to a universal family of limiting distributions. The universality classes and scaling exponents are determined by the tail thickness of the random variable under selection. Our results are supported by data from molecular biology, agriculture and sport.

q-bio.PE↗

arXiv2016

A non-equilibrium formulation of food security resilience

Resilience, the ability to recover from adverse events ("shocks"), is of fundamental importance to food security. This is especially true in poor countries, where basic needs are frequently threatened by economic, environmental, and health shocks. An empirically sound formalization of the concept of food security resilience, however, is lacking. Here we introduce a general framework for quantifying resilience based on a simple definition: a unit is resilient if $(a)$ its long-term food security trend is not deteriorating and $(b)$ the effects of shocks on this trend do not persist over time. Our approach can be applied to any food security variable for which high-frequency time-series data is available, can accommodate any unit of analysis (e.g., individuals, households, countries), and is especially useful in rapidly changing contexts wherein standard equilibrium-based economic models are ineffective. We illustrate our method with an analysis of per capita kilocalorie availability for 161 countries between 1961 and 2011. We find that resilient countries are not necessarily those that are characterized by high levels or less volatile fluctuations of kilocalorie intake. Accordingly, food security policies and programs will need to be tailored not only to welfare levels at any one time, but also to long-run welfare dynamics.

econ.GN↗

arXiv2016

Minimum relative entropy distributions with a large mean are Gaussian

We consider the following frustrated optimization problem: given a prior probability distribution $q$, find the distribution $p$ minimizing the relative entropy with respect to $q$ such that $\textrm{mean}(p)$ is fixed and large. We show that solutions to this problem are asymptotically Gaussian. As an application we derive an $H$-type theorem for evolutionary dynamics: the entropy of the (standardized) distribution of fitness of a population evolving under natural selection is eventually increasing.

cond-mat.stat-mech↗

arXiv2016

Limiting fitness distributions in evolutionary dynamics

Darwinian evolution can be modeled in general terms as a flow in the space of fitness (i.e. reproductive rate) distributions. In the diffusion approximation, Tsimring et al. have showed that this flow admits "fitness wave" solutions: Gaussian-shape fitness distributions moving towards higher fitness values at constant speed. Here we show more generally that evolving fitness distributions are attracted to a one-parameter family of distributions with a fixed parabolic relationship between skewness and kurtosis. Unlike fitness waves, this statistical pattern encompasses both positive and negative (a.k.a. purifying) selection and is not restricted to rapidly adapting populations. Moreover we find that the mean fitness of a population under the selection of pre-existing variation is a power-law function of time, as observed in microbiological evolution experiments but at variance with fitness wave theory. At the conceptual level, our results can be viewed as the resolution of the "dynamic insufficiency" of Fisher's fundamental theorem of natural selection. Our predictions are in good agreement with numerical simulations.

q-bio.PE↗