arXiv subjects

Ilya Nemenman

Publications and source records attributed to Ilya Nemenman.

At least 19 recordsLinked to original sources

arXiv2026

Multi-Agent Empowerment and Emergence of Complex Behavior in Groups

Intrinsic motivations are receiving increasing attention, i.e. behavioral incentives that are not engineered, but emerge from the interaction of an agent with its surroundings. In this work we study the emergence of behaviors driven by one such incentive, empowerment, specifically in the context of more than one agent. We formulate a principled extension of empowerment to the multi-agent setting, and demonstrate its efficient calculation. We observe that this intrinsic motivation gives rise to characteristic modes of group-organization in two qualitatively distinct environments: a pair of agents coupled by a tendon, and a controllable Vicsek flock. This demonstrates the potential of intrinsic motivations such as empowerment to not just drive behavior for only individual agents but also higher levels of behavioral organization at scale.

cs.AI↗

arXiv2026

You ain't seen nothing, and yet: Future biochemical concentrations can be predicted with surprisingly high accuracy

Accurate sensing of chemical concentrations is essential for numerous biological processes. The accuracy of this sensing, for small numbers of molecules, is limited by shot noise. Corresponding theoretical limits on sensing precision, as a function of sensing duration, have been well-studied in the context of quasi-static and randomly fluctuating concentrations. However, during development and in many other cases, concentration profiles are not random but exhibit predictable spatiotemporal patterns. We propose that leveraging prior knowledge of these structured profiles can improve and accelerate concentration sensing by utilizing information from current molecular binding events to predict future concentrations. By framing the constrained sensing problem as Bayesian inference over an allowed class of spatiotemporal profiles, we derive new theoretical limits on sensing accuracy. Our analysis reveals that maximum a posteriori (MAP) estimation can outperform the classical Berg-Purcell and maximum-likelihood (Poisson counting) limits, achieving a sensing precision of $δc/c = 1/\sqrt{a^2N}$, where $N$ is the number of binding events, and $a > 1$ in certain cases. Thus knowledge of the statistical structure of concentration profiles enhances sensing precision, providing a potential explanation for the rapid yet highly accurate cell fate decisions observed during development.

physics.bio-ph↗

arXiv2026

Accurate Estimation of Mutual Information in High Dimensional Data

Mutual information (MI) quantifies statistical dependence between variables and is widely used across scientific disciplines, yet accurate estimation from finite data remains notoriously difficult. Common approaches fail in high-dimensional, undersampled regimes ($N \lesssim K$) typical of modern experiments, and no accepted tests exist to detect when neural network-based estimators fail, making them effectively unusable as scientific instruments. We show that neural MI estimators can be made reliable when the statistical dependencies admit a low-dimensional latent representation. Sample complexity is then governed by the latent dimensionality $K_Z \ll K$ rather than the ambient dimension -- a regime shift we confirm empirically and ground theoretically via random matrix theory. Building on this insight, we develop a practical protocol that provides neural estimators with explicit statistical consistency checks, bias correction, and confidence intervals. We additionally introduce a new class of probabilistic critics (the VSIB family) that substantially reduce bias and variance at higher MI values where standard estimators break down. We validate the protocol on synthetic benchmarks ($K=500$, $N$ as low as $256$), on the standard 40-dataset benchmark suite of Czyz et al. (2023), on noisy MNIST ($K=784$), and on CIFAR-10/100 ($K=3072$) with a ResNet-20 backbone. Our protocol consistently matches or exceeds existing methods while being the only approach to report confidence intervals and flag unreliable estimates, achieving reliable MI detection well below the ambient pixel dimension on real images.

physics.data-an↗

arXiv2026

Information bottleneck for learning the phase space of dynamics from high-dimensional experimental data

Identifying the dynamical state variables of a system from high-dimensional observations is a central problem across physical sciences. The challenge is that the state variables are not directly observable and must be inferred from raw high-dimensional data without supervision. Here we introduce DySIB (Dynamical Symmetric Information Bottleneck) as a method to learn low-dimensional representations of time-series data by maximizing predictive mutual information between past and future observation windows while penalizing representation complexity. This objective operates entirely in latent space and avoids reconstruction of the observations. We apply DySIB to an experimental video dataset of a physical pendulum, where the underlying state space is known. The method, with hyperparameters of the learning architecture set self-consistently by the data, recovers a two-dimensional representation that matches the dimensionality, topology, and geometry of the pendulum phase space, with the learned coordinates aligning smoothly with the canonical angle and angular velocity. These results demonstrate, on a well-characterized experimental system, that predictive information in latent space can be used to recover interpretable dynamical coordinates directly from high-dimensional data.

physics.data-an↗

arXiv2026

Contrast encodes inductive bias: separating slow noise from dynamics in predictive representation learning

Self-supervised methods that learn representations and predict dynamics fully in the latent space, such as JEPA, have been shown to confuse slowly varying noise with the dynamical signals they aim to capture. Specifically, when noise features remain approximately constant within each trajectory, contrastive predictive objectives preferentially encode these features instead of the true latent variables governing the system. The learned representation then becomes dominated by trajectory-specific noise, so downstream performance degrades with noise strength and does not improve even as the number and duration of training trajectories increase. We argue that this failure is a property of the objective itself, shared by a long line of contrastive predictive objectives that sample negatives across trajectories. To illustrate this generality, we study the failure mode and its remedy in two settings: a standard SimCLR-style JEPA on a synthetic moving-dot dataset, and DySIB, a recently introduced method designed for extracting physically interpretable representations of dynamics, on movies of a rigid-body pendulum. When negatives are instead sampled within a single trajectory, the slow noise can no longer distinguish frames within that trajectory, removing the predictive shortcut. Training one encoder simultaneously on many such trajectories then forces it to encode the variables relevant for the dynamics, with longer trajectories yielding better representations even for strong slow noise. Our results point toward principles for designing contrastive predictive objectives in dynamical representation learning, especially for physical systems with noisy experimental observations.

cs.LG↗

arXiv2026

DustNET: enabling machine learning and AI models of dusty plasmas

Dusty plasmas are ubiquitous throughout the universe, spanning laboratory and industrial plasmas, fusion devices, planetary environments, cometary comae, and interstellar media. Despite decades of research, many aspects of their behavior remain poorly understood within a unified framework. While numerous theoretical and numerical models describe specific phenomena, such as dust charging, transport, waves, and self-organization, fully predictive models across the wide range of spatial and temporal scales in both laboratory and natural systems remain elusive. Conventional plasma descriptions rely on coupled differential equations for particle densities, momenta, and energies, but their solutions are often limited by computational cost, numerical uncertainties, and incomplete knowledge of boundary conditions and transport processes. Recent advances in machine learning (ML), particularly deep neural networks, offer new opportunities to complement traditional physics-based modeling. Here we review ML and artificial intelligence (AI) approaches, termed bottom-up data-driven methods, for dusty plasma research. Central to this effort is Dust Neural nEtworks Technology (DustNET), a community-driven dataset initiative inspired by ImageNet, integrating experimental, simulation, and synthetic data to enable predictive modeling, uncertainty quantification, and multi-scale analysis. DustNET-trained models may also be deployed in real-time experimental settings under edge computing constraints. Combined with emerging multi-modal AI foundation models and autonomous agents, this framework provides a pathway toward a unified, physics-informed understanding of dusty plasmas across laboratory, industrial, space, and astrophysical environments.

physics.plasm-ph↗

arXiv2026

Better Together: Cross and Joint Covariances Enhance Signal Detectability in Undersampled Data

Many data-science applications involve detecting a shared signal between two high-dimensional variables. Using random matrix theory methods, we determine when such signal can be detected and reconstructed from sample correlations, despite the background of sampling noise induced correlations. We consider three different covariance matrices constructed from two high-dimensional variables: their individual self covariance, their cross covariance, and the self covariance of the concatenated (joint) variable, which incorporates the self and the cross correlation blocks. We observe the expected Baik, Ben Arous, and Péché detectability phase transition in all these covariance matrices, and we show that joint and cross covariance matrices always reconstruct the shared signal earlier than the self covariances. Whether the joint or the cross approach is better depends on the mismatch of dimensionalities between the variables. We discuss what these observations mean for choosing the right method for detecting linear correlations in data and how these findings may generalize to nonlinear statistical dependencies.

cond-mat.dis-nn↗

arXiv2026

Mutual information and task-relevant latent dimensionality

Estimating the dimensionality of the latent representation needed for prediction -- the task-relevant dimension -- is a difficult, largely unsolved problem with broad scientific applications. We cast it as an Information Bottleneck question: what embedding bottleneck dimension is sufficient to compress predictor and predicted views while preserving their mutual information (MI). This repurposes neural MI estimators for dimensionality estimation. We show that standard neural estimators with separable/bilinear critics systematically inflate the inferred dimension, and we address this by introducing a hybrid critic that retains an explicit dimensional bottleneck while allowing flexible nonlinear cross-view interactions, thereby preserving the latent geometry. We further propose a one-shot protocol that reads off the effective dimension from a single over-parameterized hybrid model, without sweeping over bottleneck sizes. We validate the approach on synthetic problems with known task-relevant dimension. We extend the approach to intrinsic dimensionality by constructing paired views of a single dataset, enabling comparison with classical geometric dimension estimators. In noisy regimes where those estimators degrade, our approach remains reliable. Finally, we demonstrate the utility of the method on multiple physics datasets.

cs.LG↗

arXiv2025

Randomness with constraints: constructing minimal models for high-dimensional biology

Biologists and physicists have a rich tradition of modeling living systems with simple models composed of a few interacting components. Despite the remarkable success of this approach, it remains unclear how to use such finely tuned models to study complex biological systems composed of numerous heterogeneous, interacting components. One possible strategy for taming this biological complexity is to embrace the idea that many biological behaviors we observe are ``typical'' and can be modeled using random systems that respect biologically-motivated constraints. Here, we review recent works showing how this approach can be used to make close connection with experiments in biological systems ranging from neuroscience to ecology and evolution and beyond. Collectively, these works suggest that the ``random-with-constraints'' paradigm represents a promising new modeling strategy for capturing experimentally observed dynamical and statistical features in high-dimensional biological data and provides a powerful minimal modeling philosophy for biology.

physics.bio-ph↗

arXiv2025

Deep Variational Multivariate Information Bottleneck -- A Framework for Variational Losses

Variational dimensionality reduction methods are widely used for their accuracy, generative capabilities, and robustness. We introduce a unifying framework that generalizes both such as traditional and state-of-the-art methods. The framework is based on an interpretation of the multivariate information bottleneck, trading off the information preserved in an encoder graph (defining what to compress) against that in a decoder graph (defining a generative model for data). Using this approach, we rederive existing methods, including the deep variational information bottleneck, variational autoencoders, and deep multiview information bottleneck. We naturally extend the deep variational CCA (DVCCA) family to beta-DVCCA and introduce a new method, the deep variational symmetric information bottleneck (DVSIB). DSIB, the deterministic limit of DVSIB, connects to modern contrastive learning approaches such as Barlow Twins, among others. We evaluate these methods on Noisy MNIST and Noisy CIFAR-100, showing that algorithms better matched to the structure of the problem like DVSIB and beta-DVCCA produce better latent spaces as measured by classification accuracy, dimensionality of the latent variables, sample efficiency, and consistently outperform other approaches under comparable conditions. Additionally, we benchmark against state-of-the-art models, achieving superior or competitive accuracy. Our results demonstrate that this framework can seamlessly incorporate diverse multi-view representation learning algorithms, providing a foundation for designing novel, problem-specific loss functions.

cs.LG↗

arXiv2025

Distribution of singular values in large sample cross-covariance matrices

For two large matrices ${\mathbf X}$ and ${\mathbf Y}$ with Gaussian i.i.d.\ entries and dimensions $T\times N_X$ and $T\times N_Y$, respectively, we derive the probability distribution of the singular values of $\mathbf{X}^T \mathbf{Y}$ in different parameter regimes. This extends the Marchenko-Pastur result for the distribution of eigenvalues of empirical sample covariance matrices to singular values of empirical cross-covariances. Our results will help to establish statistical significance of cross-correlations in many data-science applications.

math.ST↗

arXiv2025

Physics-tailored machine learning reveals unexpected physics in dusty plasmas

Dusty plasma is a mixture of ions, electrons, and macroscopic charged particles that is commonly found in space and planetary environments. The particles interact through Coulomb forces mediated by the surrounding plasma, and as a result, the effective forces between particles can be non-conservative and non-reciprocal. Machine learning (ML) models are a promising route to learn these complex forces, yet their structure should match the underlying physical constraints to provide useful insight. Here we demonstrate and experimentally validate an ML approach that incorporates physical intuition to infer force laws in a laboratory dusty plasma. Trained on 3D particle trajectories, the model accounts for inherent symmetries, non-identical particles, and learns the effective non-reciprocal forces between particles with exquisite accuracy (R^2>0.99). We validate the model by inferring particle masses in two independent yet consistent ways. The model's accuracy enables precise measurements of particle charge and screening length, discovering large deviations from common theoretical assumptions. Our ability to identify new physics from experimental data demonstrates how ML-powered approaches can guide new routes of scientific discovery in many-body systems. Furthermore, we anticipate our ML approach to be a starting point for inferring laws from dynamics in a wide range of many-body systems, from colloids to living organisms.

physics.plasm-ph↗

arXiv2024

Simultaneous Dimensionality Reduction: A Data Efficient Approach for Multimodal Representations Learning

We explore two primary classes of approaches to dimensionality reduction (DR): Independent Dimensionality Reduction (IDR) and Simultaneous Dimensionality Reduction (SDR). In IDR methods, of which Principal Components Analysis is a paradigmatic example, each modality is compressed independently, striving to retain as much variation within each modality as possible. In contrast, in SDR, one simultaneously compresses the modalities to maximize the covariation between the reduced descriptions while paying less attention to how much individual variation is preserved. Paradigmatic examples include Partial Least Squares and Canonical Correlations Analysis. Even though these DR methods are a staple of statistics, their relative accuracy and data set size requirements are poorly understood. We introduce a generative linear model to synthesize multimodal data with known variance and covariance structures to examine these questions. We assess the accuracy of the reconstruction of the covariance structures as a function of the number of samples, signal-to-noise ratio, and the number of varying and covarying signals in the data. Using numerical experiments, we demonstrate that linear SDR methods consistently outperform linear IDR methods and yield higher-quality, more succinct reduced-dimensional representations with smaller datasets. Remarkably, regularized CCA can identify low-dimensional weak covarying structures even when the number of samples is much smaller than the dimensionality of the data, which is a regime challenging for all dimensionality reduction methods. Our work corroborates and explains previous observations in the literature that SDR can be more effective in detecting covariation patterns in data. These findings suggest that SDR should be preferred to IDR in real-world data analysis when detecting covariation is more important than preserving variation.

stat.ML↗

arXiv2024

Generative random latent features models and statistics of natural images

Complex, multivariable systems are often analyzed by grouping their constituent units into components, sometimes referred to as latent features, which afford physical or biological interpretation. However, a priori many different types of latent features and data decompositions can be defined, and one typically uses a trial and error approach to determine a decomposition that is natural to the system and its data. It is highly desirable to develop principled understanding of which decomposition is appropriate for given a data set. In this work, we take a step in this direction and argue that sample-sample correlations in the data carry important information to this effect. For this we construct a generative random latent feature matrix model of large data based on linear mixing of latent features. Key ingredient of our model is that we allow for statistical dependence between the mixing coefficients and argue that the model captures characteristic properties found in many types of natural data. Latent dimensionality and correlation patterns of the data are controlled by only two model parameters. The model's data patterns include (overlapping) clusters, sparse mixing, and constrained (non-negative) mixing. We describe the characteristic correlation and eigenvalue distributions of each pattern. Finally, we fit the model on correlation data from natural images and find a near perfect match with the sparse mixing regime of our model. This finding is in line with the well-known sparse coding structure in natural scene images and provides information about the appropriate data decomposition, namely a sparse coding scheme. We believe that our work will deliver similar insights for diverse data of biological systems.

cond-mat.dis-nn↗

arXiv2024

Machine learning that predicts well may not learn the correct physical descriptions of glassy systems

The complexity of glasses makes it challenging to explain their dynamics. Machine Learning (ML) has emerged as a promising pathway for understanding glassy dynamics by linking their structural features to rearrangement dynamics. Support Vector Machine (SVM) was one of the first methods used to detect such correlations. Specifically, a certain output of SVMs trained to predict dynamics from structure, the distance from the separating hyperplane, was interpreted as being linearly related to the activation energy for the rearrangement. By numerical analysis of toy models, we explore under which conditions it is possible to infer the energy barrier to rearrangements from the distance to the separating hyperplane. We observe that such successful inference is possible only under very restricted conditions. Typical tests, such as the apparent Arrhenius dependence of the probability of rearrangement on the inferred energy and the temperature, or high cross-validation accuracy do not guarantee success. We propose practical approaches for measuring the quality of the energy inference and for modifying the inferred model to improve the inference, which should be usable in the context of realistic datasets.

cond-mat.soft↗

arXiv2024

Data efficiency, dimensionality reduction, and the generalized symmetric information bottleneck

The Symmetric Information Bottleneck (SIB), an extension of the more familiar Information Bottleneck, is a dimensionality reduction technique that simultaneously compresses two random variables to preserve information between their compressed versions. We introduce the Generalized Symmetric Information Bottleneck (GSIB), which explores different functional forms of the cost of such simultaneous reduction. We then explore the dataset size requirements of such simultaneous compression. We do this by deriving bounds and root-mean-squared estimates of statistical fluctuations of the involved loss functions. We show that, in typical situations, the simultaneous GSIB compression requires qualitatively less data to achieve the same errors compared to compressing variables one at a time. We suggest that this is an example of a more general principle that simultaneous compression is more data efficient than independent compression of each of the input variables.

cs.IT↗

arXiv2023

Inferring Local Structure from Pairwise Correlations

To construct models of large, multivariate complex systems, such as those in biology, one needs to constrain which variables are allowed to interact. This can be viewed as detecting "local" structures among the variables. In the context of a simple toy model of 2D natural and synthetic images, we show that pairwise correlations between the variables -- even when severely undersampled -- provide enough information to recover local relations, including the dimensionality of the data, and to reconstruct arrangement of pixels in fully scrambled images. This proves to be successful even though higher order interaction structures are present in our data. We build intuition behind the success, which we hope might contribute to modeling complex, multivariate systems and to explaining the success of modern attention-based machine learning approaches.

physics.data-an↗

arXiv2023

Neural criticality from effective latent variables

Observations of power laws in neural activity data have raised the intriguing notion that brains may operate in a critical state. One example of this critical state is "avalanche criticality," which has been observed in various systems, including cultured neurons, zebrafish, rodent cortex, and human EEG. More recently, power laws were also observed in neural populations in the mouse under an activity coarse-graining procedure, and they were explained as a consequence of the neural activity being coupled to multiple latent dynamical variables. An intriguing possibility is that avalanche criticality emerges due to a similar mechanism. Here, we determine the conditions under which latent dynamical variables give rise to avalanche criticality. We find that populations coupled to multiple latent variables produce critical behavior across a broader parameter range than those coupled to a single, quasi-static latent variable, but in both cases, avalanche criticality is observed without fine-tuning of model parameters. We identify two regimes of avalanches, both critical but differing in the amount of information carried about the latent variable. Our results suggest that avalanche criticality arises in neural systems in which activity is effectively modeled as a population driven by a few dynamical variables and these variables can be inferred from the population activity.

q-bio.NC↗