arXiv subjects

Adam Eisen

Publications and source records attributed to Adam Eisen.

3 recordsLinked to original sources

arXiv2024

Delay Embedding Theory of Neural Sequence Models

To generate coherent responses, language models infer unobserved meaning from their input text sequence. One potential explanation for this capability arises from theories of delay embeddings in dynamical systems, which prove that unobserved variables can be recovered from the history of only a handful of observed variables. To test whether language models are effectively constructing delay embeddings, we measure the capacities of sequence models to reconstruct unobserved dynamics. We trained 1-layer transformer decoders and state-space sequence models on next-step prediction from noisy, partially-observed time series data. We found that each sequence layer can learn a viable embedding of the underlying system. However, state-space models have a stronger inductive bias than transformers-in particular, they more effectively reconstruct unobserved information at initialization, leading to more parameter-efficient models and lower error on dynamics tasks. Our work thus forges a novel connection between dynamical systems and deep learning sequence models via delay embedding theory.

cs.LG↗

arXiv2023

Beyond Geometry: Comparing the Temporal Structure of Computation in Neural Circuits with Dynamical Similarity Analysis

How can we tell whether two neural networks utilize the same internal processes for a particular computation? This question is pertinent for multiple subfields of neuroscience and machine learning, including neuroAI, mechanistic interpretability, and brain-machine interfaces. Standard approaches for comparing neural networks focus on the spatial geometry of latent states. Yet in recurrent networks, computations are implemented at the level of dynamics, and two networks performing the same computation with equivalent dynamics need not exhibit the same geometry. To bridge this gap, we introduce a novel similarity metric that compares two systems at the level of their dynamics, called Dynamical Similarity Analysis (DSA). Our method incorporates two components: Using recent advances in data-driven dynamical systems theory, we learn a high-dimensional linear system that accurately captures core features of the original nonlinear dynamics. Next, we compare different systems passed through this embedding using a novel extension of Procrustes Analysis that accounts for how vector fields change under orthogonal transformation. In four case studies, we demonstrate that our method disentangles conjugate and non-conjugate recurrent neural networks (RNNs), while geometric methods fall short. We additionally show that our method can distinguish learning rules in an unsupervised manner. Our method opens the door to comparative analyses of the essential temporal structure of computation in neural circuits.

q-bio.NC↗

arXiv2018

A Lattice Model of Charge-Pattern-Dependent Polyampholyte Phase Separation

In view of recent intense experimental and theoretical interests in the biophysics of liquid-liquid phase separation (LLPS) of intrinsically disordered proteins (IDPs), heteropolymer models with chain molecules configured as self-avoiding walks on the simple cubic lattice are constructed to study how phase behaviors depend on the sequence of monomers along the chains. To address pertinent general principles, we focus primarily on two fully charged 50-monomer sequences with significantly different charge patterns. Each monomer in our models occupies a single lattice site and all monomers interact via a screened pairwise Coulomb potential. Phase diagrams are obtained by extensive Monte Carlo sampling performed at multiple temperatures on ensembles of 300 chains in boxes of sizes ranging from $52\times 52\times 52$ to $246\times 246\times 246$ to simulate a large number of different systems with the overall polymer volume fraction $\phi$ in each system varying from $0.001$ to $0.1$. Phase separation in the model systems is characterized by the emergence of a large cluster connected by inter-monomer nearest-neighbor lattice contacts and by large fluctuations in local polymer density. The simulated critical temperatures, $T_{\rm cr}$, of phase separation for the two sequences differ significantly, whereby the sequence with a more "blocky" charge pattern exhibits a substantially higher propensity to phase separate. The trend is consistent with our sequence-specific random-phase-approximation (RPA) polymer theory, but the variation of the simulated $T_{\rm cr}$ with a previously proposed "sequence charge decoration" pattern parameter is milder than that predicted by RPA. Ramifications of our findings for the development of analytical theory and simulation protocols of IDP LLPS are discussed.

q-bio.BM↗