Searcharxiv⌕ Search

arXiv subjects

Vasudev Shyam

Publications and source records attributed to Vasudev Shyam.

At least 19 recordsLinked to original sources
arXiv2026

Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds

What is the geometry of a visual percept? The most widely used protocols for decomposing neural network representations into interpretable parts treat concepts as isolated directions, yet recent work shows that concepts are often realized as geometric structures in low dimensional regions of activation space. We turn to the literature of Structured sparsity to close this gap, and show that block sparsity, which groups directions into blocks, is the prior matched to a generative model in which a representation is a sparse sum of low-dimensional manifolds: the modern, learned form of a classical idea in visual neuroscience, where a visual feature is carried by a coordinated group of neurons rather than a single tuned one. We implement three variants of block-sparse featurizers (BSFs) and, through a minimum-description-length analysis, show that all three describe activations more compactly than direction-based featurizers, with the recovered concepts typically two- to four-dimensional. We then use BSFs to (i) recontextualize prior work, showing that curve detectors in InceptionV1 actually read from a single continuous curve manifold, (ii) discover novel manifolds including shadows and lighting in DINOv3, and (iii) support interpretable control of image generation in diffusion models (SDXL) via manifold steering.

cs.CV↗
arXiv2026

Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior

Neural representations carry rich geometric structure; but does that structure causally shape behavior? To address this question, we intervene along paths through activation space defined by different geometries, and measure the behavioral trajectories they induce. In particular, we test whether interventions that respect the geometry of activation space will yield behaviors close to those the model exhibits naturally. Concretely, we first fit an activation manifold $M_h$ to representations and a behavior manifold $M_y$ to output probability distributions. We then test the link $M_h \leftrightarrow M_y$ via interventions: we find that steering along $M_h$, which we term manifold steering, yields behavioral trajectories that follow $M_y$, while linear steering -- which assumes a Euclidean geometry -- cuts through off-manifold regions and hence produces unnatural outputs. Moreover, optimizing interventions in activation space to produce paths along $M_y$ recovers activation trajectories that trace the curvature of $M_h$. We demonstrate this bidirectional relationship between the geometry of representation and behavior across tasks and modalities. In language models, we use reasoning tasks with cyclic and sequential geometries as well as in-context learning tasks with more complex graph geometries. In a video world model, we use a task with geometry corresponding to physical dynamics. Overall, our work shows that geometry in neural representation is not merely incidental, but is in fact the proper object for enabling principled control via intervention on internals. This recasts the core problem of steering from finding the right direction to finding the right geometry.

cs.LG↗
arXiv2026

Do Sparse Autoencoders Capture Concept Manifolds?

Sparse autoencoders (SAEs) are widely used to extract interpretable features from neural network representations, often under the implicit assumption that concepts correspond to independent linear directions. However, a growing body of evidence suggests that many concepts are instead organized along low-dimensional manifolds encoding continuous geometric relationships. This raises three basic questions: what does it mean for an SAE to capture a manifold, when do existing SAE architectures do so, and how? We develop a theoretical framework that answers these questions and show that SAEs can capture manifolds in two fundamentally different ways: globally, by allocating a compact group of atoms whose linear span contains the entire manifold, or locally, by distributing it across features that each selectively tile a restricted region of the underlying geometry. Empirically, we find that SAEs suboptimally recover continuous structures, mixing the global subspace and local tiling solutions in a fragmented regime we call dilution. This explains why manifold structure is rarely visible at the level of individual concepts and motivates post-hoc unsupervised discovery methods that search for coherent groups of atoms rather than isolated directions. More broadly, our results suggest that future representation learning methods should treat geometric objects, not just individual directions, as the basic units of interpretability.

cs.LG↗
arXiv2026

Symmetry Breaking in Transformers for Efficient and Interpretable Training

The attention mechanism in its standard implementation contains extraneous rotational degrees of freedom that are carried through computation but do not affect model activations or outputs. We introduce a simple symmetry-breaking protocol that inserts a preferred direction into this rotational space through batchwise-sampled, unlearned query and value biases. This modification has two theoretically motivated and empirically validated consequences. First, it can substantially improve the performance of simple, memory-efficient optimizers, narrowing -- and in some cases closing -- the gap to successful but more complex memory-intensive adaptive methods. We demonstrate this by pretraining 124M parameter transformer models with four optimization algorithms (AdamW, SOAP, SGDM, and Energy Conserving Descent(ECD)) and evaluating both validation loss and downstream logical reasoning. Second, it enables an interpretable use of otherwise redundant rotational degrees of freedom, selectively amplifying semantically meaningful token classes within individual attention heads. Overall, our results show that minimal, principled architectural changes can simultaneously improve performance and interpretability.

cs.LG↗
arXiv2025

Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters

Our formulation reveals that the reduction across the sequence axis can be efficiently computed in parallel through a tree reduction. Our algorithm, called Tree Attention, for parallelizing exact attention computation across multiple GPUs enables cross-device decoding to be performed asymptotically faster (up to 8x faster in our experiments) than state-of-the-art approaches such as Ring Attention, while also requiring significantly less communication volume and incurring 2x less peak memory. We demonstrate that Tree Attention speeds up decoding up to 4x on Llama 3.1-8B and can be applied to a variety of hardware and networking setups such as H100 DGX nodes, AMD MI300x nodes, and PCIe connected NVIDIA RTX 4090s. Our code is publicly available here: https://github.com/Zyphra/tree_attention

cs.LG↗
arXiv2024

The Zamba2 Suite: Technical Report

In this technical report, we present the Zamba2 series -- a suite of 1.2B, 2.7B, and 7.4B parameter hybrid Mamba2-transformer models that achieve state of the art performance against the leading open-weights models of their class, while achieving substantial gains in inference latency, throughput, and memory efficiency. The Zamba2 series builds upon our initial work with Zamba1-7B, optimizing its architecture, training and annealing datasets, and training for up to three trillion tokens. We provide open-source weights for all models of the Zamba2 series as well as instruction-tuned variants that are strongly competitive against comparable instruct-tuned models of their class. We additionally open-source the pretraining dataset, which we call Zyda-2, used to train the Zamba2 series of models. The models and datasets used in this work are openly available at https://huggingface.co/Zyphra

cs.LG↗
arXiv2024

A Comment on Deriving the Gibbons-Hawking-York Term From the String Worldsheet

In this note, we show that the noncovariant metric boundary term obtained from the nonlinear sigma model worldsheet derivation of the bulk off-shell sphere partition function is closely related to the Einstein boundary term in the Gamma-Gamma noncovariant action. In fact, when expressed in terms of the trace of the extrinsic curvature tensor, we illustrate that this boundary term has one-half the coefficient of the Gibbons-Hawking-York boundary term required such that the total (bulk plus boundary) off-shell classical action has a well-posed variational principle with Dirichlet boundary conditions.

hep-th↗
arXiv2023

$T\bar{T}$ deformed scattering happens within matrices

We show the $T\bar{T}$ deformation of two-dimensional quantum field theories is equivalent to replacing the spacetime dependence of the fields with dependence on the indices of infinitely large matrices. We show how this correspondence explains the CDD phase dressing of the $S$-matrix and the general formula for the deformation of arbitrary correlation functions. We also describe how the Moyal deformation of self-dual gravity is a $T\bar{T}$ deformation of the theory described by the Chalmers-Siegel action, where the $T\bar{T}$ deformation is defined on the two-dimensional plane of interactions.

hep-th↗
arXiv2022

Non-local Field Theory from Matrix Models

We show that a class of matrix theories can be understood as an extension of quantum field theory which has non-local interactions. This reformulation is based on the Wigner-Weyl transformation, and the interactions take the form of Moyal product on a doubled geometry. We recover local dynamics on the spacetime as a low-energy limit. This framework opens up the possibility for studying novel high-energy phenomena, including the unification of gauge and geometric symmetries in a gauge theory.

hep-th↗
arXiv2022

de Sitter Microstates from $T\bar T+Λ_2$ and the Hawking-Page Transition

We obtain microstates accounting for the Gibbons-Hawking entropy in $dS_3$, along with a subleading logarithmic correction, from the solvable $T\bar T+Λ_2$ deformation of a seed CFT with sparse light spectrum. The microstates arise as the dressed CFT states near dimension $Δ=c/6$, associated with the Hawking-Page transition; they dominate the real spectrum of the deformed theory. We exhibit an analogue of the Hawking-Page transition in de Sitter. Appropriate generalizations of the $T\bar T+Λ_2$ deformation are required to treat model-dependent local bulk physics (subleading at large central charge) and higher dimensions. These results add considerably to the already strong motivation for the continued pursuit of such generalizations along with a more complete characterization of $T\bar T$ type theories, building from existing results in these directions.

hep-th↗
arXiv2022

$T\bar{T}+Λ_2$ Deformed CFT on the Stretched dS$_3$ Horizon

The $T\bar{T}+Λ_2$ deformation of a Holographic CFT on a cylinder (or torus in Euclidean signature) inhabits the stretched cosmological horizon in dS$_3$ when the deformation parameter is tuned to a particular value. I will describe how this insight allows us to compute the entropy of the stretched cosmic horizon including the logarithmic correction.

hep-th↗
arXiv2020

Conformal Boundary Conditions from Cutoff AdS$_3$

We construct a particular flow in the space of 2D Euclidean QFTs on a torus, which we argue is dual to a class of solutions in 3D Euclidean gravity with conformal boundary conditions. This new flow comes from a Legendre transform of the kernel which implements the $T\bar{T}$ deformation, and is motivated by the need for boundary conditions in Euclidean gravity to be elliptic, i.e. that they have well-defined propagators for metric fluctuations. We demonstrate equivalence between our flow equation and variants of the Wheeler de-Witt equation for a torus universe in the so-called Constant Mean Curvature (CMC) slicing. We derive a kernel for the flow, and we compute the corresponding ground state energy in the low-temperature limit. Once deformation parameters are fixed, the existence of the ground state is independent of the initial data, provided the seed theory is a CFT. The high-temperature density of states has Cardy-like behavior, rather than the Hagedorn growth characteristic of $T\bar{T}$-deformed theories.

hep-th↗
arXiv2020

$T\bar{T}$ deformed YM$_{2}$ on general backgrounds from an integral transformation

We consider the $T\bar{T}$ deformation of two dimensional Yang--Mills theory on general curved backgrounds. We compute the deformed partition function through an integral transformation over frame fields weighted by a Gaussian kernel. We show that this partition function satisfies a flow equation which has been derived previously in the literature, which now holds on general backgrounds. We connect ambiguities associated to first derivative terms in the flow equation to the normalization of the functional integral over frame fields. We then compute the entanglement entropy for a general state in the theory. The connection to the string theoretic description of the theory is also investigated.

hep-th↗
arXiv2019

Quantum corrections to finite radius holography and holographic entanglement entropy

We calculate quantum corrections to holographic entanglement entropy in the proposed duality between $T\bar{T}$-deformed holographic 2D CFTs and gravity in AdS$_{3}$ with a finite cutoff. We first establish the dictionary between the two theories by mapping the flow equation of the deformed CFT to the bulk Wheeler-DeWitt equation. The latter reduces to an ordinary differential equation for the sphere partition function, which we solve to find the entanglement entropy for an entangling surface consisting of two antipodal points on a sphere. The entanglement entropy in the inverse central charge expansion yields the expectation value of the bulk length operator plus the entropy of length fluctuations, in accordance with the Ryu--Takayanagi formula and its generalization due to Faulkner, Lewkowycz, and Maldacena. Special attention is paid to the conformal mode problem and its resolution by a choice of contour for the gravitational path integral.

hep-th↗
arXiv2019

A $T \bar{T}$ Deformation for Curved Spacetimes from 3d Gravity

We propose a generalisation of the $T \bar{T}$ deformation to curved spaces by defining, and solving, a suitable flow equation for the partition function. We provide evidence it is well-defined at the quantum level. This proposal identifies, for any CFT, the $T \bar{T}$ deformed partition function and a certain wavefunction of 3d quantum gravity. This connection, true for any $c$, is not a holographic duality --- the 3d theory is a "fake bulk." We however emphasise that this reduces to the known holographic connection in the classical limit. Concretely, this means the deformed partition function solves exactly not just one global equation, defining the $T \bar{T}$ flow, but in fact a local Wheeler-de Witt equation, relating the $T \bar{T}$ operator to the trace of the stress tensor. This also immediately suggests a version of the $T \bar{T}$ deformation with locally varying deformation parameter. We flesh out the connection to 3d gravity, showing that the partition function of the deformed theory is precisely a 3d gravity path integral. In particular, in the classical limit, this path integral reproduces the holographic picture of Dirichlet boundary conditions at a finite radius and mixed boundary conditions at the asymptotic boundary. Further, we reproduce known results in the flat space limit, as well as the large $c$ $S^2$ partition function, and conjecture an answer for the finite $c$ $S^2$ partition function.

hep-th↗
arXiv2019

Sphere partition functions and cut-off AdS

We consider sphere partition functions of TT deformed large N conformal field theories in d=2,3,4,5 and 6 dimensions, computed using the flow equation. These are shown to non-perturbatively match with bulk computations of $AdS_{d+1}$ with a finite radial cut-off. We then demonstrate how the flow equation can be independently derived from a regularization procedure in defining TT operators through a local Callan-Symanzik equation. Finally, we show that the sphere partition functions, modulo bulk-counterterm contributions, can be reproduced from Wheeler-DeWitt wave functions.

hep-th↗
arXiv2018

Finite Cutoff AdS$_{5}$ Holography and the Generalized Gradient Flow

Recently proposed double trace deformations of large $N$ holographic CFTs in four dimensions define a one parameter family of quantum field theories, which are interpreted in the bulk dual as living on successive finite radius hypersurfaces. The transformation of variables that turns the equation defining the deformation of a four dimensional large $N$ CFT by such operators into the expression for the radial ADM Hamiltonian in the bulk is found. This prescription clarifies the role of various functions of background fields that appear in the flow equation defining the deformed holographic CFT, and also their relationship to the holographic anomaly. The effect of these deformations can also be seen as triggering a generalized gradient flow for the fields of the induced gravity theory obtained from integrating out the fundamental fields of the holographic CFT. The potential for this gradient flow is found to resemble the two derivative effective action previously derived using holographic renormalization.

hep-th↗