arXiv subjects

Vishal Misra

Publications and source records attributed to Vishal Misra.

At least 19 recordsLinked to original sources

arXiv2026

Bayesian Wind Tunnels for Model Selection

Prior work has shown that transformers can perform exact Bayesian filtering within a fixed hypothesis class. Can they also perform Bayesian model selection -- identifying the correct hypothesis class from data? We introduce model-selection Bayesian wind tunnels: controlled environments where ground-truth posteriors over hypothesis classes are available in closed form. Using fixed-point-free involutions -- whose defining property f(f(x))=x is purely relational -- a 2.8M-parameter transformer achieves 0.01-bit entropy agreement with the Bayesian optimum (3 seeds), with both integer tokens and opaque symbols whose meanings change every episode. This extends to non-nested comparisons: involutions vs. 3-cycles (where neither class is a subset of the other) achieve class-posterior MAE under 0.001, demonstrating genuine model selection beyond simplicity/subset bias. We then identify a sharp perceptual access condition: when the discriminative statistic requires arithmetic -- modular addition (rotations) or multiplication (f(x)=cx mod p) -- model selection succeeds with integer tokens but fails completely with opaque symbols, and this boundary persists under 112x scaling (2.8M to 316M parameters). A stationarity control confirms the operative factor: opaque tokens with a fixed relabeling succeed (0.009-bit MAE), showing that stable semantics, not integer identity, enable circuit compilation. Header subtask diagnostics localize the failure to the composition of header inversion with arithmetic rather than header parsing itself. Probing frontier LLMs on the same tasks shows qualitative Bayesian behavior but a large calibration gap (~55x), measured through lossy probes and therefore directional rather than exact.

cs.LG↗

arXiv2026

Time-Aware Synthetic Control

The synthetic control (SC) framework is widely used for observational causal inference with time-series panel data. SC has been successful in diverse applications, but existing methods typically treat the ordering of pre-intervention time indices interchangeable. This invariance means they may not fully take advantage of temporal structure when strong trends are present. We propose Time-Aware Synthetic Control (TASC), which employs a state-space model with a constant trend while preserving a low-rank structure of the signal. TASC uses the Kalman filter and Rauch-Tung-Striebel smoother: it first fits a generative time-series model with expectation-maximization and then performs counterfactual inference. We evaluate TASC on both simulated and real-world datasets, including policy evaluation and sports prediction. Our results suggest that TASC offers advantages in settings with strong temporal trends and high levels of observation noise.

cs.LG↗

arXiv2025

The Bayesian Geometry of Transformer Attention

Transformers often appear to perform Bayesian reasoning in context, but verifying this rigorously has been impossible: natural data lack analytic posteriors, and large models conflate reasoning with memorization. We address this by constructing \emph{Bayesian wind tunnels} -- controlled environments where the true posterior is known in closed form and memorization is provably impossible. In these settings, small transformers reproduce Bayesian posteriors with $10^{-3}$-$10^{-4}$ bit accuracy, while capacity-matched MLPs fail by orders of magnitude, establishing a clear architectural separation. Across two tasks -- bijection elimination and Hidden Markov Model (HMM) state tracking -- we find that transformers implement Bayesian inference through a consistent geometric mechanism: residual streams serve as the belief substrate, feed-forward networks perform the posterior update, and attention provides content-addressable routing. Geometric diagnostics reveal orthogonal key bases, progressive query-key alignment, and a low-dimensional value manifold parameterized by posterior entropy. During training this manifold unfurls while attention patterns remain stable, a \emph{frame-precision dissociation} predicted by recent gradient analyses. Taken together, these results demonstrate that hierarchical attention realizes Bayesian inference by geometric design, explaining both the necessity of attention and the failure of flat architectures. Bayesian wind tunnels provide a foundation for mechanistically connecting small, verifiable systems to reasoning phenomena observed in large language models.

cs.LG↗

arXiv2025

Gradient Dynamics of Attention: How Cross-Entropy Sculpts Bayesian Manifolds

Transformers empirically perform precise probabilistic reasoning in carefully constructed ``Bayesian wind tunnels'' and in large-scale language models, yet the mechanisms by which gradient-based learning creates the required internal geometry remain opaque. We provide a complete first-order analysis of how cross-entropy training reshapes attention scores and value vectors in a transformer attention head. Our core result is an \emph{advantage-based routing law} for attention scores, \[ \frac{\partial L}{\partial s_{ij}} = \alpha_{ij}\bigl(b_{ij}-\mathbb{E}_{\alpha_i}[b]\bigr), \qquad b_{ij} := u_i^\top v_j, \] coupled with a \emph{responsibility-weighted update} for values, \[ \Delta v_j = -\eta\sum_i \alpha_{ij} u_i, \] where $u_i$ is the upstream gradient at position $i$ and $\alpha_{ij}$ are attention weights. These equations induce a positive feedback loop in which routing and content specialize together: queries route more strongly to values that are above-average for their error signal, and those values are pulled toward the queries that use them. We show that this coupled specialization behaves like a two-timescale EM procedure: attention weights implement an E-step (soft responsibilities), while values implement an M-step (responsibility-weighted prototype updates), with queries and keys adjusting the hypothesis frame. Through controlled simulations, including a sticky Markov-chain task where we compare a closed-form EM-style update to standard SGD, we demonstrate that the same gradient dynamics that minimize cross-entropy also sculpt the low-dimensional manifolds identified in our companion work as implementing Bayesian inference. This yields a unified picture in which optimization (gradient flow) gives rise to geometry (Bayesian manifolds), which in turn supports function (in-context probabilistic reasoning).

stat.ML↗

arXiv2025

Geometric Scaling of Bayesian Inference in LLMs

Recent work has shown that small transformers trained in controlled "wind-tunnel'' settings can implement exact Bayesian inference, and that their training dynamics produce a geometric substrate -- low-dimensional value manifolds and progressively orthogonal keys -- that encodes posterior structure. We investigate whether this geometric signature persists in production-grade language models. Across Pythia, Phi-2, Llama-3, and Mistral families, we find that last-layer value representations organize along a single dominant axis whose position strongly correlates with predictive entropy, and that domain-restricted prompts collapse this structure into the same low-dimensional manifolds observed in synthetic settings. To probe the role of this geometry, we perform targeted interventions on the entropy-aligned axis of Pythia-410M during in-context learning. Removing or perturbing this axis selectively disrupts the local uncertainty geometry, whereas matched random-axis interventions leave it intact. However, these single-layer manipulations do not produce proportionally specific degradation in Bayesian-like behavior, indicating that the geometry is a privileged readout of uncertainty rather than a singular computational bottleneck. Taken together, our results show that modern language models preserve the geometric substrate that enables Bayesian inference in wind tunnels, and organize their approximate Bayesian updates along this substrate.

cs.LG↗

arXiv2025

ClusterSC: Advancing Synthetic Control with Donor Selection

In causal inference with observational studies, synthetic control (SC) has emerged as a prominent tool. SC has traditionally been applied to aggregate-level datasets, but more recent work has extended its use to individual-level data. As they contain a greater number of observed units, this shift introduces the curse of dimensionality to SC. To address this, we propose Cluster Synthetic Control (ClusterSC), based on the idea that groups of individuals may exist where behavior aligns internally but diverges between groups. ClusterSC incorporates a clustering step to select only the relevant donors for the target. We provide theoretical guarantees on the improvements induced by ClusterSC, supported by empirical demonstrations on synthetic and real-world datasets. The results indicate that ClusterSC consistently outperforms classical SC approaches.

cs.LG↗

arXiv2024

Personalized Predictions from Population Level Experiments: A Study on Alzheimer's Disease

The purpose of this article is to infer patient level outcomes from population level randomized control trials (RCTs). In this pursuit, we utilize the recently proposed synthetic nearest neighbors (SNN) estimator. At its core, SNN leverages information across patients to impute missing data associated with each patient of interest. We focus on two types of missing data: (i) unrecorded outcomes from discontinuing the assigned treatments and (ii) unobserved outcomes associated with unassigned treatments. Data imputation in the former powers and de-biases RCTs, while data imputation in the latter simulates "synthetic RCTs" to predict the outcomes for each patient under every treatment. The SNN estimator is interpretable, transparent, and causally justified under a broad class of missing data scenarios. Relative to several standard methods, we empirically find that SNN performs well for the above two applications using Phase 3 clinical trial data on patients with Alzheimer's Disease. Our findings directly suggest that SNN can tackle a current pain point within the clinical trial workflow on patient dropouts and serve as a new tool towards the development of precision medicine. Building on our insights, we discuss how SNN can further generalize to real-world applications.

stat.AP↗

arXiv2024

Beyond the Black Box: A Statistical Model for LLM Reasoning and Inference

This paper introduces a novel Bayesian learning model to explain the behavior of Large Language Models (LLMs), focusing on their core optimization metric of next token prediction. We develop a theoretical framework based on an ideal generative text model represented by a multinomial transition probability matrix with a prior, and examine how LLMs approximate this matrix. Key contributions include: (i) a continuity theorem relating embeddings to multinomial distributions, (ii) a demonstration that LLM text generation aligns with Bayesian learning principles, (iii) an explanation for the emergence of in-context learning in larger models, (iv) empirical validation using visualizations of next token probabilities from an instrumented Llama model Our findings provide new insights into LLM functioning, offering a statistical foundation for understanding their capabilities and limitations. This framework has implications for LLM design, training, and application, potentially guiding future developments in the field.

cs.LG↗

arXiv2023

Estimating Ground Reaction Forces from Inertial Sensors

Objective: Our aim is to determine if data collected with inertial measurement units (IMUs) during steady-state running could be used to estimate ground reaction forces (GRFs) and to derive biomechanical variables (e.g., contact time, impulse, change in velocity) using lightweight machine-learning approaches. In contrast, state-of-the-art estimation using LSTMs suffers from prohibitive inference times on edge devices, requires expensive training and hyperparameter optimization, and results in black box models. Methods: We proposed a novel lightweight solution, SVD Embedding Regression (SER), using linear regression between SVD embeddings of IMU data and GRF data. We also compared lightweight solutions including SER and k-Nearest-Neighbors (KNN) regression with state-of-the-art LSTMs. Results: We performed extensive experiments to evaluate these techniques under multiple scenarios and combinations of IMU signals and quantified estimation errors for predicting GRFs and biomechanical variables. We did this using training data from different athletes, from the same athlete, or both, and we explored the use of acceleration and angular velocity data from sensors at different locations (sacrum and shanks). Conclusion: Our results illustrated that lightweight solutions such as SER and KNN can be similarly accurate or more accurate than LSTMs. The use of personal data reduced estimation errors of all methods, particularly for most biomechanical variables (as compared to GRFs); moreover, this gain was more pronounced in the lightweight methods. Significance: The study of GRFs is used to characterize the mechanical loading experienced by individuals in movements such as running, which is clinically applicable to identify athletes at risk for stress-related injuries.

cs.LG↗

arXiv2023

Differentially Private Synthetic Control

Synthetic control is a causal inference tool used to estimate the treatment effects of an intervention by creating synthetic counterfactual data. This approach combines measurements from other similar observations (i.e., donor pool ) to predict a counterfactual time series of interest (i.e., target unit) by analyzing the relationship between the target and the donor pool before the intervention. As synthetic control tools are increasingly applied to sensitive or proprietary data, formal privacy protections are often required. In this work, we provide the first algorithms for differentially private synthetic control with explicit error bounds. Our approach builds upon tools from non-private synthetic control and differentially private empirical risk minimization. We provide upper and lower bounds on the sensitivity of the synthetic control query and provide explicit error bounds on the accuracy of our private synthetic control algorithms. We show that our algorithms produce accurate predictions for the target unit, and that the cost of privacy is small. Finally, we empirically evaluate the performance of our algorithm, and show favorable performance in a variety of parameter regimes, as well as providing guidance to practitioners for hyperparameter tuning.

cs.LG↗

arXiv2022

Bandwidth Allocation Games

Internet providers often offer data plans that, for each user's monthly billing cycle, guarantee a fixed amount of data at high rates until a byte threshold is reached, at which point the user's data rate is throttled to a lower rate for the remainder of the cycle. In practice, the thresholds and rates of throttling can appear and may be somewhat arbitrary. In this paper, we evaluate the choice of threshold and rate as an optimization problem (regret minimization) and demonstrate that intuitive formulations of client regret, which preserve desirable fairness properties, lead to optimization problems that have tractably computable solutions. We begin by exploring the effectiveness of using thresholding mechanisms to modulate overall bandwidth consumption. Next, we separately consider the regret of heterogeneous users who are {\em streamers}, wishing to view content over a finite period of fixed rates, and users who are {\em file downloaders}, desiring a fixed amount of bandwidth per month at their highest obtainable rate. We extend our analysis to a game-theoretic setting where users can choose from a variety of plans that vary the cap on the unbounded-rate data, and demonstrate the convergence of the game. Our model provides a fresh perspective on a fair allocation of resources where the demand is higher than capacity, while focusing on the real-world phenomena of bandwidth throttling practiced by ISPs. We show how the solution to the optimization problem results in allocations that exhibit several desirable fairness properties among the users between whom the capacity must be partitioned.

cs.GT↗

arXiv2021

On the Assumptions of Synthetic Control Methods

Synthetic control (SC) methods have been widely applied to estimate the causal effect of large-scale interventions, e.g., the state-wide effect of a change in policy. The idea of synthetic controls is to approximate one unit's counterfactual outcomes using a weighted combination of some other units' observed outcomes. The motivating question of this paper is: how does the SC strategy lead to valid causal inferences? We address this question by re-formulating the causal inference problem targeted by SC with a more fine-grained model, where we change the unit of the analysis from "large units" (e.g., states) to "small units" (e.g., individuals in states). Under this re-formulation, we derive sufficient conditions for the non-parametric causal identification of the causal effect. We highlight two implications of the reformulation: (1) it clarifies where "linearity" comes from, and how it falls naturally out of the more fine-grained and flexible model, and (2) it suggests new ways of using available data with SC methods for valid causal inference, in particular, new ways of selecting observations from which to estimate the counterfactual.

stat.ME↗

arXiv2020

Synthetic Control, Synthetic Interventions, and COVID-19 spread: Exploring the impact of lockdown measures and herd immunity

The synthetic control method is an empirical methodology forcausal inference using observational data. By observing thespread of COVID-19 throughout the world, we analyze the dataon the number of deaths and cases in different regions usingthe power of prediction, counterfactual analysis, and syntheticinterventions of the synthetic control and its extensions. Weobserve that the number of deaths and cases in different re-gions would have been much smaller had the lockdowns beenimposed earlier and had the re-openings been done later, es-pecially among indoor bars and restaurants. We also analyzethe speculated impact of herd immunity on the spread giventhe population of each region and show that lockdown policieshave a very strong impact on the spread regardless of the levelof prior infections. Our most up-to-date code, model, and data can be foundon github: https://github.com/niloofarbayat/COVID19-synthetic-control-analysis

cs.CY↗

arXiv2020

Zero-Rating and Net Neutrality: Who Wins, Who Loses?

An objective of network neutrality is that the design of regulations for the Internet will ensure that it remains a public, open platform where innovations can thrive. While there is broad agreement that preserving the content quality of service falls under the purview of net neutrality, the role of differential pricing, especially the practice of \emph {zero-rating} remains controversial. Even though some countries (India, Canada) have banned zero-rating, others have either taken no stance or explicitly allowed it (South Africa, Kenya, U.S.). In this paper, we model zero-rating options available between Internet service providers (ISPs) and content providers (CPs) and use these models to better understand the conditions under which offering zero-rated services are preferred, and who specifically gains in utility. We develop a formulation in which providers' incomes vary, from low-income startups to high-income incumbents, and where their decisions to zero-rate are a variation of the traditional prisoner's dilemma game. We find that if zero-rating is permitted, low-income CPs often lose utility, whereas high-income CPs often gain utility. We also study the competitiveness of the CP markets via the \emph{Herfindahl Index}. Our findings suggest that in most cases the introduction of zero-rating \emph{reduces} competitiveness.

cs.GT↗

arXiv2019

Down for Failure: Active Power Status Monitoring

Despite society's strong dependence on electricity, power outages remain prevalent. Standard methods for directly measuring power availability are complex, often inaccurate, and are prone to attack. This paper explores an alternative approach to identifying power outages through intelligent monitoring of IP address availability. In finding these outages, we explore the trade-off between the accuracy of detection and false alarms. We begin by experimentally demonstrating that static, residential Internet connections serve as good indicators of power, as they are mostly active unless power fails and rarely have battery backups. We construct metrics that dynamically score the reliability of each residential IP, where a higher score indicates a higher correlation between that IP's availability and its regional power. We monitor specifically selected subsets of residential IPs and evaluate the accuracy with which they can indicate current county power status. Using data gathered during the power outages caused by Hurricane Florence, we demonstrate that we can track power outages at different granularities, state and county, in both sparse and dense regions. By comparing our detection with the reports gathered from power utility companies, we achieve an average detection accuracy of $90\%$, where we also show some of our false alarms and missed outage events could be due to imperfect ground truth data. Therefore, our method can be used as a complementary technique of power outage detection.

cs.NI↗

arXiv2019

mRSC: Multi-dimensional Robust Synthetic Control

When evaluating the impact of a policy on a metric of interest, it may not be possible to conduct a randomized control trial. In settings where only observational data is available, Synthetic Control (SC) methods provide a popular data-driven approach to estimate a "synthetic" control by combining measurements of "similar" units (donors). Recently, Robust SC (RSC) was proposed as a generalization of SC to overcome the challenges of missing data high levels of noise, while removing the reliance on domain knowledge for selecting donors. However, SC, RSC, and their variants, suffer from poor estimation when the pre-intervention period is too short. As the main contribution, we propose a generalization of unidimensional RSC to multi-dimensional RSC, mRSC. Our proposed mechanism incorporates multiple metrics to estimate a synthetic control, thus overcoming the challenge of poor inference from limited pre-intervention data. We show that the mRSC algorithm with $K$ metrics leads to a consistent estimator of the synthetic control for the target unit under any metric. Our finite-sample analysis suggests that the prediction error decays to zero at a rate faster than the RSC algorithm by a factor of $K$ and $\sqrt{K}$ for the training and testing periods (pre- and post-intervention), respectively. Additionally, we provide a diagnostic test that evaluates the utility of including additional metrics. Moreover, we introduce a mechanism to validate the performance of mRSC: time series prediction. That is, we propose a method to predict the future evolution of a time series based on limited data when the notion of time is relative and not absolute, i.e., we have access to a donor pool that has undergone the desired future evolution. Finally, we conduct experimentation to establish the efficacy of mRSC on synthetic data and two real-world case studies (retail and Cricket).

stat.ME↗

arXiv2017

Delay Bounds for Multiclass FIFO

FIFO is perhaps the simplest scheduling discipline. For single-class FIFO, its delay guarantee performance has been extensively studied: The well-known results include a stochastic delay bound for $GI/GI/1$ by Kingman and a deterministic delay bound for $D/D/1$ by Cruz. However, for multiclass FIFO, few such results are available. To fill the gap, we prove delay bounds for multiclass FIFO in this work, considering both deterministic and stochastic cases. Specifically, delay bounds are presented for multiclass D/D/1, GI/GI/1 and G/G/1. In addition, examples are provided for several basic settings to demonstrate the obtained bounds in more explicit forms, which are also compared with simulation results.

cs.PF↗

arXiv2012

On the Evolution of the Internet Economic Ecosystem

The evolution of the Internet has manifested itself in many ways: the traffic characteristics, the interconnection topologies and the business relationships among the autonomous components. It is important to understand why (and how) this evolution came about, and how the interplay of these dynamics may affect future evolution and services. We propose a network aware, macroscopic model that captures the characteristics and interactions of the application and network providers, and show how it leads to a market equilibrium of the ecosystem. By analyzing the driving forces and the dynamics of the market equilibrium, we obtain some fundamental understandings of the cause and effect of the Internet evolution, which explain why some historical and recent evolutions have happened. Furthermore, by projecting the likely future evolutions, our model can help application and network providers to make informed business decisions so as to succeed in this competitive ecosystem.

cs.NI↗