arXiv subjects

Abir Sarkar

Publications and source records attributed to Abir Sarkar.

10 recordsLinked to original sources

arXiv2026

Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization

We provide a theoretical analysis of Adam under non-stationary stochastic objectives, separating two regimes: Euclidean tracking under adaptive strong monotonicity of the Adam-preconditioned mean-gradient operator, and high-probability projected stationarity guarantees under general $L$-smooth objectives. In the tracking regime, we derive finite-time expected and high-probability bounds that decompose sharply into four components: initialization, objective drift, a first-moment tracking error governed by $β_1$, and a preconditioner perturbation governed by $β_2$. We characterize the burn-in time required for the transient terms to decay to the asymptotic tracking bound under constant and step-decay schedules. We also prove a high-probability bound on the average projected stationarity gap for Adam under distribution shift. Across both analyses, our bounds reveal a noise--drift tradeoff: in noise-dominated regimes, first-moment averaging and adaptive preconditioning can yield favorable upper guarantees, whereas in drift-dominated regimes, stale first-moment information and preconditioner perturbations can enlarge Adam's tracking guarantee, potentially allowing vanilla SGD to attain a smaller tracking error. Our explicit $(β_1,β_2,ε)$-dependent bounds identify mechanisms through which adaptive step-sizing can help or hurt under nonstationarity and provide theoretical explanations consistent with Adam's empirical instability and stabilization under distribution shift.

stat.ML↗

arXiv2026

Double Local-to-Unity: Inference under Nearly Nonstationary Volatility

This article develops a moderate-deviation limit theory for autoregressive models with jointly persistent mean and volatility dynamics. The autoregressive coefficient is allowed to drift toward unity slower than the classical 1/n rate, while the volatility persistence parameter also converges to one at an even slower, logarithmic order, so that the conditional variance process is itself nearly nonstationary and its unconditional moments may diverge. This double localization allows the variance process to be nearly nonstationary and to evolve slowly, as observed in financial data and during asset price bubble episodes. Under standard regularity conditions, we establish consistency and distributional limits for the OLS estimator of the autoregressive coefficient that remains valid in the presence of highly persistent stochastic volatility. We show that the effective normalization for least squares inference is governed by an average volatility scale, and we derive martingale limit theorems for the OLS estimator under joint drift and volatility dynamics. In a mildly stationary regime (where the autoregressive root approaches one from below), the OLS estimator is asymptotically normal. In a mildly explosive regime (where the root approaches one from above), an OLS based self normalized statistic converges to a Cauchy limit. Strikingly, in both regimes, the limiting laws of our statistics are invariant to the detailed specification of the volatility process, even though the conditional variance is itself nearly nonstationary. Overall, the results extend moderate-deviation asymptotics to settings with drifting volatility persistence, unify local to unity inference with nearly nonstationary stochastic volatility, and deliver practically usable volatility robust statistics for empirical work in settings approaching instability and exhibiting bubbles.

math.ST↗

arXiv2026

Is There an AI Bubble? Robust Date-Stamping for Periods of Exuberance

The recent surge in valuations among AI related firms has renewed concerns that markets may be entering a new phase of speculative exuberance, especially in the technology and semiconductor sectors at the center of the AI investment wave. This paper develops a practical econometric framework for detecting, date-stamping, and drawing inference on the origination and collapse of bubble episodes when prices evolve under persistent, time-varying volatility. Standard bubble tests are typically derived under homoskedasticity or weak heteroskedasticity and may therefore yield misleading inference in more general settings. We extend right-tailed Dickey-Fuller unit root tests to autoregressive models with highly persistent mean and volatility dynamics, delivering a stochastic-volatility-robust ADF (SV-ADF) test that accommodates persistent variance without imposing strict parametric structure. Building on a moderate-deviation asymptotic theory, the SV-ADF yields nuisance-parameter-free procedures with distinct critical values for origination and collapse, producing more stable alarms and fewer transient false positives around volatility spikes. We establish consistency of the date-stamping estimator and show that it remains asymptotically tractable. Monte Carlo simulations document strong power and substantial gains over homoskedastic (PWY) procedures when volatility dynamics are pronounced. An empirical analysis of AI-exposed equities, including the "Magnificent Seven" and leading semiconductor firms, finds pervasive exuberance with substantial heterogeneity in timing, intensity, and duration. The evidence points to especially strong bubble dynamics for Alphabet and TSMC in the current cycle, while Tesla and Nvidia exhibited pronounced explosive episodes in earlier phases of the AI-driven market cycle.

stat.ME↗

arXiv2026

Empirical Bayes Predictive Density Estimation under Covariate Shift in Large Imbalanced Linear Mixed Models

We study empirical Bayes (EB) predictive density estimation in linear mixed models (LMMs) with large number of units, which induce a high dimensional random effects space. Focusing on Kullback Leibler (KL) risk minimization, we develop a calibration framework to optimally tune predictive densities derived from on a broad class of flexible priors. Our proposed method addresses two key challenges in predictive inference: (a) severe data scarcity leading to highly imbalanced designs, in which replicates are available for only a small subset of units; and (b) distributional shifts in future covariates. To estimate predictive KL risk in LMMs, we use a data-fission approach that leverages exchangeability in the covariate distribution. We establish convergence rates for our proposed risk estimators and show how their efficiency deteriorates as data scarcity increases. Our results imply the decision-theoretic optimality of the proposed EB predictive density estimator. The theoretical development relies on a novel probabilistic analysis of the interaction between data fission, sample reuse, and the predictive heat-equation representation of George et al. (2006), which expresses predictive KL risk through expected log-marginals. Extensive simulation studies demonstrate strong predictive performance and robustness of the proposed approach across diverse regimes with varying degrees of data scarcity and covariate shift.

stat.ME↗

arXiv2025

Asymptotic Breakdown Point Analysis for a General Class of Minimum Divergence Estimators

Robust inference based on the minimization of statistical divergences has proved to be a useful alternative to classical techniques based on maximum likelihood and related methods. Basu et al. (1998) introduced the density power divergence (DPD) family as a measure of discrepancy between two probability density functions and used this family for robust estimation of the parameter for independent and identically distributed data. Ghosh et al. (2017) proposed a more general class of divergence measures, namely the S-divergence family and discussed its usefulness in robust parametric estimation through several asymptotic properties and some numerical illustrations. In this paper, we develop the results concerning the asymptotic breakdown point for the minimum S-divergence estimators (in particular the minimum DPD estimator) under general model setups. The primary result of this paper provides lower bounds to the asymptotic breakdown point of these estimators which are independent of the dimension of the data, in turn corroborating their usefulness in robust inference under high dimensional data.

math.ST↗

arXiv2021

Comparison of numerical methods for computing the repeated Compton scattering of photons in isotropic media

Repeated Compton scattering of photons with thermal electrons is one of the fundamental processes at work in many astrophysical plasma. Solving the exact evolution equations is hard and one common simplification is based on Fokker-Planck (FP) approximations of the Compton collision term. Here we carry out a detailed numerical comparison of several FP approaches with the exact scattering kernel solution for a range of test problems assuming isotropic media and thermal electrons at various temperatures. The Kompaneets equation, being one of the most widely used FP approximations, fails to account for Klein-Nishina corrections and enhanced Doppler boosts and recoil at high energies. These can be accounted for with an alternative FP approach based on the exact first and second moments of the scattering kernel. As demonstrated here, the latter approach works very well in dilute media, but inherently fails to reproduce the correct equilibrium solution in the limit of many scattering. Conditions for the applicability of the FP approximations are clarified, overall showing that the Kompaneets equation provides the most robust approximation to the full problem, even if inaccurate in many cases. We close our numerical analysis by briefly illustrating the solutions for the spectral distortions of the cosmic microwave background (CMB) after photon injection at redshift $z\lesssim 10^5$, when double Compton and Bremsstrahlung emission can be omitted. We demonstrate that the exact treatment using the scattering kernel computed with {\tt CSpack} is often needed. This work should provide an important step towards accurate computations of the CMB spectral distortions from high-energy particle cascades.

astro-ph.CO↗

arXiv2019

Dissecting the Compton scattering kernel I: Isotropic media

Compton scattering between electrons and photons plays a crucial role in astrophysical plasmas. Many important aspects of this process can be captured by using the so-called Compton scattering kernel. For isotropic media, exact analytic expressions (valid at all electron and photon energies) do exist but are hampered by numerical issues and often are presented in complicated ways. In this paper, we summarize, simplify and improve existing analytic expressions for the Compton scattering kernel, with an eye on clarity and physical understanding. We provide a detailed overview of important properties of the kernel covering a wide range of energies and highlighting aspects that have not been appreciated as much previously. We discuss analytic expressions for the moments of the kernel, comparing various approximations and demonstrating their precision. We also illustrate the properties of the scattering kernel for thermal electrons at various temperatures and photon energies, introducing new analytic approximations valid to high temperatures. The obtained improved formulae for the kernel and its moments should prove useful in many astrophysical computations, one of them being the evolution of spectral distortions of the cosmic microwave background in the early Universe. A novel code, CSpack, for efficient computations of the Compton scattering kernel and its properties (in the future also including anisotropies in the initial electron and photon distributions) is being developed in a series of papers and will be available within one month.

astro-ph.CO↗

arXiv2017

The effects of the small-scale behaviour of dark matter power spectrum on CMB spectral distortion

After numerous astronomical and experimental searches, the precise particle nature of dark matter is still unknown. The standard Weakly Interacting Massive Particle(WIMP) dark matter, despite successfully explaining the large-scale features of the universe, has long-standing small-scale issues. The spectral distortion in the Cosmic Microwave Background(CMB) caused by Silk damping in the pre-recombination era allows one to access information on a range of small scales $0.3 \, {\rm Mpc} < k < 10^4 \, \rm Mpc^{-1}$, whose dynamics can be precisely described using linear theory. In this paper, we investigate the possibility of using the Silk damping induced CMB spectral distortion as a probe of the small-scale power. We consider four suggested alternative dark matter candidates---Warm Dark Matter (WDM), Late Forming Dark Matter, Ultra Light Axion dark matter and Charged Decaying Dark Matter; the matter power in all these models deviate significantly from the $Λ$CDM model at small scales. We compute the spectral distortion of CMB for these alternative models and compare our results with the $Λ$CDM model. We show that the main impact of alternative models is to alter the sub-horizon evolution of the Newtonian potential which affects the late-time behaviour of spectral distortion of CMB. The $y$-parameter diminishes by a few percent as compared to the $Λ$CDM model for a range of parameters of these models: LFDM for formation redshift $z_f = 10^5$ (7\%); WDM for mass $m_{\rm wdm} = 1 \, \rm keV$ (2\%); CHDM for decay redshift $z_{\rm decay} = 10^5$ (5\%); ULA for mass $m_a = 10^{-24} \, \rm eV$ (3\%). We also briefly discuss the detectability of this deviation in light of the upcoming CMB experiment PIXIE, which might have the sensitivity to detect this signal from the pre-recombination phase.

astro-ph.CO↗

arXiv2016

The effects of the small-scale DM power on the cosmological neutral hydrogen (\HI) distribution at high redshifts

The particle nature of dark matter remains a mystery. In this paper, we consider two dark matter models---Late Forming Dark Matter (LFDM) and Ultra-Light Axion (ULA) models---where the matter power spectra show novel effects on small scales. The high redshift universe offers a powerful probe of their parameters. In particular, we study two cosmological observables: the neutral hydrogen (HI) redshifted 21-cm signal from the epoch of reionization, and the evolution of the collapsed fraction of HI in the redshift range $2 < z < 5$. We model the theoretical predictions of the models using CDM-like N-body simulations with modified initial conditions, and generate reionization fields using an excursion-set model. The N-body approximation is valid on the length and halo mass scales studied. We show that LFDM and ULA models predict an increase in the HI power spectrum from the epoch of reionization by a factor between 2--10 for a range of scales $0.1 4 \times 10^5$ (for LFDM) and the axion mass $m_a > 2.6 \times 10^{-23} \, \rm eV$ (for ULA). The comparison of the collapsed mass fraction inferred from damped Lyman-$α$ observations to the theoretical predictions of our models lead to the weaker bounds: $z_f > 2 \times 10^5$ and $m_a > 10^{-23} \, \rm eV$. These bounds are consistent with other constraints in the literature using different observables; we briefly discuss how these bounds compare with possible constraints from the observation of luminosity function of galaxies at high redshifts. In the case of ULAs, these constraints are also consistent with a solution to the cusp-core problem of CDM.

astro-ph.CO↗

arXiv2015

How Late can the Dark Matter form in our universe?

We put constraints on the epoch of dark matter formation for a class of non-WIMP (Weakly Interacting Massive Particle) dark matter candidates. These models allow a fraction of Cold Dark Matter (CDM) to be formed between the epoch of Big Bang Nucleosynthesis (BBN) and the matter radiation equality. We show that for such models the matter power spectra might get strong suppression even on scales that could be probed by linear perturbation theory at low redshifts. Unlike the case of Warm Dark Matter (WDM), where the mass of the dark matter particle controls the suppression scale, in Late Forming Dark Matter (LFDM) scenario, it is the redshift of the dark matter formation which determines the form of the matter power spectra. We use the Sloan Digital Sky Survey (SDSS) galaxy clustering data and the linear matter power spectrum reconstructed from the Lyman-$α$ data to find the latest epoch of the dark matter formation in our universe. If all the observed dark matter is late forming, we find lower bounds on the redshift of dark matter formation $z_f > 1.08 \times 10^5 $ at 99.73 $\%$ C.L from the SDSS data and $z_f > 9 \times 10^5$, at the same C.L, from the Lyman-$α$ data. If onlya fraction of the dark matter is late forming then we find tentative evidence of the presence of LFDM from the Lyman-$α$ data. Upcoming data from SDSS-III/BOSS (Baryon Oscillation Spectroscopic Survey) will allow us to explore this issue in more detail.

astro-ph.CO↗