SearcharxivSearch

arXiv subjects

Sandeep Juneja

Publications and source records attributed to Sandeep Juneja.

At least 19 recordsLinked to original sources

Steering Diffusion Priors with Sparse Observations for High-Resolution Temperature Downscaling

Local heatwave hazard depends on fine-scale air temperature, but ground stations are sparse and reanalysis products such as ERA5 cannot resolve the terrain and land-surface contrasts that shape real heat exposure. We present a conditional diffusion emulator for high-resolution 2-m temperature downscaling, conditioned on static geography, a training climatology, exact-time ERA5 temperature, and solar and temporal features, guided at inference by score-based data assimilation (SDA): a differentiable Gaussian observation likelihood steers the diffusion score toward sparse revealed temperature observations without any retraining. On a controlled 32-case synthetic-grid protocol over AORC, guidance improves hidden-cell reconstruction over both ERA5 and a strong observation-proximal nearest-neighbor baseline once observation density reaches 1\% (RMSE 0.318 vs.\ 0.431~K, winning all 32 cases), while sparser regimes still favor direct interpolation. We further map the full guidance-strength landscape across three observation densities, showing that the optimal strength shifts systematically with density and that over-guiding causes sharp, predictable degradation -- giving a concrete operating recipe rather than a single untuned setting. The resulting fields are intended as a temperature layer for downstream heatwave-hazard products such as threshold exceedance and cumulative heat-burden. The present evidence is a controlled synthetic-grid validation; station-network and held-out-year evaluations are the next steps toward deployment.

physics.ao-ph

GraphCast Skill and Systematic Biases in Indian Summer Monsoon Forecasts: Evaluation Against ERA5 and IMERG

The Indian Summer Monsoon (ISM) is one of the most dynamically complex components of the global climate system, yet its accurate prediction remains challenging for both physics-based and emerging AI weather models. We present a climatological evaluation of Google's GraphCast against ERA5 reanalysis and the Integrated Multi-satellitE Retrievals for GPM (IMERG) precipitation dataset during the boreal summer (June-September, JJAS) for 2021-2024. Deterministic GraphCast forecasts initialized at 00 UTC are composited to evaluate +24 h, +48 h, and +72 h lead times over the full ISM domain. The analysis examines the climatological mean state (rainfall, surface temperature, and low-level winds), rainfall intensity distribution, thermodynamic structure (tropospheric temperature gradient, DTT; apparent heat source and moisture sink, Q1 and Q2), monsoon dynamics (vertical wind shear), and rainfall variability across intraseasonal, synoptic, and spectral timescales. GraphCast reproduces the broad spatial pattern and seasonal evolution of monsoon rainfall with good fidelity at short lead times but exhibits a domain-averaged wet bias over the core monsoon region. It also substantially suppresses rainfall variability across nearly all timescales (regional power-spectrum variance ratio of 0.14 relative to IMERG) and produces a compressed rainfall intensity distribution, overestimating moderately heavy rainfall (95th percentile) while systematically underrepresenting the most extreme events. These biases are accompanied by a deficient lower-tropospheric Q1 profile and weaker northward-propagating intraseasonal variability. Together, the results reveal a consistent deterministic-smoothing signature in GraphCast's precipitation forecasts and provide benchmark diagnostics for evaluating next-generation AI weather models for tropical medium-range forecasting (+24 h to +72 h).

physics.ao-ph

AdaWeather: Adaptively Mixing Probabilistic Weather Forecasts with Logarithmic Regret

Recent advances in machine learning have produced probabilistic weather forecasting models comparable to state-of-the-art numerical weather predictors. But no model consistently dominates spatio-temporally, and relative performance is highly context-dependent. This motivates adaptive methods for combining multiple forecasts to obtain improvements and robustness. While combined forecasts have been proposed in the literature, these are achieved either through supervised learning or through prediction with expert advice methods. We introduce AdaWeather, an adaptive framework that combines many probabilistic forecasts using both machine learning as well as mixture of experts to arrive at a unified improved probabilistic forecast. While traditional expert methods develop the regret bounds with respect to the best single expert in hindsight, we extend the algorithm and analysis to show our method has logarithmic regret compared to the best static mixture of experts in hindsight. Empirically, we focus on forecasting temperature, and observe improvements over existing methods.

cs.LG

AirCast-SR: A Foundation Model for Kilometer-Scale Atmospheric Super-Resolution via Latent Consistency Diffusion

Operational weather prediction at kilometer scales remains computationally prohibitive for traditional numerical weather prediction (NWP) models, limiting forecast access for applications in energy, agriculture, and disaster management that require fine-grained spatiotemporal detail. Here we introduce AirCast-SR, a foundation model for atmospheric super-resolution that downscales global AI weather forecasts from 0.25 degree (~28 km) to 1 km horizontal resolution at hourly temporal resolution, producing 67-hour forecasts of eight coupled surface variables simultaneously. EarthMind-SR employs a three-dimensional U-Net conditioned within a Latent Consistency Model (LCM) diffusion framework, trained on patch-based samples over the contiguous United States (CONUS) using GraphCast forecasts as input and NOAA's Analysis of Record for Calibration (AORC) as the target. The model achieves near-zero bias across all variables and lead times, and its radial power spectral density analysis demonstrates preservation of fine-scale atmospheric structure at wavelengths of 10 km to 100 km where coarser models lose spectral power. We validate EarthMind-SR across three CONUS case studies spanning winter, summer, and spring seasons, and demonstrate zero-shot global transferability over India and Germany using independent surface station observations without any retraining or fine-tuning. As an open-weights foundation model, EarthMind-SR establishes a new paradigm for kilometer-scale AI weather prediction and provides a platform for regional fine-tuning, distillation, and downstream applications in climate services and hazard forecasting.

cs.LG

Generating DDPM-based Samples from Tilted Distributions

Given $n$ independent samples from a $d$-dimensional probability distribution, our aim is to generate diffusion-based samples from a distribution obtained by tilting the original, where the degree of tilt is parametrized by $\theta \in \mathbb{R}^d$. We define a plug-in estimator and show that it is minimax-optimal. We develop Wasserstein bounds between the distribution of the plug-in estimator and the true distribution as a function of $n$ and $\theta$, illustrating regimes where the output and the desired true distribution are close. Further, under some assumptions, we prove the TV-accuracy of running Diffusion on these tilted samples. Our theoretical results are supported by extensive simulations. Applications of our work include finance, weather and climate modelling, and many other domains, where the aim may be to generate samples from a tilted distribution that satisfies practically motivated moment constraints.

cs.LG

Fundamental limits for weighted empirical approximations of tilted distributions

Consider the task of generating samples from a tilted distribution of a random vector whose underlying distribution is unknown, but samples from it are available. This finds applications in fields such as finance and climate science, and in rare event simulation. In this article, we discuss the asymptotic efficiency of a self-normalized importance sampler of the tilted distribution. We provide a sharp characterization of its accuracy, given the number of samples and the degree of tilt. Our findings reveal a surprising dichotomy: while the number of samples needed to accurately tilt a bounded random vector increases polynomially in the tilt amount, it increases at a super polynomial rate for unbounded distributions.

math.ST

Optimal Algorithms for Bandit Learning in Matching Markets

We study the problem of pure exploration in matching markets under uncertain preferences, where the goal is to identify a stable matching with confidence parameter $\delta$ and minimal sample complexity. Agents learn preferences via stochastic rewards, with expected values indicating preferences. This finds use in labor market platforms like Upwork, where firms and freelancers must be matched quickly despite noisy observations and no prior knowledge, in a stable manner that prevents dissatisfaction. We consider markets with unique stable matching and establish information-theoretic lower bounds on sample complexity for (1) one-sided learning, where one side of the market knows its true preferences, and (2) two-sided learning, where both sides are uncertain. We propose a computationally efficient algorithm and prove that it asymptotically ($\delta\to 0$) matches the lower bound to a constant for one-sided learning. Using the insights from the lower bound, we extend our algorithm to the two-sided learning setting and provide experimental results showing that it closely matches the lower bound on sample complexity. Finally, using a system of ODEs, we characterize the idealized fluid path that our algorithm chases.

cs.GT

Asymptotic optimality theory of confidence intervals of the mean

We address the classical problem of constructing confidence intervals (CIs) for the mean of a distribution, given \(N\) i.i.d. samples, such that the CI contains the true mean with probability at least \(1 - \delta\), where \(\delta \in (0,1)\). We characterize three distinct learning regimes based on the minimum achievable limiting width of any CI as the sample size \(N_{\delta} \to \infty\) and \(\delta \to 0\). In the first regime, where \(N_{\delta}\) grows slower than \(\log(1/\delta)\), the limiting width of any CI equals the width of the distribution's support, precluding meaningful inference. In the second regime, where \(N_{\delta}\) scales as \(\log(1/\delta)\), we precisely characterize the minimum limiting width, which depends on the scaling constant. In the third regime, where \(N_{\delta}\) grows faster than \(\log(1/\delta)\), complete learning is achievable, and the limiting width of the CI collapses to zero, converging to the true mean. We demonstrate that CIs derived from concentration inequalities based on Kullback--Leibler (KL) divergences achieve asymptotically optimal performance, attaining the minimum limiting width in both sufficient and complete learning regimes for distributions in two families: single-parameter exponential and bounded support. Additionally, these results extend to one-sided CIs, with the width notion adjusted appropriately. Finally, we generalize our findings to settings with random per-sample costs, motivated by practical applications such as stochastic simulators and cloud service selection. Instead of a fixed sample size, we consider a cost budget \(C_{\delta}\), identifying analogous learning regimes and characterizing the optimal CI construction policy.

math.ST

Optimal Top-Two Method for Best Arm Identification and Fluid Analysis

Top-$2$ methods have become popular in solving the best arm identification (BAI) problem. The best arm, or the arm with the largest mean amongst finitely many, is identified through an algorithm that at any sequential step independently pulls the empirical best arm, with a fixed probability $\beta$, and pulls the best challenger arm otherwise. The probability of incorrect selection is guaranteed to lie below a specified $\delta >0$. Information theoretic lower bounds on sample complexity are well known for BAI problem and are matched asymptotically as $\delta \rightarrow 0$ by computationally demanding plug-in methods. The above top 2 algorithm for any $\beta \in (0,1)$ has sample complexity within a constant of the lower bound. However, determining the optimal $\beta$ that matches the lower bound has proven difficult. In this paper, we address this and propose an optimal top-2 type algorithm. We consider a function of allocations anchored at a threshold. If it exceeds the threshold then the algorithm samples the empirical best arm. Otherwise, it samples the challenger arm. We show that the proposed algorithm is optimal as $\delta \rightarrow 0$. Our analysis relies on identifying a limiting fluid dynamics of allocations that satisfy a series of ordinary differential equations pasted together and that describe the asymptotic path followed by our algorithm. We rely on the implicit function theorem to show existence and uniqueness of these fluid ode's and to show that the proposed algorithm remains close to the ode solution.

cs.LG

Comparing skill of historical rainfall data based monsoon rainfall prediction in India with NWP forecasts

The Indian summer monsoon is a highly complex and critical weather system that directly affects the livelihoods of over a billion people across the Indian subcontinent. Accurate short-term forecasting remains a major scientific challenge due to the monsoon's intrinsic nonlinearity and its sensitivity to multi-scale drivers, including local land-atmosphere interactions and large-scale ocean-atmosphere phenomena. In this study, we address the problem of forecasting daily rainfall across India during the summer months, focusing on both one-day and three-day lead times. We use Autoformers - deep learning transformer-based architectures designed for time series forecasting. These are trained on historical gridded precipitation data from the Indian Meteorological Department (1901--2023) at spatial resolutions of $0.25^\circ \times 0.25^\circ$, as well as $1^\circ \times 1^\circ$. The models also incorporate auxiliary meteorological variables from ECMWFs reanalysis datasets, namely, cloud cover, humidity, temperature, soil moisture, vorticity, and wind speed. Forecasts at $0.25^\circ \times 0.25^\circ$ are benchmarked against ECMWFs High-Resolution Ensemble System (HRES), widely regarded as the most accurate numerical weather predictor, and at $1^\circ \times 1^\circ $ with those from National Centre for Environmental Prediction (NCEP). We conduct both nationwide evaluations and localized analyses for major Indian cities. Our results indicate that transformer-based deep learning models consistently outperform both HRES and NCEP, as well as other climatological baselines. Specifically, compared to our model, forecasts from HRES and NCEP model have about 22\% and 43\% higher error, respectively, for a single day prediction, and over 27\% and 66\% higher error respectively, for a three day prediction.

cs.LG

Game of arrivals at a two queue network with heterogeneous customer routes

We consider a queuing network that opens at a specified time, where customers are non-atomic and belong to different classes. Each class has its own route, and as is typical in the literature, the costs are a linear function of waiting and service completion time. We restrict ourselves to a two class, two queue network: this simplification is well motivated as the diversity in solution structure as a function of problem parameters is substantial even in this simple setting (e.g., a specific routing structure involves eight different regimes), suggesting a combinatorial blow up as the number of queues, routes and customer classes increase. We identify the unique Nash equilibrium customer arrival profile when the customer linear cost preferences are different. This profile is a function of problem parameters including the size of each class, service rates at each queue, and customer cost preferences. When customer cost preferences match, under certain parametric settings, the equilibrium arrival profiles may not be unique and may lie in a convex set. We further make a surprising observation that in some parametric settings, customers in one class may arrive in disjoint intervals. Further, the two classes may arrive in contiguous intervals or in overlapping intervals, and at varying rates within an interval, depending upon the problem parameters.

cs.PF

Optimal Best-Arm Identification in Bandits with Access to Offline Data

Learning paradigms based purely on offline data as well as those based solely on sequential online learning have been well-studied in the literature. In this paper, we consider combining offline data with online learning, an area less studied but of obvious practical importance. We consider the stochastic $K$-armed bandit problem, where our goal is to identify the arm with the highest mean in the presence of relevant offline data, with confidence $1-\delta$. We conduct a lower bound analysis on policies that provide such $1-\delta$ probabilistic correctness guarantees. We develop algorithms that match the lower bound on sample complexity when $\delta$ is small. Our algorithms are computationally efficient with an average per-sample acquisition cost of $\tilde{O}(K)$, and rely on a careful characterization of the optimality conditions of the lower bound problem.

cs.LG

Agent Based Simulators for Epidemic Modelling: Simulating Larger Models Using Smaller Ones

Agent-based simulators (ABS) are a popular epidemiological modelling tool to study the impact of various non-pharmaceutical interventions in managing an epidemic in a city (or a region). They provide the flexibility to accurately model a heterogeneous population with time and location varying, person-specific interactions as well as detailed governmental mobility restrictions. Typically, for accuracy, each person is modelled separately. This however may make computational time prohibitive when the city population and the simulated time is large. In this paper, we dig deeper into the underlying probabilistic structure of a generic, locally detailed ABS for epidemiology to arrive at modifications that allow smaller models (models with less number of agents) to give accurate statistics for larger ones, thus substantially speeding up the simulation. We observe that simply considering a smaller aggregate model and scaling up the output leads to inaccuracies. We exploit the observation that in the initial disease spread phase, the starting infections create a family tree of infected individuals more-or-less independent of the other trees and are modelled well as a multi-type super-critical branching process. Further, although this branching process grows exponentially, the relative proportions amongst the population types stabilise quickly. Once enough people have been infected, the future evolution of the epidemic is closely approximated by its mean field limit with a random starting state. We build upon these insights to develop a shifted, scaled and restart-based algorithm that accurately evaluates the ABS's performance using a much smaller model while carefully reducing the bias that may otherwise arise. We apply our algorithm for Covid-19 epidemic and theoretically support the proposed algorithm through an asymptotic analysis where the population size increases to infinity.

physics.soc-ph

Potential 3rd COVID Wave in Mumbai: Scenario Analysis

The second wave of Covid-19 that started in mid-February 2021 in Mumbai is now subsiding. Increasingly the focus amongst the policy makers and general public is on the potential third wave. Due to uncertainties regarding emergence of new variants and reinfections, instead of projecting our best guess scenario, in this report we conduct an extensive scenario analysis for Mumbai and track peak fatalities in the coming months in each of these scenarios. Our key conclusions are - As per our model, about 80% of Mumbai population has been exposed to Covid-19 by June 1, 2021. Under the assumption that all who are exposed have immunity against further infection, it is unlikely that Mumbai would see a large third wave. It is the reinfections that may lead to a large wave. Reinfections could occur because of declining antibodies amongst the infected as well as by variants that can break through the immunity provided by prior infections. Even under a reasonably pessimistic scenario we observe the resulting peak to be no larger than that under the second wave. We further observe that under the scenario where the reinfections are mild so that they affect the fatality figures negligibly, where the new variants (beyond the existing delta variant) have a mild impact, as the city opens up, we observe a small wave in the coming months. However, if by then the vaccine coverage is extensive, this wave will be barely noticeable. We also plot $R_t$, the infection growth rate at time $t$, and highlight some interesting observations.

q-bio.PE

Modelling the Second Covid-19 Wave in Mumbai

India has been hit by a huge second wave of Covid-19 that started in mid-February 2021. Mumbai was amongst the first cities to see the increase. In this report, we use our agent based simulator to computationally study the second wave in Mumbai. We build upon our earlier analysis, where projections were made from November 2020 onwards. We use our simulator to conduct an extensive scenario analysis - we play out many plausible scenarios through varying economic activity, reinfection levels, population compliance, infectiveness, prevalence and lethality of the possible variant strains, and infection spread via local trains to arrive at those that may better explain the second wave fatality numbers. We observe and highlight that timings of peak and valley of the fatalities in the second wave are robust to many plausible scenarios, suggesting that they are likely to be accurate projections for Mumbai. During the second wave, the observed fatalities were low in February and mid-March and saw a phase change or a steep increase in the growth rate after around late March. We conduct extensive experiments to replicate this observed sharp convexity. This is not an easy phenomena to replicate, and we find that explanations such as increased laxity in the population, increased reinfections, increased intensity of infections in Mumbai transportation, increased lethality in the virus, or a combination amongst them, generally do a poor job of matching this pattern. We find that the most likely explanation is presence of small amount of extremely infective variant on February 1 that grows rapidly thereafter and becomes a dominant strain by Mid-March. From a prescriptive view, this points to an urgent need for extensive and continuous genome sequencing to establish existence and prevalence of different virus strains in Mumbai and in India, as they evolve over time.

q-bio.PE

Regret Minimization in Heavy-Tailed Bandits

We revisit the classic regret-minimization problem in the stochastic multi-armed bandit setting when the arm-distributions are allowed to be heavy-tailed. Regret minimization has been well studied in simpler settings of either bounded support reward distributions or distributions that belong to a single parameter exponential family. We work under the much weaker assumption that the moments of order $(1+\epsilon)$ are uniformly bounded by a known constant B, for some given $\epsilon > 0$. We propose an optimal algorithm that matches the lower bound exactly in the first-order term. We also give a finite-time bound on its regret. We show that our index concentrates faster than the well known truncated or trimmed empirical mean estimators for the mean of heavy-tailed distributions. Computing our index can be computationally demanding. To address this, we develop a batch-based algorithm that is optimal up to a multiplicative constant depending on the batch size. We hence provide a controlled trade-off between statistical optimality and computational cost.

cs.LG

COVID-19 Epidemic in Mumbai: Projections, full economic opening, and containment zones versus contact tracing and testing: An Update

Mumbai, amongst the most densely populated cities in the world, has witnessed the fourth largest number of cases and the largest number of deaths among all the cities in India (as of 28th October 2020). Along with the rest of India, lockdowns (of varying degrees) have been in effect in Mumbai since March 25, 2020. Given the large economic toll on the country from the lockdown and the related restrictions on mobility of people and goods, swift opening of the economy especially in a financial hub such as Mumbai becomes critical. In this report, we use the IISc-TIFR agent based simulator to develop long term projections for Mumbai under realistic scenarios related to Mumbai's opening of the workplaces, or equivalently, the economy, and the associated public transportation through local trains and buses. These projections were developed taking into account a possible second wave if the economy and the local trains are fully opened either on November 1, 2020 or on January 1, 2021. The impact on infection spread in Mumbai if the schools and colleges open on January first week 2021 is also considered. We also try to account for the increased intermingling amongst the population during the Ganeshotsav festival as well as around the Navratri/Dussehra and Diwali festival. Our conclusion, based on our simulations, is that the impact of fully opening up the economy on November 1 is manageable provided reasonable medical infrastructure is in place. Further, schools and colleges opening in January do not lead to excessive increase in infections. The report also explores the relative effectiveness of contact tracing vs containment zones, and also includes very rudimentary results of the effect of vaccinating the elderly population in February 2021.

physics.soc-ph

Optimal Best-Arm Identification Methods for Tail-Risk Measures

Conditional value-at-risk (CVaR) and value-at-risk (VaR) are popular tail-risk measures in finance and insurance industries as well as in highly reliable, safety-critical uncertain environments where often the underlying probability distributions are heavy-tailed. We use the multi-armed bandit best-arm identification framework and consider the problem of identifying the arm from amongst finitely many that has the smallest CVaR, VaR, or weighted sum of CVaR and mean. The latter captures the risk-return trade-off common in finance. Our main contribution is an optimal $\delta$-correct algorithm that acts on general arms, including heavy-tailed distributions, and matches the lower bound on the expected number of samples needed, asymptotically (as $\delta$ approaches $0$). The algorithm requires solving a non-convex optimization problem in the space of probability measures, that requires delicate analysis. En-route, we develop new non-asymptotic empirical likelihood-based concentration inequalities for tail-risk measures which are tighter than those for popular truncation-based empirical estimators.

cs.LG