arXiv subjects

Palash Ghosh

Publications and source records attributed to Palash Ghosh.

9 recordsLinked to original sources

arXiv2026

Long-Memory Reservoir Computing for Data-Scarce Dengue Forecasting

Accurate dengue forecasting is crucial for public health planning, but remains challenging because incidence series are often short, noisy, non-stationary, nonlinear, and often affected by long-range temporal dependence. Fractional differencing in Autoregressive Fractionally Integrated Moving Average (ARFIMA) helps balance non-stationarity and persistence, but its linear structure limits its ability to capture nonlinear dynamics. Deep neural networks can model nonlinear patterns, but usually require large training samples and do not explicitly encode statistical long memory. Echo State Networks (ESNs), a widely used reservoir computing framework, are attractive in this setting because they retain nonlinear recurrent dynamics while training only a simple readout, making them suitable for data-scarce scenarios. However, standard ESNs lack long-term memory from a time-series perspective. This study proposes a long-memory reservoir computing framework that integrates dedicated long-memory and short-memory ESN reservoirs with a ridge-regression readout. We introduce two variants: Fractional ESN (fESN), which incorporates fractional-differencing dynamics into the reservoir to encode long-range dependence directly, and Wavelet ESN (wESN), which extracts stable low-frequency components through wavelet smoothing before modeling them with a memory-aware reservoir. We establish theoretical guarantees for closed-loop reservoir dynamics, showing that standard ESNs induce short-memory processes under mild conditions, whereas the proposed long-memory reservoirs generate polynomially decaying dependence consistent with statistical long memory. Across multiple dengue datasets and forecasting horizons, fESN and wESN outperform statistical and deep learning baselines. Combining conformal prediction with fESN and wESN provides distribution-free calibrated uncertainty intervals.

stat.ML↗

arXiv2026

Deep Generative Transformers for Probabilistic Time Series and Spatiotemporal Forecasting

Reliable uncertainty quantification is paramount for forecasting multivariate time series and spatiotemporal data. While Transformer architectures excel at sequence modeling, current probabilistic approaches typically rely on restrictive parametric likelihoods or quantile-based objectives, thereby limiting their ability to capture complex joint distributions in correlated time series. To overcome these limitations, we propose \textit{Enformer} and its spatiotemporal extension, \textit{GEnformer}. These models synthesize the expressive power of Transformers with engression, a stochastic learning paradigm for modeling conditional distributions. By injecting stochastic noise and optimizing a strictly proper scoring objective, our frameworks directly learn conditional predictive distributions without imposing parametric assumptions. This design ensures the generation of coherent multivariate trajectories while maintaining the Transformer's efficacy in modeling long-range dependencies and cross-series interactions. The probabilistic capability of Enformer is achieved with an asymptotic overhead of only a constant factor over a deterministic Transformer with an identical configuration. We extensively evaluate our frameworks on prominent multivariate benchmarks for temporal dynamics and real-world epidemic datasets for spatiotemporal dynamics. Empirical results demonstrate that both frameworks yield calibrated probabilistic forecasts and consistently outperform state-of-the-art baselines.

cs.LG↗

arXiv2025

Optimal Differentially Private Randomized Response Designs to Collect Sensitive Binary Data

Randomized response is a long-standing method for estimating the prevalence of sensitive attributes while protecting respondent privacy. It is increasingly used to generate synthetic binary data from real personal records, enabling data storage and sharing while protecting individual privacy. While surveys emphasize accurate estimation, synthetic data generation prioritizes privacy. Statisticians typically set sample sizes to achieve a target statistical power. However, we show that high power can increase the risk of privacy violations. We consider established randomized response designs with respect to statistical power and differential privacy, which quantifies the leakage of privacy. Our results reveal that common design strategies can yield either insufficient power or excessive privacy loss. We provide optimal parameter choices for randomized response models that jointly satisfy desired power and differential privacy constraints. We motivate and evaluate our approaches using a dataset from a randomized response survey conducted via Amazon Mechanical Turk on tax return misreporting, providing a policy-relevant testbed. Simulation studies evaluate the existence of optimal design parameters, identify designs that minimize the required sample size, and quantify sample size inflation relative to direct questioning. We provide a user-friendly web application (Shiny App) available at https://iitg.ac.in/pgapps/DP_RR/ for designing randomized response studies to facilitate adoption.

stat.ME↗

arXiv2023

Optimal Adaptive SMART Designs with Binary Outcomes

In a sequential multiple-assignment randomized trial (SMART), a sequence of treatments is given to a patient over multiple stages. In each stage, randomization may be done to allocate patients to different treatment groups. Even though SMART designs are getting popular among clinical researchers, the methodologies for adaptive randomization at different stages of a SMART are few and not sophisticated enough to handle the complexity of optimal allocation of treatments at every stage of a trial. Lack of optimal allocation methodologies can raise serious concerns about SMART designs from an ethical point of view. In this work, we develop an optimal adaptive allocation procedure to minimize the expected number of treatment failures for a SMART with a binary primary outcome. Issues related to optimal adaptive allocations are explored theoretically with supporting simulations. The applicability of the proposed methodology is demonstrated using a recently conducted SMART study named M-Bridge for developing universal and resource-efficient dynamic treatment regimes (DTRs) for incoming first-year college students as a bridge to desirable treatments to address alcohol-related risks.

stat.ME↗

arXiv2022

MDAS: A Diagnostic Approach to Assess the Quality of Data Splitting in Machine Learning

In the field of machine learning, model performance is usually assessed by randomly splitting data into training and test sets. Different random splits, however, can yield markedly different performance estimates, so a genuinely good model may be discarded or a poor one selected purely due to an unlucky partition. This motivates a principled way to diagnose the quality of a given data split. We propose a diagnostic framework based on a new discrepancy measure, the Mahalanobis Distribution Alignment Score (MDAS). MDAS is a symmetric dissimilarity measure between two multivariate samples, rather than a strict metric. MDAS captures both mean and covariance differences and is affine invariant. Building on this, we construct a Monte Carlo test that evaluates whether an observed split is statistically compatible with typical random splits, yielding an interpretable p-value for split quality. Using several real data sets, we study the relationship between MDAS and model robustness, including its association with the normalized Akaike information criterion. Finally, we apply MDAS to compare existing state-of-the-art deterministic data-splitting strategies with standard random splitting. The experimental results show that MDAS provides a simple, model-agnostic tool for auditing data splits and improving the reliability of empirical model evaluation.

stat.CO↗

arXiv2022

A Novel Approach To Assess Dynamic Treatment Regimes Embedded In A Smart With An Ordinal Outcome

Sequential multiple assignment randomized trials (SMARTs) are used to construct data-driven optimal intervention strategies for subjects based on their intervention and covariate histories in different branches of health and behavioral sciences where a sequence of interventions is given to a participant. Sequential intervention strategies are often called dynamic treatment regimes (DTR). In the existing literature, the majority of the analysis methodologies for SMART data assume a continuous primary outcome. However, ordinal outcomes are also quite common in clinical practice. In this work, first, we introduce the notion of generalized odds ratio (GOR) to compare two DTRs embedded in a SMART with an ordinal outcome and discuss some combinatorial properties of this measure. Next, we propose a likelihood-based approach to estimate GOR from SMART data, and derive the asymptotic properties of its estimate. We discuss alternative ways to estimate GOR using concordant-discordant pairs and two-sample U-statistic. We derive the required sample size formula for designing SMARTs with ordinal outcomes based on GOR. A simulation study shows the performance of the estimated GOR in terms of the estimated power corresponding to the derived sample size. The methodology is applied to analyze data from the SMART+ study, conducted in the UK, to improve carbohydrate periodization behavior in athletes using a menu planner mobile application, Hexis Performance. A freely available Shiny web app using R is provided to make the proposed methodology accessible to other researchers and practitioners.

stat.ME↗

arXiv2021

A Penalized Shared-parameter Algorithm for Estimating Optimal Dynamic Treatment Regimens

A dynamic treatment regimen (DTR) is a set of decision rules to personalize treatments for an individual using their medical history. The Q-learning-based Q-shared algorithm has been used to develop DTRs that involve decision rules shared across multiple stages of intervention. We show that the existing Q-shared algorithm can suffer from non-convergence due to the use of linear models in the Q-learning setup, and identify the condition under which Q-shared fails. We develop a penalized Q-shared algorithm that not only converges in settings that violate the condition, but can outperform the original Q-shared algorithm even when the condition is satisfied. We give evidence for the proposed method in a real-world application and several synthetic simulations.

stat.ML↗

arXiv2021

Knee Osteoarthritis Severity Prediction using an Attentive Multi-Scale Deep Convolutional Neural Network

Knee Osteoarthritis (OA) is a destructive joint disease identified by joint stiffness, pain, and functional disability concerning millions of lives across the globe. It is generally assessed by evaluating physical symptoms, medical history, and other joint screening tests like radiographs, Magnetic Resonance Imaging (MRI), and Computed Tomography (CT) scans. Unfortunately, the conventional methods are very subjective, which forms a barrier in detecting the disease progression at an early stage. This paper presents a deep learning-based framework, namely OsteoHRNet, that automatically assesses the Knee OA severity in terms of Kellgren and Lawrence (KL) grade classification from X-rays. As a primary novelty, the proposed approach is built upon one of the most recent deep models, called the High-Resolution Network (HRNet), to capture the multi-scale features of knee X-rays. In addition, we have also incorporated an attention mechanism to filter out the counterproductive features and boost the performance further. Our proposed model has achieved the best multiclass accuracy of 71.74% and MAE of 0.311 on the baseline cohort of the OAI dataset, which is a remarkable gain over the existing best-published works. We have also employed the Gradient-based Class Activation Maps (Grad-CAMs) visualization to justify the proposed network learning.

eess.IV↗

arXiv2018

Non-Inferiority and Equivalence Tests in A Sequential Multiple-Assignment Randomized Trial (SMART)

Adaptive interventions (AIs) are increasingly becoming popular in medical and behavioral sciences. An AI is a sequence of individualized intervention options that specify for whom and under what conditions different intervention options should be offered, in order to address the changing needs of individuals as they progress over time. The sequential, multiple assignment, randomized trial (SMART) is a novel trial design that was developed to aid in empirically constructing effective AIs. The sequential randomizations in a SMART often yield multiple AIs that are embedded in the trial by design. Many SMARTs are motivated by scientific questions pertaining to the comparison of such embedded AIs. Existing data analytic methods and sample size planning resources for SMARTs are suitable for superiority testing, namely for testing whether one embedded AI yields better primary outcomes on average than another. This represents a major scientific gap since AIs are often motivated by the need to deliver support/care in a less costly or less burdensome manner, while still yielding benefits that are equivalent or non-inferior to those produced by a more costly/burdensome standard of care. Here, we develop data analytic methods and sample size formulas for SMART studies aiming to test the non-inferiority or equivalence of one AI over another. Sample size and power considerations are discussed with supporting simulations, and online sample size planning resources are provided. For illustration, we use an example from a SMART study aiming to develop an AI for promoting weight loss among overweight/obese adults.

stat.AP↗