arXiv subjects

Shantanu Jain

Publications and source records attributed to Shantanu Jain.

18 recordsLinked to original sources

arXiv2026

The DIRECD coronal mass ejection direction catalog: three-dimensional propagation inferred from coronal dimmings

Coronal mass ejections (CMEs) are among the primary drivers of space weather disturbances at Earth, yet their early propagation in the low corona remains poorly constrained owing to occultation and projection effects inherent to coronagraph observations. Coronal dimmings offer an alternative diagnostic to infer CME propagation direction directly from the low corona. We present the DIRECD CME Direction Catalog, a unified dataset of three-dimensional CME propagation directions derived via the DIRECD (Dimming Inferred Estimation of CME Direction) method, which reconstructs CME cone geometry from SDO/AIA 211 \AA\ coronal dimming observations. The catalog comprises 64 events spanning 2010-2026, combining 31 Solar Cycle 25 (2021-2026) events with 33 events from prior studies. For each event we report the 3D propagation direction, 2D inclination angles, angular width, and cone height. Statistical analysis reveals a systematic asymmetry in low-coronal CME propagation: meridional inclinations exhibit a systematic tendency toward latitude-dependent deflection, with poleward deflections more pronounced at higher source latitudes, while equatorial inclinations show no significant dependence on source longitude. This is consistent with low-coronal CME trajectories being governed primarily by local active-region magnetic topology, with the large-scale coronal field and heliospheric current sheet assuming a progressively dominant role at greater heliocentric distances. Incorporating DIRECD-derived propagation directions into CME arrival forecasts improves predictions of geomagnetic storm intensity, raising the correlation with observed storm strength from r = 0.58 to r = 0.70 when combined with CME speed. The DIRECD software, catalog, and all associated data products are publicly available to support space weather research.

astro-ph.SR↗

arXiv2025

Validating DIRECD: Statistical Evaluation of Coronal Mass Ejections Direction Estimates from Coronal Dimmings

Coronal mass ejections (CMEs) are among the most energetic phenomena in our solar system, with significant implications for space weather. Understanding their early dynamics remains challenging due to observational limitations in the low corona. We present a statistical evaluation of the DIRECD (Dimming InfeRred Estimation of CME Direction) method, which provides a novel approach to determining initial CME propagation directions using coronal dimmings. We analyze 33 coronal dimming events well observed by SDO/AIA and validate our DIRECD results with 3D reconstructions from the Graduated Cylindrical Shell (GCS) model. We find generally good agreement between the DIRECD-derived inclinations and the GCS model. In the meridional plane (north--south direction), the mean difference in inclinations is $0.3^\circ \pm 7.8^\circ$. In the equatorial plane (east--west direction), the mean difference is $-2.9^\circ \pm 18.9^\circ$. In 3D, the inclinations show a mean difference of $1.2^\circ \pm 10.4^\circ$. We further visually compare our method by projecting the DIRECD cones onto LASCO/C2 observations, and verify the model's ability to capture both the primary CME structure and associated secondary dimming regions. This work establishes DIRECD as a powerful, observationally grounded technique for determining the initial CME direction, offering new insights that complement existing reconstruction methods. The technique's unique capability to determine early CME direction in the low corona using coronal dimmings observed in EUV images makes it particularly valuable for improving space weather forecasting models.

astro-ph.SR↗

arXiv2025

Coronal dimmings from active region 13664 during the May 2024 solar energetic events

Coronal dimmings are regions of transiently reduced brightness in extreme ultraviolet (EUV) and soft X-ray (SXR) emissions associated with coronal mass ejections (CMEs), providing key insights into CME initiation and early evolution. During May 2024, AR 13664 was among the most flare-productive regions in recent decades, generating 55 M-class and 12 X-class flares along with multiple Earth-directed CMEs. The rapid succession of these CMEs triggered the most intense geomagnetic storm in two decades. We study coronal dimmings from a single active region (AR 13664) and compare them with statistical dimming properties. We investigate how coronal dimming parameters - such as area, brightness, and magnetic flux - relate to key flare and CME properties. We systematically identified all flares above M1.0, all coronal dimmings and all CMEs (from the CDAW SOHO/LASCO catalogue) produced by AR 13664 during 2024 May 1 - 15, and studied the associations between the different phenomena and their characteristic parameters. We detect coronal dimmings in 22 events, with 16 occurring on-disc and six off-limb. Approximately 83% of X-class flares and 23% of M-class flares are associated with CMEs, with 13 out of 16 on-disc dimmings linked to CME activity. Our results support the strong interplay between coronal dimmings and flares, as we find increased correlations between flare and dimming parameters in this single-AR study compared to the general dimming population. Furthermore, we confirm that coronagraphic observations, unable to observe the lower corona, underestimate correlations between CME velocities and dimming parameters, as they fail to capture the early CME acceleration phase. This highlights the critical role of dimming observations in providing a more comprehensive understanding of CME dynamics.

astro-ph.SR↗

arXiv2024

Three-part structure of solar coronal mass ejection observed in low coronal signatures of Solar Orbiter

This study examines the relationship between early solar coronal mass ejection (CME) propagation, the associated filament eruption, and coronal dimming in the rare event observed on March 28, 2022, which featured a three-part CME in the low corona of active region AR 12975, including a bright core/filament, dark cavity, and bright front edge. We employ 3D filament and CME shock reconstructions using data from SolO, STEREO-A, and SDO to track the filament's path, height, and kinematics. Our analysis across three viewpoints shows the outer front in SolO/EUI 304 \r{A} aligns with shock structures in STEREO-A/EUVI 195 \r{A}, showing a full 3D EUV wave dome, later matching the outer CME front in STEREO-A COR2. We introduce the method ATLAS-3D (Advanced Technique for single Line-of-sight Acquisition of Structures in 3D) and validate it against traditional approaches to reconstruct CME shock using SOLO data exclusively. Additionally, we estimate early CME propagation characteristics based on coronal dimming evolution with the DIRECD method. Results indicate that the filament height increased from 28 to 616 Mm (0.04 to 0.89 Rs) within 30 minutes (11:05 to 11:35 UT), reaching peak velocity of around 648 km/s and acceleration of around 1624 m/s$^2$. At 11:45 UT, the filament deflected by 12{\deg} to a height of 841 Mm (1.21 Rs), while the CME shock expanded from 383 to 837 Mm (0.55 to 1.2 Rs) over 10 minutes. Key parameters include a CME direction inclined by 6{\deg}, a 21{\deg} half-width, and a 1.12 Rs cone height at the dimming's impulsive phase end. This event demonstrates that expanding dimming correlates with early CME development, with the DIRECD method linking 2D dimming to 3D CME evolution. These insights underscore the value of multi-viewpoint observations and advanced reconstructions for improving space weather forecasting.

astro-ph.SR↗

arXiv2024

Estimating early coronal mass ejection propagation direction with DIRECD during the severe May 8 and follow-up June 8, 2024 events

On May 8, 2024, solar active region 13664 produced an X-class flare, several M-class flares, and multiple Earth-directed Coronal Mass Ejections (CMEs). The initial CME caused coronal dimmings, characterized by localized reductions in extreme-ultraviolet (EUV) emissions, indicating mass loss and expansion during the eruption. After one solar rotation, on June 8, 2024, the same region produced another M-class flare followed by coronal dimmings observed by the SDO and STEREO spacecraft. We analyzed early CME evolution and direction from coronal dimming expansion at the end of the impulsive phase using the DIRECD (Dimming Inferred Estimation of CME Direction) method. To validate the 3D CME cone, we compared CME properties from the low corona with white-light coronagraph data. The May 8 CME expanded radially, with a 7.7 deg inclination, 70 deg angular width, and 0.81 Rsun cone height, while the June 8 CME had a 15.7 deg inclination, 81 deg width, and 0.89 Rsun height. Our study shows that tracking low coronal signatures, like coronal dimming expansion, can estimate CME direction early, providing crucial lead time for space weather forecasts.

astro-ph.SR↗

arXiv2024

The McNish and Lincoln Solar Activity Predictions: The Method and its Performance

The McNish and Lincoln (ML) method, introduced in 1949, was one of the first attempts to produce mid-term forecasts of solar activity, up to 12 months ahead. However, it has been poorly described and evaluated in the past literature, in particular its actual operational implementation by NOAA. Here, we reconstruct the exact formulation of the method, as it was applied since the early 1970s, and we provide a full mathematical derivation of the prediction errors. For bench-marking the method, we also produce monthly predictions over the past 190 years, from 1833 (Cycle 8) to 2023 (Cycle 25), and develop statistics of the differences between the predictions and the observed 13-month smoothed sunspot number (SSN) time series, according to the phase in the solar cycle. Our analysis shows that the ML method is heavily constrained because it is primarily based on the mean of all past cycles, which imposes a fixed amplitude and length and suffers from a temporal smearing that grows towards the end of the solar cycle. We find that predictions are completely unreliable in the first 12 months of the cycle, and over the last two years preceding the ending minimum (around 130 months), and beyond this minimum. By contrast, in the course of the cycle (months 18 to 65), ML predictions prove to be reliable over a time range of up to 50 months (4.2 years), thus much longer than the 12-month conventional range used so far. However, we find that predictions then suffer from systematic under-(over-)estimates for cycles that have a higher (lower) amplitude than the base mean cycle. Overall, we conclude that although the ML method provides valid prediction errors, it suffers from strong limitations, with very little room for improvement, as it indifferently merges all past cycles into a single fixed statistics.

astro-ph.SR↗

arXiv2023

Coronal dimmings as indicators of early CME propagation direction

Coronal mass ejections (CMEs) are solar eruptions of plasma and magnetic fields that significantly impact Space Weather, causing disruptions in technological systems and potential damage to power grids when directed towards Earth. Traditional coronagraphs along the Sun-Earth line struggle to precisely track the early evolution of Earth-directed CMEs. Coronal dimmings, localized reductions in extreme-ultraviolet (EUV) and soft X-ray emissions, are key indicators of CMEs in the low corona, resulting from mass loss and expansion during the eruption. This study introduces a novel method, DIRECD (Dimming InfeRred Estimate of CME Direction), to estimate the early propagation direction of CMEs based on the expansion of coronal dimmings. The approach involves 3D simulations of CMEs using a geometric cone model, exploring parameters like width, height, source location, and deflection from the radial direction. The dominant direction of dimming evolution is then determined, and an inverse problem is solved to reconstruct an ensemble of CME cones at various heights, widths, and deflections. By comparing the CME orthogonal projections onto the solar sphere with the dimming geometry, the 3D CME direction is derived. Validated through case studies on October 1, 2011, and September 6, 2011, the DIRECD method reveals the early propagation directions of CMEs. The CME on October 1, 2011, predominantly expands towards the South-East, while the CME on September 6, 2011, inclines towards the North-West. These findings align with previous studies using multi-viewpoint coronagraphic observations. The study demonstrates the utility of coronal dimming information for early CME direction estimation, providing valuable data for space weather forecasting and mitigating potential adverse impacts on Earth before observation in coronographs' field-of-view.

astro-ph.SR↗

arXiv2023

GPT-4 Technical Report

We report the development of GPT-4, a large-scale, multimodal model which can accept image and text inputs and produce text outputs. While less capable than humans in many real-world scenarios, GPT-4 exhibits human-level performance on various professional and academic benchmarks, including passing a simulated bar exam with a score around the top 10% of test takers. GPT-4 is a Transformer-based model pre-trained to predict the next token in a document. The post-training alignment process results in improved performance on measures of factuality and adherence to desired behavior. A core component of this project was developing infrastructure and optimization methods that behave predictably across a wide range of scales. This allowed us to accurately predict some aspects of GPT-4's performance based on models trained with no more than 1/1,000th the compute of GPT-4.

cs.CL↗

arXiv2022

Leveraging Structure for Improved Classification of Grouped Biased Data

We consider semi-supervised binary classification for applications in which data points are naturally grouped (e.g., survey responses grouped by state) and the labeled data is biased (e.g., survey respondents are not representative of the population). The groups overlap in the feature space and consequently the input-output patterns are related across the groups. To model the inherent structure in such data, we assume the partition-projected class-conditional invariance across groups, defined in terms of the group-agnostic feature space. We demonstrate that under this assumption, the group carries additional information about the class, over the group-agnostic features, with provably improved area under the ROC curve. Further assuming invariance of partition-projected class-conditional distributions across both labeled and unlabeled data, we derive a semi-supervised algorithm that explicitly leverages the structure to learn an optimal, group-aware, probability-calibrated classifier, despite the bias in the labeled data. Experiments on synthetic and real data demonstrate the efficacy of our algorithm over suitable baselines and ablative models, spanning standard supervised and semi-supervised learning approaches, with and without incorporating the group directly as a feature.

stat.ML↗

arXiv2022

Maximal growth rate of the ascending phase of a sunspot cycle for predicting its amplitude

Forecasting the solar cycle amplitude is important for a better understanding of the solar dynamo as well as for many space weather applications. We demonstrated a steady relationship between the maximal growth rate of sunspot activity in the ascending phase of a cycle and the subsequent cycle amplitude on the basis of four data sets of solar activity indices: total sunspot numbers, hemispheric sunspot numbers from the new catalogue from 1874 onwards, total sunspot areas, and hemispheric sunspot areas. For all the data sets, a linear regression based on the maximal growth rate precursor shows a significant correlation. Validation of predictions for cycles 1-24 shows high correlations between the true and predicted cycle amplitudes reaching r = 0.93 for the total sunspot numbers. The lead time of the predictions varies from 2 to 49 months, with a mean value of 21 months. Furthermore, we demonstrated that the sum of maximal growth rate indicators determined separately for the north and the south hemispheric sunspot numbers provides more accurate predictions than that using total sunspot numbers. The advantages reach 27% and 11% on average in terms of rms and correlation coefficient, respectively. The superior performance is also confirmed with hemispheric sunspot areas with respect to total sunspot areas. The maximal growth rate of sunspot activity in the ascending phase of a solar cycle serves as a reliable precursor of the subsequent cycle amplitude. Furthermore, our findings provide a strong foundation for supporting regular monitoring, recording, and predictions of solar activity with hemispheric sunspot data, which capture the asymmetric behaviour of the solar activity and solar magnetic field and enhance solar cycle prediction methods.

astro-ph.SR↗

arXiv2022

WebGPT: Browser-assisted question-answering with human feedback

We fine-tune GPT-3 to answer long-form questions using a text-based web-browsing environment, which allows the model to search and navigate the web. By setting up the task so that it can be performed by humans, we are able to train models on the task using imitation learning, and then optimize answer quality with human feedback. To make human evaluation of factual accuracy easier, models must collect references while browsing in support of their answers. We train and evaluate our models on ELI5, a dataset of questions asked by Reddit users. Our best model is obtained by fine-tuning GPT-3 using behavior cloning, and then performing rejection sampling against a reward model trained to predict human preferences. This model's answers are preferred by humans 56% of the time to those of our human demonstrators, and 69% of the time to the highest-voted answer from Reddit.

cs.CL↗

arXiv2021

Evaluating Large Language Models Trained on Code

We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.

cs.LG↗

arXiv2021

Hemispheric sunspot numbers 1874--2020

We create a continuous series of daily and monthly hemispheric sunspot numbers (HSNs) from 1874 to 2020, which will be continuously expanded in the future with the HSNs provided by SILSO. Based on the available daily measurements of hemispheric sunspot areas from 1874 to 2016 from Greenwich Royal Observatory and NOAA, we derive the relative fractions of the northern and southern activity. These fractions are applied to the international sunspot number (ISN) to derive the HSNs. This method and obtained data are validated against published HSNs for the period 1945--2020. We provide a continuous data series and catalogue of daily, monthly mean, and 13-month smoothed monthly mean HSNs for the time range 1874--2020 that are consistent with the newly calibrated ISN. Validation of the reconstructed HSNs against the direct data available since 1945 reveals a high level of consistency, with a correlation of r=0.94 (0.97) for the daily (monthly) data. The cumulative hemispheric asymmetries for cycles 12-24 give a mean value of 16%, with no obvious pattern in north-south predominance over the cycle evolution. The strongest asymmetry occurs for cycle no. 19, in which the northern hemisphere shows a cumulated predominance of 42%. The phase shift between the peaks of solar activity in the two hemispheres may be up to 28 months, with a mean absolute value of 16.4 months. The phase shifts reveal an overall asymmetry of the northern hemisphere reaching its cycle maximum earlier (in 10 out of 13 cases). Relating the ISN and HSN peak growth rates during the cycle rise phase with the cycle amplitude reveals higher correlations when considering the two hemispheres individually, with r = 0.9. Our findings demonstrate that empirical solar cycle prediction methods can be improved by investigating the solar cycle dynamics in terms of the hemispheric sunspot numbers.

astro-ph.SR↗

arXiv2020

New mixture models for decoy-free false discovery rate estimation in mass-spectrometry proteomics

Motivation: Accurate estimation of false discovery rate (FDR) of spectral identification is a central problem in mass spectrometry-based proteomics. Over the past two decades, target decoy approaches (TDAs) and decoy-free approaches (DFAs), have been widely used to estimate FDR. TDAs use a database of decoy species to faithfully model score distributions of incorrect peptide-spectrum matches (PSMs). DFAs, on the other hand, fit two-component mixture models to learn the parameters of correct and incorrect PSM score distributions. While conceptually straightforward, both approaches lead to problems in practice, particularly in experiments that push instrumentation to the limit and generate low fragmentation-efficiency and low signal-to-noise-ratio spectra. Results: We introduce a new decoy-free framework for FDR estimation that generalizes present DFAs while exploiting more search data in a manner similar to TDAs. Our approach relies on multi-component mixtures, in which score distributions corresponding to the correct PSMs, best incorrect PSMs, and second-best incorrect PSMs are modeled by the skew normal family. We derive EM algorithms to estimate parameters of these distributions from the scores of best and second-best PSMs associated with each experimental spectrum. We evaluate our models on multiple proteomics datasets and a HeLa cell digest case study consisting of more than a million spectra in total. We provide evidence of improved performance over existing DFAs and improved stability and speed over TDAs without any performance degradation. We propose that the new strategy has the potential to extend beyond peptide identification and reduce the need for TDA on all analytical platforms.

q-bio.QM↗

arXiv2017

Identifiability of two-component skew normal mixtures with one known component

We give sufficient identifiability conditions for estimating mixing proportions in two-component mixtures of skew normal distributions with one known component. We consider the univariate case as well as two multivariate extensions: a multivariate skew normal distribution (MSN) by Azzalini and Dalla Valle (1996) and the canonical fundamental skew normal distribution (CFUSN) by Arellano-Valle and Genton (2005). The characteristic function of the CFUSN distribution is additionally derived.

math.ST↗

arXiv2017

Recovering True Classifier Performance in Positive-Unlabeled Learning

A common approach in positive-unlabeled learning is to train a classification model between labeled and unlabeled data. This strategy is in fact known to give an optimal classifier under mild conditions; however, it results in biased empirical estimates of the classifier performance. In this work, we show that the typically used performance measures such as the receiver operating characteristic curve, or the precision-recall curve obtained on such data can be corrected with the knowledge of class priors; i.e., the proportions of the positive and negative examples in the unlabeled data. We extend the results to a noisy setting where some of the examples labeled positive are in fact negative and show that the correction also requires the knowledge of the proportion of noisy examples in the labeled positives. Using state-of-the-art algorithms to estimate the positive class prior and the proportion of noise, we experimentally evaluate two correction approaches and demonstrate their efficacy on real-life data.

stat.ML↗

arXiv2017

Estimating the class prior and posterior from noisy positives and unlabeled data

We develop a classification algorithm for estimating posterior distributions from positive-unlabeled data, that is robust to noise in the positive labels and effective for high-dimensional data. In recent years, several algorithms have been proposed to learn from positive-unlabeled data; however, many of these contributions remain theoretical, performing poorly on real high-dimensional data that is typically contaminated with noise. We build on this previous work to develop two practical classification algorithms that explicitly model the noise in the positive labels and utilize univariate transforms built on discriminative classifiers. We prove that these univariate transforms preserve the class prior, enabling estimation in the univariate space and avoiding kernel density estimation for high-dimensional data. The theoretical development and both parametric and nonparametric algorithms proposed here constitutes an important step towards wide-spread use of robust classification algorithms for positive-unlabeled data.

stat.ML↗

arXiv2016

Nonparametric semi-supervised learning of class proportions

The problem of developing binary classifiers from positive and unlabeled data is often encountered in machine learning. A common requirement in this setting is to approximate posterior probabilities of positive and negative classes for a previously unseen data point. This problem can be decomposed into two steps: (i) the development of accurate predictors that discriminate between positive and unlabeled data, and (ii) the accurate estimation of the prior probabilities of positive and negative examples. In this work we primarily focus on the latter subproblem. We study nonparametric class prior estimation and formulate this problem as an estimation of mixing proportions in two-component mixture models, given a sample from one of the components and another sample from the mixture itself. We show that estimation of mixing proportions is generally ill-defined and propose a canonical form to obtain identifiability while maintaining the flexibility to model any distribution. We use insights from this theory to elucidate the optimization surface of the class priors and propose an algorithm for estimating them. To address the problems of high-dimensional density estimation, we provide practical transformations to low-dimensional spaces that preserve class priors. Finally, we demonstrate the efficacy of our method on univariate and multivariate data.

stat.ML↗