arXiv subjects

David A. van Dyk

Publications and source records attributed to David A. van Dyk.

At least 19 recordsLinked to original sources

arXiv2026

Bayesian Modeling and Prediction of Generalized Contact Matrices

Social contact matrices are essential tools in infectious disease epidemiology as they quantify close-range human contact patterns which directly drive the transmission of airborne infectious diseases. In this work we propose a Bayesian modeling framework for inferring generalized contact matrices which stratify contact matrices beyond contemporary age dimensions. The model is designed to satisfy fundamental structural assumptions of contacts while leveraging tensor structures and smoothing constraints to make high-dimensional matrix estimation computationally feasible and statistically stable. We discover a link between multi-dimensional matrix stratification subject to structural constraints with the theory of contingency tables. This enables us to approach a challenging missing-data problem commonly encountered in real-world analysis where feature information on the contacts is unobserved. We benchmark the framework against existing methods through simulation studies and illustrate the framework's practical utility through two real-world datasets: BICS (United States) and COVIMOD (Germany). Our models are implemented in an open-source Python package to facilitate adoption in the wider scientific community.

stat.ME↗

arXiv2026

Using fractional derivatives to derive marginal densities

This paper presents a novel method for analytical derivations of marginal densities using the fractional derivatives of moment-generating functions. Although the method requires likelihood functions to take specific forms, its assumptions are otherwise modest. It only requires that the prior moment-generating functions exist, are finite, and are continuous and differentiable at certain points. We also present the probabilistic and statistical insights behind this method.

stat.ME↗

arXiv2025

StratLearn-z: Improved photo-$z$ estimation from spectroscopic data subject to selection effects

A precise measurement of photometric redshifts (photo-z) is key for the success of modern photometric galaxy surveys. Machine learning (ML) methods show great promise in this context, but suffer from covariate shift (CS) in training sets due to selection bias where interesting sources are underrepresented, and the corresponding ML models show poor generalisation properties. We present an application of the StratLearn method to the estimation of photo-z, validating against simulations where we enforce the presence of CS to different degrees. StratLearn is a statistically principled approach that relies on splitting the source and target datasets into strata based on estimated propensity scores (i.e. the probability for an object to be in the source set given its observed covariates). After stratification, two conditional density estimators are fit separately to each stratum, then combined via a weighted average. We benchmark our results against the GPz algorithm, quantifying the performance of the two codes with a set of metrics. Our results show that the StratLearn-z metrics are only marginally affected by the presence of CS, while GPz shows a significant degradation of performance in the photo-z prediction for fainter objects. For the strongest CS scenario, StratLearn-z yields a reduced fraction of catastrophic errors, a factor of 2 improvement for the RMSE and one order of magnitude improvement on the bias. We also assess the quality of the conditional redshift estimates with the probability integral transform (PIT). The PIT distribution obtained from StratLearn-z features fat fewer outliers and is symmetric, i.e. the predictions appear to be centered around the true redshift value, despite showing a conservative estimation of the spread of the conditional redshift distributions. Our julia implementation of the method is available at https://github.com/chiaramoretti/StratLearn-z.

astro-ph.CO↗

arXiv2024

Six Maxims of Statistical Acumen for Astronomical Data Analysis

The production of complex astronomical data is accelerating, especially with newer telescopes producing ever more large-scale surveys. The increased quantity, complexity, and variety of astronomical data demand a parallel increase in skill and sophistication in developing, deciding, and deploying statistical methods. Understanding limitations and appreciating nuances in statistical and machine learning methods and the reasoning behind them is essential for improving data-analytic proficiency and acumen. Aiming to facilitate such improvement in astronomy, we delineate cautionary tales in statistics via six maxims, with examples drawn from the astronomical literature. Inspired by the significant quality improvement in business and manufacturing processes by the routine adoption of Six Sigma, we hope the routine reflection on these Six Maxims will improve the quality of both data analysis and scientific findings in astronomy.

astro-ph.IM↗

arXiv2024

Separating States in Astronomical Sources Using Hidden Markov Models: With a Case Study of Flaring and Quiescence on EV Lac

We present a new method to distinguish between different states (e.g., high and low, quiescent and flaring) in astronomical sources with count data. The method models the underlying physical process as latent variables following a continuous-space Markov chain that determines the expected Poisson counts in observed light curves in multiple passbands. For the underlying state process, we consider several autoregressive processes, yielding continuous-space hidden Markov models of varying complexity. Under these models, we can infer the state that the object is in at any given time. The continuous state predictions from these models are then dichotomized with the help of a finite mixture model to produce state classifications. We apply these techniques to X-ray data from the active dMe flare star EV Lac, splitting the data into quiescent and flaring states. We find that a first-order vector autoregressive process efficiently separates flaring from quiescence: flaring occurs over 30-40% of the observation durations, a well-defined persistent quiescent state can be identified, and the flaring state is characterized by higher plasma temperatures and emission measures.

astro-ph.SR↗

arXiv2024

Effect of Systematic Uncertainties on Density and Temperature Estimates in Coronae of Capella

We estimate the coronal density of Capella using the O VII and Fe XVII line systems in the soft X-ray regime that have been observed over the course of the Chandra mission. Our analysis combines measures of error due to uncertainty in the underlying atomic data with statistical errors in the Chandra data to derive meaningful overall uncertainties on the plasma density of the coronae of Capella. We consider two Bayesian frameworks. First, the so-called pragmatic-Bayesian approach considers the atomic data and their uncertainties as fully specified and uncorrectable. The fully-Bayesian approach, on the other hand, allows the observed spectral data to update the atomic data and their uncertainties, thereby reducing the overall errors on the inferred parameters. To incorporate atomic data uncertainties, we obtain a set of atomic data replicates, the distribution of which captures their uncertainty. A principal component analysis of these replicates allows us to represent the atomic uncertainty with a lower-dimensional multivariate Gaussian distribution. A $t$-distribution approximation of the uncertainties of a subset of plasma parameters including a priori temperature information, obtained from the temperature-sensitive-only Fe XVII spectral line analysis, is carried forward into the density- and temperature-sensitive O VII spectral line analysis. Markov Chain Monte Carlo based model fitting is implemented including Multi-step Monte Carlo Gibbs Sampler and Hamiltonian Monte Carlo. Our analysis recovers an isothermally approximated coronal plasma temperature of $\approx$5 MK and a coronal plasma density of $\approx$10$^{10}$ cm$^{-3}$, with uncertainties of 0.1 and 0.2 dex respectively.

astro-ph.SR↗

arXiv2024

Joint Deconvolution of Astronomical Images in the Presence of Poisson Noise

We present a new method for joint likelihood deconvolution (Jolideco) of a set of astronomical observations of the same sky region in the presence of Poisson noise. The observations may be obtained from different instruments with different resolution, and different point spread functions. Jolideco reconstructs a single flux image by optimizing the posterior distribution based on the joint Poisson likelihood of all observations under a patch-based image prior. The patch prior is parameterised via a Gaussian Mixture model which we train on high-signal-to-noise astronomical images, including data from the James Webb Telescope and the GLEAM radio survey. This prior favors correlation structures among the reconstructed pixel intensities that are characteristic of those observed in the training images. It is, however, not informative for the mean or scale of the reconstruction. By applying the method to simulated data we show that the combination of multiple observations and the patch-based prior leads to much improved reconstruction quality in many different source scenarios and signal to noise regimes. We demonstrate that with the patch prior Jolideco yields superior reconstruction quality relative to alternative standard methods such as the Richardson-Lucy method. We illustrate the results of Jolideco applied to example data from the Chandra X-ray Observatory and the Fermi-LAT Gamma-ray Space Telescope. By comparing the measured width of a counts based and the corresponding Jolideco flux profile of an X-ray filament in SNR 1E 0102.2-721} we find the deconvolved width of 0.58+- 0.02 arcsec to be consistent with the theoretical expectation derived from the known width of the PSF.

astro-ph.IM↗

arXiv2024

Improved Weak Lensing Photometric Redshift Calibration via StratLearn and Hierarchical Modeling

Discrepancies between cosmological parameter estimates from cosmic shear surveys and from recent Planck cosmic microwave background measurements challenge the ability of the highly successful $Λ$CDM model to describe the nature of the Universe. To rule out systematic biases in cosmic shear survey analyses, accurate redshift calibration within tomographic bins is key. In this paper, we improve photo-$z$ calibration via Bayesian hierarchical modeling of full galaxy photo-$z$ conditional densities, by employing $\textit{StratLearn}$, a recently developed statistical methodology, which accounts for systematic differences in the distribution of the spectroscopic training/source set and the photometric target set. Using realistic simulations that were designed to resemble the KiDS+VIKING-450 dataset, we show that $\textit{StratLearn}$-estimated conditional densities improve the galaxy tomographic bin assignment, and that our $\textit{StratLearn}$-Bayesian framework leads to nearly unbiased estimates of the target population means. This leads to a factor of $\sim 2$ improvement upon the previously best photo-$z$ calibration method. Our approach delivers a maximum bias per tomographic bin of $Δ\langle z \rangle = 0.0095 \pm 0.0089$, with an average absolute bias of $0.0052 \pm 0.0067$ across the five tomographic bins.

astro-ph.CO↗

arXiv2023

TD-CARMA: Painless, accurate, and scalable estimates of gravitational-lens time delays with flexible CARMA processes

Cosmological parameters encoding our understanding of the expansion history of the Universe can be constrained by the accurate estimation of time delays arising in gravitationally lensed systems. We propose TD-CARMA, a Bayesian method to estimate cosmological time delays by modelling the observed and irregularly sampled light curves as realizations of a Continuous Auto-Regressive Moving Average (CARMA) process. Our model accounts for heteroskedastic measurement errors and microlensing, an additional source of independent extrinsic long-term variability in the source brightness. The semi-separable structure of the CARMA covariance matrix allows for fast and scalable likelihood computation using Gaussian Process modeling. We obtain a sample from the joint posterior distribution of the model parameters using a nested sampling approach. This allows for ``painless'' Bayesian Computation, dealing with the expected multi-modality of the posterior distribution in a straightforward manner and not requiring the specification of starting values or an initial guess for the time delay, unlike existing methods. In addition, the proposed sampling procedure automatically evaluates the Bayesian evidence, allowing us to perform principled Bayesian model selection. TD-CARMA is parsimonious, and typically includes no more than a dozen unknown parameters. We apply TD-CARMA to six doubly lensed quasars HS 2209+1914, SDSS J1001+5027, SDSS J1206+4332, SDSS J1515+1511, SDSS J1455+1447, SDSS J1349+1227, estimating their time delays as $-21.96 \pm 1.448$, $120.93 \pm 1.015$, $111.51 \pm 1.452$, $210.80 \pm 2.18$, $45.36 \pm 1.93$ and $432.05 \pm 1.950$ respectively. These estimates are consistent with those derived in the relevant literature, but are typically two to four times more precise.

astro-ph.IM↗

arXiv2023

Stratified Learning: A General-Purpose Statistical Method for Improved Learning under Covariate Shift

We propose a simple, statistically principled, and theoretically justified method to improve supervised learning when the training set is not representative, a situation known as covariate shift. We build upon a well-established methodology in causal inference, and show that the effects of covariate shift can be reduced or eliminated by conditioning on propensity scores. In practice, this is achieved by fitting learners within strata constructed by partitioning the data based on the estimated propensity scores, leading to approximately balanced covariates and much-improved target prediction. We demonstrate the effectiveness of our general-purpose method on two contemporary research questions in cosmology, outperforming state-of-the-art importance weighting methods. We obtain the best reported AUC (0.958) on the updated "Supernovae photometric classification challenge", and we improve upon existing conditional density estimation of galaxy redshift from Sloan Data Sky Survey (SDSS) data.

stat.ML↗

arXiv2022

Identifying diffuse spatial structures in high-energy photon lists

Data from high-energy observations are usually obtained as lists of photon events. A common analysis task for such data is to identify whether diffuse emission exists, and to estimate its surface brightness, even in the presence of point sources that may be superposed. We have developed a novel non-parametric event list segmentation algorithm to divide up the field of view into distinct emission components. We use photon location data directly, without binning them into an image. We first construct a graph from the Voronoi tessellation of the observed photon locations and then grow segments using a new adaptation of seeded region growing, that we call Seeded Region Growing on Graph, after which the overall method is named SRGonG. Starting with a set of seed locations, this results in an over-segmented dataset, which SRGonG then coalesces using a greedy algorithm where adjacent segments are merged to minimize a model comparison statistic; we use the Bayesian Information Criterion. Using SRGonG we are able to identify point-like and diffuse extended sources in the data with equal facility. We validate SRGonG using simulations, demonstrating that it is capable of discerning irregularly shaped low surface-brightness emission structures as well as point-like sources with strengths comparable to that seen in typical X-ray data. We demonstrate SRGonG's use on the Chandra data of the Antennae galaxies, and show that it segments the complex structures appropriately.

astro-ph.IM↗

arXiv2021

Bayesian model selection in additive partial linear models via locally adaptive splines

We provide a flexible framework for selecting among a class of additive partial linear models that allows both linear and nonlinear additive components. In practice, it is challenging to determine which additive components should be excluded from the model while simultaneously determining whether nonzero additive components should be represented as linear or non-linear components in the final model. In this paper, we propose a Bayesian model selection method that is facilitated by a carefully specified class of models, including the choice of a prior distribution and the nonparametric model used for the nonlinear additive components. We employ a series of latent variables that determine the effect of each variable among the three possibilities (no effect, linear effect, and nonlinear effect) and that simultaneously determine the knots of each spline for a suitable penalization of smooth functions. The use of a pseudo-prior distribution along with a collapsing scheme enables us to deploy well-behaved Markov chain Monte Carlo samplers, both for model selection and for fitting the preferred model. Our method and algorithm are deployed on a suite of numerical studies and are applied to a nutritional epidemiology study. The numerical results show that the proposed methodology outperforms previously available methods in terms of effective sample sizes of the Markov chain samplers and the overall misclassification rates.

stat.ME↗

arXiv2021

Concordance: In-flight Calibration of X-ray Telescopes without Absolute References

We describe a process for cross-calibrating the effective areas of X-ray telescopes that observe common targets. The targets are not assumed to be "standard candles" in the classic sense, in that we assume that the source fluxes have well-defined, but {\it a priori} unknown values. Using a technique developed by Chen et al. (2019, arXiv:1711.09429) that involves a statistical method called {\em shrinkage estimation}, we determine effective area correction factors for each instrument that brings estimated fluxes into the best agreement, consistent with prior knowledge of their effective areas. We expand the technique to allow unique priors on systematic uncertainties in effective areas for each X-ray astronomy instrument and to allow correlations between effective areas in different energy bands. We demonstrate the method with several data sets from various X-ray telescopes.

astro-ph.IM↗

arXiv2021

eBASCS: Disentangling Overlapping Astronomical Sources II, using Spatial, Spectral, and Temporal Information

The analysis of individual X-ray sources that appear in a crowded field can easily be compromised by the misallocation of recorded events to their originating sources. Even with a small number of sources, that nonetheless have overlapping point spread functions, the allocation of events to sources is a complex task that is subject to uncertainty. We develop a Bayesian method designed to sift high-energy photon events from multiple sources with overlapping point spread functions, leveraging the differences in their spatial, spectral, and temporal signatures. The method probabilistically assigns each event to a given source. Such a disentanglement allows more detailed spectral or temporal analysis to focus on the individual component in isolation, free of contamination from other sources or the background. We are also able to compute source parameters of interest like their locations, relative brightness, and background contamination, while accounting for the uncertainty in event assignments. Simulation studies that include event arrival time information demonstrate that the temporal component improves event disambiguation beyond using only spatial and spectral information. The proposed methods correctly allocate up to 65% more events than the corresponding algorithms that ignore event arrival time information. We apply our methods to two stellar X-ray binaries, UV Cet and HBC515 A, observed with Chandra. We demonstrate that our methods are capable of removing the contamination due to a strong flare on UV Cet B in its companion approximately 40 times weaker during that event, and that evidence for spectral variability at timescales of a few ks can be determined in HBC515 Aa and HBC515 Ab.

astro-ph.IM↗

arXiv2021

Identification of high-energy astrophysical point sources via hierarchical Bayesian nonparametric clustering

The light we receive from distant astrophysical objects carries information about their origins and the physical mechanisms that power them. The study of these signals, however, is complicated by the fact that observations are often a mixture of the light emitted by multiple localized sources situated in a spatially-varying background. A general algorithm to achieve robust and accurate source identification in this case remains an open question in astrophysics. This paper focuses on high-energy light (such as X-rays and gamma-rays), for which observatories can detect individual photons (quanta of light), measuring their incoming direction, arrival time, and energy. Our proposed Bayesian methodology uses both the spatial and energy information to identify point sources, that is, separate them from the spatially-varying background, to estimate their number, and to compute the posterior probabilities that each photon originated from each identified source. This is accomplished via a Dirichlet process mixture while the background is simultaneously reconstructed via a flexible Bayesian nonparametric model based on B-splines. Our proposed method is validated with a suite of simulation studies and illustrated with an application to a complex region of the sky observed by the \emph{Fermi} Gamma-ray Space Telescope.

stat.AP↗

arXiv2020

STACCATO: A Novel Solution to Supernova Photometric Classification with Biased Training Sets

We present a new solution to the problem of classifying Type Ia supernovae from their light curves alone given a spectroscopically confirmed but biased training set, circumventing the need to obtain an observationally expensive unbiased training set. We use Gaussian processes (GPs) to model the supernovae's (SN) light curves, and demonstrate that the choice of covariance function has only a small influence on the GPs ability to accurately classify SNe. We extend and improve the approach of Richards et al (2012} -- a diffusion map combined with a random forest classifier -- to deal specifically with the case of biassed training sets. We propose a novel method, called STACCATO (SynThetically Augmented Light Curve ClassificATiOn') that synthetically augments a biased training set by generating additional training data from the fitted GPs. Key to the success of the method is the partitioning of the observations into subgroups based on their propensity score of being included in the training set. Using simulated light curve data, we show that STACCATO increases performance, as measured by the area under the Receiver Operating Characteristic curve (AUC), from 0.93 to 0.96, close to the AUC of 0.977 obtained using the 'gold standard' of an unbiased training set and significantly improving on the previous best result of 0.88. STACCATO also increases the true positive rate for SNIa classification by up to a factor of 50 for high-redshift/low brightness SNe.

astro-ph.IM↗

arXiv2019

Testing One Hypothesis Multiple Times: The Multidimensional Case

The identification of new rare signals in data, the detection of a sudden change in a trend, and the selection of competing models, are among the most challenging problems in statistical practice. These challenges can be tackled using a test of hypothesis where a nuisance parameter is present only under the alternative, and a computationally efficient solution can be obtained by the "Testing One Hypothesis Multiple times" (TOHM) method. In the one-dimensional setting, a fine discretization of the space of the non-identifiable parameter is specified, and a global p-value is obtained by approximating the distribution of the supremum of the resulting stochastic process. In this paper, we propose a computationally efficient inferential tool to perform TOHM in the multidimensional setting. Here, the approximations of interest typically involve the expected Euler Characteristics (EC) of the excursion set of the underlying random field. We introduce a simple algorithm to compute the EC in multiple dimensions and for arbitrary large significance levels. This leads to an highly generalizable computational tool to perform inference under non-standard regularity conditions.

stat.ME↗

arXiv2019

Testing One Hypothesis Multiple times

In applied settings, tests of hypothesis where a nuisance parameter is only identifiable under the alternative often reduces into one of Testing One Hypothesis Multiple times (TOHM). Specifically, a fine discretization of the space of the non-identifiable parameter is specified, and the null hypothesis is tested against a set of sub-alternative hypothesis, one for each point of the discretization. The resulting sub-test statistics are then combined to obtain a global p-value. In this paper, we discuss a computationally efficient inferential tool to perform TOHM under stringent significance requirements, such as those typically required in the physical sciences, (e.g., p-value $<10^{-7}$). The resulting procedure leads to a generalized approach to perform inference under non-standard conditions, including non-nested models comparisons.

stat.ME↗