arXiv subjects

Jeffrey D. Scargle

Publications and source records attributed to Jeffrey D. Scargle.

At least 37 records · Page 2Linked to original sources

arXiv2010

False-alarm probability in relation to over-sampled power spectra, with application to Super-Kamiokande solar neutrino data

The term "false-alarm probability" denotes the probability that at least one out of M independent power values in a prescribed search band of a power spectrum computed from a white-noise time series is expected to be as large as or larger than a given value. The usual formula is based on the assumption that powers are distributed exponentially, as one expects for power measurements of normally distributed random noise. However, in practice one typically examines peaks in an over-sampled power spectrum. It is therefore more appropriate to compare the strength of a particular peak with the distribution of peaks in over-sampled power spectra derived from normally distributed random noise. We show that this leads to a formula for the false-alarm probability that is more conservative than the familiar formula. We also show how to combine these results with a Bayesian method for estimating the probability of the null hypothesis (that there is no oscillation in the time series), and we discuss as an example the application of these procedures to Super-Kamiokande solar neutrino data.

hep-ph↗

arXiv2010

Threshold for Extended Emission in Short Gamma-Ray Bursts

The initial pulse complex (IPC) in short gamma-ray bursts is sometimes accompanied by a softer, low-intensity extended emission (EE) component. In cases where such a component is not observed, it is not clear if it is present but below the detection threshold. Using Bayesian Block (BB) methods, we measure the EE component and show that it is present in one quarter of a Swift/BAT sample of 51 short bursts, as was found for the Compton/BATSE sample. We simulate bursts with EE to calibrate the BAT threshold for EE detection and show that this component would have been detected in nearly half of BAT short bursts if it were present, to intensities ~ 10^-2 counts cm^-2 s^-1, a factor of five lower than actually observed in short bursts. In the BAT sample the ratio of average EE intensity to IPC peak intensity, Rint, ranges over a factor of 25, Rint ~ 3 x 10^-3 to 8 x 10^-2. In comparison, for the average of the 39 bursts without an EE component, the 2-sigma upper limit is Rint < 8 x 10^-4. These results suggest that a physical threshold effect operates near Rint ~ few x 10^-3, below which the EE component is not manifest.

astro-ph.HE↗

arXiv2009

Photon dispersion in causal sets

A very small dispersion in the speed of light may be observable in Fermi time- and energy-tagged data on variable sources, such as gamma-ray bursts (GRB) and active galactic nuclei (AGN). We describe a method to compute the size of this effect by applying the Feynman sum-over-histories formalism for relativistic quantum electrodynamics to a discrete model of space-time called \emph{causal set theory}.

astro-ph.IM↗

arXiv2008

A Generalized Publication Bias Model

Scargle (2000) has discussed Rosenthal and Rubin's (1978) "fail-safe number" (FSN) method for estimating the number of unpublished studies in meta-analysis. He concluded that this FSN cannot possibly be correct because a central assumption the authors used conflicts with the very definition of publication bias. While this point has been made by others before (Elsahoff, 1978; Darlington, 1980; Thomas, 1985, Iyengar & Greenhouse, 1988), Scargle showed, by way of a simple 2-parameter model, how far off Rosenthal & Rubin' s estimate can be in practice. However, his results relied on the assumption that the decision variable is normally distributed with zero mean. In this case the ratio of unpublished to published papers is large only in a tiny region of the parameter plane. Building on these results, we now show that (1) Replacing densities with probability masses greatly simplifies Scargle's derivations and permits an explicit statement of the relation between the probability alpha of Type I errors and the step-size beta; (2) This result does not require any distribution assumptions; (3) The distinction between 1-sided and 2-sided rejection regions becomes immaterial; (4) This distribution-free approach leads to an immediate generalization to partitions involving more than two intervals, and thus covers more general selection functions.

stat.ME↗

arXiv2008

Using Bayesian Blocks to Partition Self-Organizing Maps

Self organizing maps (SOMs) are widely-used for unsupervised classification. For this application, they must be combined with some partitioning scheme that can identify boundaries between distinct regions in the maps they produce. We discuss a novel partitioning scheme for SOMs based on the Bayesian Blocks segmentation algorithm of Scargle [1998]. This algorithm minimizes a cost function to identify contiguous regions over which the values of the attributes can be represented as approximately constant. Because this cost function is well-defined and largely independent of assumptions regarding the number and structure of clusters in the original sample space, this partitioning scheme offers significant advantages over many conventional methods. Sample code is available.

cs.NE↗

arXiv2007

An Algorithm for Detecting Quantum-Gravity Photon Dispersion in Gamma-Ray Bursts: DISCAN

DisCan is a new algorithm implementing photon dispersion cancellation in order to measure energy-dependent delays in variable sources. This method finds the amount of reversed dispersion that optimally cancels any actual dispersion present. It applies to any time- and energy-tagged photon data, and can avoid binning in both time and energy. The primary motivation here is the search for quantum gravity based dispersion in future gamma ray burst data from the Gamma Ray Large Area Space Telescope (GLAST). Extrapolation of what is know about bursts at lower energies yields a reasonable prospect that photon dispersion effects consistent with some quantum gravity formalisms may be detected in sufficiently bright bursts. Short bursts have no or very small inherent lags, and are therefore better prospects than long ones, but even they suffer systematic error due to pulse asymmetry that may yield an irreducible uncertainty. We note that data at energies higher than about 0.1 TeV may not be useful for detecting dispersion in GRBs. Of several variants of the proposed algorithm, one based on Shannon information is consistently somewhat superior to all of the others we investigated.

astro-ph↗

arXiv2004

An Algorithm for Optimal Partitioning of Data on an Interval

Many signal processing problems can be solved by maximizing the fitness of a segmented model over all possible partitions of the data interval. This letter describes a simple but powerful algorithm that searches the exponentially large space of partitions of $N$ data points in time $O(N^2)$. The algorithm is guaranteed to find the exact global optimum, automatically determines the model order (the number of segments), has a convenient real-time mode, can be extended to higher dimensional data spaces, and solves a surprising variety of problems in signal detection and characterization, density estimation, cluster analysis and classification.

math.NA↗

arXiv2003

Long-term X-ray Variability of Circinus X-1

We present an analysis of long term X-ray monitoring observations of Circinus X-1 (Cir X-1) made with four different instruments: Vela 5B, Ariel V ASM, Ginga ASM, and RXTE ASM, over the course of more than 30 years. We use Lomb-Scargle periodograms to search for the ~16.5 day orbital period of Cir X-1 in each of these data sets and from this derive a new orbital ephemeris based solely on X-ray measurements, which we compare to the previous ephemerides obtained from radio observations. We also use the Phase Dispersion Minimization (PDM) technique, as well as FFT analysis, to verify the periods obtained from periodograms. we obtain dynamic periodograms (both Lomb-Scargle and PDM) of Cir X-1 during the RXTE era, showing the period evolution of Cir X-1, and also displaying some unexplained discrete jumps in the location of the peak power.

astro-ph↗

arXiv2002

Observation of X-ray variability in the BL Lac object 1ES1959+65

This paper reports X-ray spectral observations of a relatively nearby z = 0.048 BL Lacertae (BL Lac) object 1ES1959+650, which is a potential TeV emitter. The observations include 31 short pointings made by the Unconventional Stellar Aspect (USA) Experiment on board the Advanced Research and Global Observation Satellite (ARGOS), and 17 pointings by the PCA on board the Rossi X-ray Timing Explorer (RXTE). Most of these observations were spaced by less than 1 day. \es was detected by the ARGOS USA detector in the range 1-16 keV, and by the PCA in the 2-16 keV range but at different times. During the closely spaced RXTE observations beginning on 2000 July 28, an ending of one flare and a start of another are visible, associated with spectral changes, where the photon index Gamma ranges between ~ 1.4 and 1.7, and the spectrum is harder when the source is brighter. This implies that 1ES1959 is an XBL-type blazar, with the X-ray emission likely to originate via the synchrotron process. The USA observations reveal another flare that peaked on 2000 November 14 and doubled the flux within a few days, again associated with spectral changes of the same form. The spectral variability correlated with the flux and timing characteristics of this object that are similar to those of other nearby BL Lacs, and suggest relativistic beaming with a Doppler factor delta > 1.6 and magnetic fields of the order of a few mG. We also suggest that the steady component of the X-ray emission -- present in this object as well as in other XBLs -- may be due to the large-scale relativistic jet (such as measured by Chandra in many radio-loud AGN), but pointing very closely to our line of sight.

astro-ph↗

arXiv2001

Bayesian Estimation of Time Series Lags and Structure

This paper derives practical algorithms, based on Bayesian inference methods, for several data analysis problems common in time series analysis of astronomical and other data. One problem is the determination of the lag between two time series, for which the cross-correlation function is a sufficient statistic. The second problem is the estimation of structure in a time series of measurements which are a weighted integral over a finite range of the independent variable.

math.NA↗

arXiv2001

Bayesian Blocks in Two or More Dimensions: Image Segmentation and Cluster Analysis

This paper describes an extension, to higher dimensions, of the Bayesian Blocks algorithm for estimating signals in noisy time series data (Scargle 1998, 2000). The mathematical problem is to find the partition of the data space with the maximum posterior probability for a model consisting of a homogeneous Poisson process for each partition element. For model M_{n}, attributing the data within region n of the data space to a Poisson process with a fixed event rate lambda_{n}, the global posterior is: P(M_{n}) = Phi(N,V) = Gamma(N+1)Gamma(V-N+1) / Gamma(V+2) = N!(V-N)!/(V+1)! . Note that lambda_{n} does not appear, since it has been marginalized, using a flat, improper prior. Other priors yield similar formulas. This expression is valid for a data space of any dimension. It depends on only N, the number of data points within the region, and V, the volume of the region. No information about the actual locations of the points enters this expression. Suppose two such regions, described by N_{1},V_{1} and N_{2},V_{2}, are candidates for being merged into one. From the above equation, construct a Bayes merge factor, giving the ratio of posteriors for the two regions merged and not merged, respectively: P(Merge) = Phi(N_{1}+N_{2},V_{1}+V_{2}) / Phi(N_{1},V_{1}) Phi(N_{2},V_{2}) . Then collect data points into blocks with a greedy cell coalescence algorithm.

math.NA↗

arXiv2001

USA Observation of Spectral and Timing Evolution During the 2000 Outburst of XTE J1550-564

We report on timing and spectral observations of the 2000 outburst of XTE J1550-564 made by the Unconventional Stellar Aspect (USA) Experiment on board the Advanced Research and Global Observation Satellite (ARGOS). We observe a low-frequency quasi-periodic oscillation (LFQPO) with a centroid frequency that tends to increase with increasing flux and a fractional rms amplitude which is correlated with the hardness ratio. The evolution of the hardness ratio (4--16 keV/1--4 keV) with time and source flux is examined. The hardness-intensity diagram (HID) shows a cyclical movement in the clockwise direction and possibly indicates the presence of two independent accretion flows. We observe a relationship between the USA 4--16 keV count rate and radio observations and discuss this in the context of previously observed correlations between X-ray, radio, optical and IR data. We examine our results in the context of models invoking two accretion flows: a thin disk and a hot sub-Keplerian flow.

astro-ph↗

arXiv2001

Pulses, Spectral Lags, Durations, and Hardness Ratios in Long GRBs

We analyze BATSE 64-ms data for long gamma-ray bursts (T90 > 2.6 s), exploring the relationships between spectral lag, duration, and the number of distinct pulses per burst, Npulses. We measure durations using a brightness-independent technique. Within a similarly brightness-independent framework, we use a "Bayesian Block" method to find significant valleys and peaks, and thereby identify distinct pulses in bursts. Our results show that, across large dynamic ranges in peak flux and the Npulses measure, bursts have short lags and narrow pulses, while bursts with long lags tend to have just a few significant, wide pulses. There is a tendency for harder bursts to have few pulses, and these hardest bursts have short lags. Spectral lag and duration appear to be nearly independent - even for those bursts with relatively long lags: wider pulses tend to make up in duration for fewer pulses. Our brightness-independent analysis adds to the nascent picture of an intimate connection between pulse width, spectral lag, and peak luminosity for bursts with known redshifts: We infer that lower-luminosity bursts should have fewer episodes of organized emission - wider pulses with longer spectral lags.

astro-ph↗

arXiv2001

Time-Evolution of the Power Spectrum of the Black Hole X-ray Nova XTE J1550-564

We have studied the time evolution of the power spectrum of XTE J1550-564, using X-ray luminosity time series data obtained by the Rossi X-Ray Timing Explorer satellite. A number of important practical fundamental issues arise in the analysis of these data, including dealing with time-tagged event data, removal of noise from a highly non-stationary signal, and comparison of different time-frequency distributions. We present two new methods to understand the time frequency variations, and compare them to the dynamic power spectrum of Homan et al. All of the approaches provide evidence that the QPO frequency varies in a systematic way during the time evolution of the signal.

astro-ph↗

arXiv2000

Histogram Analysis of Gallex, GNO and Sage Neutrino Data: Further Evidence for Variability of the Solar Neutrino Flux

If the solar neutrino flux were constant, as is widely assumed, the histogram of flux measurements would be unimodal. On the other hand, sinusoidal or square-wave modulation (either periodic or stochastic) may lead to a bimodal histogram. We here present evidence that the neutrino flux histogram is in fact bimodal. We analyze all available data from gallium experiments, coordinating results from the GALLEX and GNO experiments into one data set, and adopting results from the SAGE experiment as another data set. The two histograms, from the two data sets, are consistent in showing peaks in the range 45-75 SNU and 90-120 SNU, with a valley in between. By combining the data into one data set, we may form more detailed histograms; these strengthen the case that the flux is bimodal. A preliminary statistical analysis indicates that the bimodal character of the solar neutrino flux is highly significant. Since the upper peak is close to the expected flux (120-140 SNU), we may infer that the neutrino deficit is due to time-varying attenuation of the flux produced in the core. We estimate the time scale of this variation to be in the range 10-60 days. Attenuation that varies on such a time scale is suggestive of the influence of solar rotation, and points towards a process involving the solar magnetic field in conjunction with a nonzero neutrino magnetic moment.

astro-ph↗

arXiv2000

Bayesian Blocks: Divide and Conquer, MCMC, and Cell Coalescence Approaches

Identification of local structure in intensive data -- such as time series, images, and higher dimensional processes -- is an important problem in astronomy. Since the data are typically generated by an inhomogeneous Poisson process, an appropriate model is one that partitions the data space into cells, each of which is described by a homogeneous (constant event rate) Poisson process. It is key that the sizes and locations of the cells are determined by the data, and are not predefined or even constrained to be evenly spaced. For one-dimensional time series, the method amounts to Bayesian changepoint detection. Three approaches to solving the multiple changepoint problem are sketched, based on: (1) divide and conquer with single changepoints, (2) maximum posterior for the number of changepoints, and (3) cell coalescence. The last method starts from the Voronoi tessellation of the data, and thus should easily generalize to spaces of higher dimension.

physics.data-an↗

arXiv1999

Publication Bias (The "File-Drawer Problem") in Scientific Inference

Publication bias arises whenever the probability that a study is published depends on the statistical significance of its results. This bias, often called the file-drawer effect since the unpublished results are imagined to be tucked away in researchers' file cabinets, is potentially a severe impediment to combining the statistical results of studies collected from the literature. With almost any reasonable quantitative model for publication bias, only a small number of studies lost in the file-drawer will produce a significant bias. This result contradicts the well known Fail Safe File Drawer (FSFD) method for setting limits on the potential harm of publication bias, widely used in social, medical and psychic research. This method incorrectly treats the file drawer as unbiased, and almost always misestimates the seriousness of publication bias. A large body of not only psychic research, but medical and social science studies, has mistakenly relied on this method to validate claimed discoveries. Statistical combination can be trusted only if it is known with certainty that all studies that have been carried out are included. Such certainty is virtually impossible to achieve in literature surveys.

physics.data-an↗

arXiv1998

Bayesian Blocks, A New Method to Analyze Structure in Photon Counting Data

I describe a new time-domain algorithm for detecting localized structures (bursts), revealing pulse shapes, and generally characterizing intensity variations. The input is raw counting data, in any of three forms: time-tagged photon events (TTE), binned counts, or time-to-spill (TTS) data. The output is the most likely segmentation of the observation into time intervals during which the photon arrival rate is perceptibly constant -- i.e. has a fixed intensity without statistically significant variations. Since the analysis is based on Bayesian statistics, I call the resulting structures Bayesian Blocks. Unlike most, this method does not stipulate time bins -- instead the data themselves determine a piecewise constant representation. Therefore the analysis procedure itself does not impose a lower limit to the time scale on which variability can be detected. Locations, amplitudes, and rise and decay times of pulses within a time series can be estimated, independent of any pulse-shape model -- but only if they do not overlap too much, as deconvolution is not incorporated. The Bayesian Blocks method is demonstrated by analyzing pulse structure in BATSE $γ$-ray data. The MatLab scripts and sample data can be found on the WWW at: http://george.arc.nasa.gov/~scargle/papers.html

astro-ph↗