arXiv subjects

Martin Modrák

Publications and source records attributed to Martin Modrák.

5 recordsLinked to original sources

arXiv2026

Multiverse analysis, abdication of responsibility and manufacturing of doubt

I argue that multiverse analysis is highly suited to two undesirable uses: abdication of researcher's responsibility for their conclusion and manufacturing of doubt. A review of multiverse analyses published in 2025 provides tentative empirical support that abdication of responsibility is present in the literature and I mention anecdotal evidence that multiverse has been used for manufacturing of doubt about Covid-19 precautions. To mitigate negative effects if multiverse analysis becomes widely used I suggest the community adopts two conventions for evaluating multiverse analyzes: evaluating multiverses by the single worst universe they contain and considering large size of a multiverse as a sign of weakness rather than a praiseworthy achievement.

stat.ME↗

arXiv2025

Simulation-based validation of Bayes factor computation

We propose and evaluate two methods that validate the computation of Bayes factors: one based on an improved variant of simulation-based calibration checking (SBC) and one based on calibration metrics for binary predictions. We show that in theory, binary prediction calibration is equivalent to a special case of SBC, but with limited resources, binary prediction calibration is typically more sensitive to the problems we investigated. With well-designed test quantities, SBC can however detect all possible problems in computation, including some that cannot be uncovered by binary prediction calibration. Previous work on Bayes factor validation includes checks based on the data-averaged posterior and the Good check method. We demonstrate that both checks miss many problems in Bayes factor computation detectable with SBC and binary prediction calibration. Moreover, we find that the Good check as originally described fails to control its error rates. Our proposed checks also typically use simulation results more efficiently than data-averaged posterior checks. Finally, we show that a special approach based on posterior SBC is necessary when checking Bayes factor computation under improper priors and we validate several models with such priors. We recommend that novel methods for Bayes factor computation be validated with SBC, binary prediction calibration and data-averaged posterior with at least several hundred simulations. For all the models we tested, the bridgesampling and BayesFactor R packages satisfy all available checks and thus are likely safe to use in standard scenarios.

stat.ME↗

arXiv2024

How accurate are Bayes factor-based null hypothesis tests? A simulation study

Bayes factor null hypothesis tests provide a viable alternative to frequentist measures of evidence quantification. Bayes factors for realistic data sets in areas like psychology cannot be calculated exactly and require numerical approximations to complex integrals. Crucially, the accuracy of these approximations, i.e., whether an approximate Bayes factor corresponds to the exact Bayes factor, is unknown, and may depend on data, prior, and likelihood. We have recently developed a novel statistical procedure, namely marginal simulation-based calibration (SBC) for Bayes factors, to test whether the computed Bayes factors for a given analysis are accurate. Here, we use marginal SBC for Bayes factors and calibration plots to test for some common cognitive designs, whether Bayes factors are calculated accurately. We use the bridgesampling/brms packages in R. We run analyses for three commonly used designs in psychology and psycholinguistics: (a) a design with random effects for subjects only, (b) a Latin square design with crossed random effects for subjects and items, but a single fixed-factor, and (c) a Latin square 2x2 design with crossed random effects for subjects and items. We find that Bayes factor estimates turn out accurate in cases when the bridgesampling algorithm does not issue a warning message, but can be biased and variable when a warning message is shown. These results support the use of brms/bridgesampling for null hypothesis Bayes factor tests in commonly used factorial designs. They also suggest that when a warning message is issued, Bayes factor results should not be trusted. The results show that it is practical to check whether Bayes factors are computed correctly.

stat.ME↗

arXiv2023

Simulation-Based Calibration Checking for Bayesian Computation: The Choice of Test Quantities Shapes Sensitivity

Simulation-based calibration checking (SBC) is a practical method to validate computationally-derived posterior distributions or their approximations. In this paper, we introduce a new variant of SBC to alleviate several known problems. Our variant allows the user to in principle detect any possible issue with the posterior, while previously reported implementations could never detect large classes of problems including when the posterior is equal to the prior. This is made possible by including additional data-dependent test quantities when running SBC. We argue and demonstrate that the joint likelihood of the data is an especially useful test quantity. Some other types of test quantities and their theoretical and practical benefits are also investigated. We provide theoretical analysis of SBC, thereby providing a more complete understanding of the underlying statistical mechanisms. We also bring attention to a relatively common mistake in the literature and clarify the difference between SBC and checks based on the data-averaged posterior. We support our recommendations with numerical case studies on a multivariate normal example and a case study in implementing an ordered simplex data type for use with Hamiltonian Monte Carlo. The SBC variant introduced in this paper is implemented in the $\mathtt{SBC}$ R package.

stat.ME↗

arXiv2020

Bayesian Workflow

The Bayesian approach to data analysis provides a powerful way to handle uncertainty in all observations, model parameters, and model structure using probability theory. Probabilistic programming languages make it easier to specify and fit Bayesian models, but this still leaves us with many options regarding constructing, evaluating, and using these models, along with many remaining challenges in computation. Using Bayesian inference to solve real-world problems requires not only statistical skills, subject matter knowledge, and programming, but also awareness of the decisions made in the process of data analysis. All of these aspects can be understood as part of a tangled workflow of applied Bayesian statistics. Beyond inference, the workflow also includes iterative model building, model checking, validation and troubleshooting of computational problems, model understanding, and model comparison. We review all these aspects of workflow in the context of several examples, keeping in mind that in practice we will be fitting many models for any given problem, even if only a subset of them will ultimately be relevant for our conclusions.

stat.ME↗