arXiv subjects

Amanda Luby

Publications and source records attributed to Amanda Luby.

5 recordsLinked to original sources

arXiv2025

A Variance Decomposition Approach to Inconclusives in Forensic Black Box Studies

In the US, `black box' studies are increasingly being used to estimate the error rate of forensic disciplines. A sample of forensic examiner participants are asked to evaluate a set of items whose source is known to the researchers but not to the participants. Participants are asked to make a source determination (typically an identification, exclusion, or some kind of inconclusive). We study inconclusives in two black box studies, one on fingerprints and one on bullets. Rather than treating all inconclusive responses as functionally correct (as is the practice in reported error rates in the two studies we address), irrelevant to reported error rates (as some would do), or treating them all as potential errors (as others would do), we propose that the overall pattern of inconclusives in a particular black box study can shed light on the proportion of inconclusives that are due to examiner variability. Raw item and examiner variances are computed, and compared with the results of a logistic regression model that takes account of which items were addressed by which examiner. The error rates reported in black box studies are substantially smaller than ``failure rate" analyses that take inconclusives into account. The magnitude of this difference is highly dependent on the particular study at hand.

stat.AP↗

arXiv2024

Methodological Problems in Every Black-Box Study of Forensic Firearm Comparisons

Reviews conducted by the National Academy of Sciences (2009) and the President's Council of Advisors on Science and Technology (2016) concluded that the field of forensic firearm comparisons has not been demonstrated to be scientifically valid. Scientific validity requires adequately designed studies of firearm examiner performance in terms of accuracy, repeatability, and reproducibility. Researchers have performed ``black-box'' studies with the goal of estimating these performance measures. As statisticians with expertise in experimental design, we conducted a literature search of such studies to date and then evaluated the design and statistical analysis methods used in each study. Our conclusion is that all studies in our literature search have methodological flaws that are so grave that they render the studies invalid, that is, incapable of establishing scientific validity of the field of firearms examination. Notably, error rates among firearms examiners, both collectively and individually, remain unknown. Therefore, statements about the common origin of bullets or cartridge cases that are based on examination of ``individual" characteristics do not have a scientific basis. We provide some recommendations for the design and analysis of future studies.

stat.AP↗

arXiv2022

Think-aloud interviews: A tool for exploring student statistical reasoning

Think-aloud interviews have been a valuable but underused tool in statistics education research. Think-alouds, in which students narrate their reasoning in real time while solving problems, differ in important ways from other types of cognitive interviews and related education research methods. Beyond the uses already found in the statistics literature -- mostly validating the wording of statistical concept inventory questions and studying student misconceptions -- we suggest other possible use cases for think-alouds and summarize best-practice guidelines for designing think-aloud interview studies. Using examples from our own experiences studying the local student body for our introductory statistics courses, we illustrate how research goals should inform study-design decisions and what kinds of insights think-alouds can provide. We hope that our overview of think-alouds encourages more statistics educators and researchers to begin using this method.

stat.OT↗

arXiv2021

A probabilistic formalization of contextual bias in forensic analysis: Evidence that examiner bias leads to systemic bias in the criminal justice system

Although researchers have found evidence contextual bias in forensic science, the discussion of contextual bias is currently qualitative. We formalize years of empirical research and extend this research by showing quantitatively how biases can be propagated throughout the legal system, all the way up to the final determination of guilt in a criminal trial. We provide a probabilistic framework for describing how information is updated in a forensic analysis setting by using the ratio form of Bayes' rule. We analyze results from empirical studies using our framework and use simulations to demonstrate how bias can be compounded where experiments do not exist. We find that even minor biases in the earlier stages of forensic analysis lead to large, compounded biases in the final determination of guilt in a criminal trial.

stat.AP↗

arXiv2019

Psychometric Analysis of Forensic Examiner Behavior

Forensic science often involves the comparison of crime-scene evidence to a known-source sample to determine if the evidence and the reference sample came from the same source. Even as forensic analysis tools become increasingly objective and automated, final source identifications are often left to individual examiners' interpretation of the evidence. Each source identification relies on judgements about the features and quality of the crime-scene evidence that may vary from one examiner to the next. The current approach to characterizing uncertainty in examiners' decision-making has largely centered around the calculation of error rates aggregated across examiners and identification tasks, without taking into account these variations in behavior. We propose a new approach using IRT and IRT-like models to account for differences among examiners and additionally account for the varying difficulty among source identification tasks. In particular, we survey some recent advances (Luby, 2019a) in the application of Bayesian psychometric models, including simple Rasch models as well as more elaborate decision tree models, to fingerprint examiner behavior.

stat.AP↗