arXiv subjects

Andrew Caplin

Publications and source records attributed to Andrew Caplin.

4 recordsLinked to original sources

arXiv2026

Managing Cognitive Bias in Human Labeling Operations for Rare-Event AI: Evidence from a Field Experiment

Many operational AI systems depend on large-scale human annotation to detect rare but consequential events (e.g., fraud, defects, and medical abnormalities). When positives are rare, the prevalence effect induces systematic cognitive biases that inflate misses and can propagate through the AI lifecycle via biased training labels. We analyze prior experimental evidence and run a field experiment on DiagnosUs, a medical crowdsourcing platform, in which we hold the true prevalence in the unlabeled stream fixed (20% blasts) while varying (i) the prevalence of positives in the gold-standard feedback stream (20% vs. 50%) and (ii) the response interface (binary labels vs. elicited probabilities). We then post-process probabilistic labels using a linear-in-log-odds recalibration approach at the worker and crowd levels, and train convolutional neural networks on the resulting labels. Balanced feedback and probabilistic elicitation reduce rare-event misses, and pipeline-level recalibration substantially improves both classification performance and probabilistic calibration; these gains carry through to downstream CNN reliability out of sample.

cs.HC↗

arXiv2025

Misaligned by Design: Incentive Failures in Machine Learning

The cost of error in many high-stakes settings is asymmetric: misdiagnosing pneumonia when absent is an inconvenience, but failing to detect it when present can be life-threatening. Because of this, artificial intelligence (AI) models used to assist such decisions are frequently trained with asymmetric loss functions that incorporate human decision-makers' trade-offs between false positives and false negatives. In two focal applications, we show that this standard alignment practice can backfire. In both cases, it would be better to train the machine learning model with a loss function that ignores the human's objective and then adjust predictions ex post according to that objective. We rationalize this result using an economic model of incentive design with endogenous information acquisition. The key insight from our theoretical framework is that machine classifiers perform not one but two incentivized tasks: choosing how to classify and learning how to classify. We show that while the adjustments engineers use correctly incentivize choosing, they can simultaneously reduce the incentives to learn. Our formal treatment of the problem reveals that methods embraced for their intuitive appeal can in fact misalign human and machine objectives in predictable ways.

econ.GN↗

arXiv2025

Testing Capacity-Constrained Learning

We introduce the first general test of capacity-constrained learning models. Cognitive economic models of this type share the common feature that constraints on perception are exogenously fixed, as in the widely used fixed-capacity versions of rational inattention (Sims 2003) and efficient coding (Woodford 2012). We show that choice data are consistent with capacity-constrained learning if and only if they satisfy a No Improving (Action or Attention) Switches (NIS) condition. Based on existing experiments in which the incentives for being correct are varied, we find strong evidence that participants fail NIS for a wide range of standard perceptual tasks: identifying the proportion of ball colors, recognizing shapes, and counting the number of balls. However, we find that this is not true for all existing perceptual tasks in the literature, which offers insights into settings where we do or do not expect incentives to impact the extent of attention.

econ.GN↗

arXiv2022

Calibrating for Class Weights by Modeling Machine Learning

A much studied issue is the extent to which the confidence scores provided by machine learning algorithms are calibrated to ground truth probabilities. Our starting point is that calibration is seemingly incompatible with class weighting, a technique often employed when one class is less common (class imbalance) or with the hope of achieving some external objective (cost-sensitive learning). We provide a model-based explanation for this incompatibility and use our anthropomorphic model to generate a simple method of recovering likelihoods from an algorithm that is miscalibrated due to class weighting. We validate this approach in the binary pneumonia detection task of Rajpurkar, Irvin, Zhu, et al. (2017).

cs.LG↗