Distributed Peer Review at ALMA: An Empirical Comparison with Panel-Based Review
Large astronomical observatories are increasingly adopting distributed peer review (DPR) to manage growing proposal volumes, yet empirical comparisons with panel-based systems remain limited. Beginning in 2021 (Cycle 8), the Atacama Large Millimeter/submillimeter Array (ALMA) transitioned to DPR for the majority of proposals, with DPR applied to nearly all proposals from Cycle 9 onward. Under DPR, each Principal Investigator (PI) designates a reviewer to evaluate a set of proposals without collective discussion. This study analyzes over 20,000 proposals and 160,000 reviews spanning 13 cycles to assess the impact of this change. We find that DPR largely reproduces the systematic ranking trends observed in panel evaluations across PI demographics, technical characteristics, and scientific areas, consistent with panel outcomes both before and after discussion. Scientific diversity among the top-ranked proposals is similar between DPR and post-discussion panel rankings. Individual proposal ranks show substantial dispersion under both DPR and panel assessments prior to discussion, with discussion only partially reducing this variance. The observed dispersion therefore reflects intrinsic variation in reviewer judgments rather than a byproduct of the distributed process itself. In Cycle 12, reviewers rated the majority of DPR written comments as high or adequate quality, with no dependence on reviewer career stage. However, 10% of reviews were rated as low quality, highlighting the challenge of maintaining quality standards across the approximately 16,000 reviews produced each cycle. Overall, our results indicate that DPR reproduces the population-level ranking structure obtained in panel review, despite differences in review mechanics and the role of discussion.