arXiv · 2611.18616
The Judge Is Not Impartial: Self-Preference in Medical LLM Evaluation
Abstract
Large language models (LLMs) are increasingly used to evaluate model-generated medical content, creating a potential circularity: an LLM judge may favor outputs generated by the same model independently of clinical quality. To characterize LLM judge self-preference in medical settings, we measured same-model affinity in single-turn specialist question answering and multi-turn standardized-patient encounters. We evaluated eight models, two from OpenAI, Anthropic, Google, and Qwen, on 620 real point of care questions from clinicians (Real-POCQi). The same eight models also generated and judged 200 standardized patient-clinician scenarios (MedSP1000) up to 8 visible turns. In total, our analysis spans 13,760 LLM judgments. We define three complementary measures of self-preference. Answer Rank Bias, our primary measure, compared the rank assigned to a fixed response by its own-model judge with its average rank from outside judges. Answer Score Bias made the analogous comparison using rubric scores. Own-Answer Win Rate measured how often a judge ranked its own model's response above a competitor. Higher values across all these measures indicate greater degrees of self-preference. In Real-POCQi, pooled Answer Rank Bias was $+1.22$ positions, and all eight judges favored their own answers. Self-preference persisted both with and without a grading rubric. Revealing generator identities modestly weakened self-preference by an average of $-0.094$ positions. In MedSP1000, pooled Answer Rank Bias ranged from $+0.53$ to $+0.62$ positions, with no clear difference by conversation length ($p=0.17$). LLM judges systematically favored responses generated by their own models across multiple formats. Clinical AI deployments should exercise increased vigilance including multiple evaluation measures and judge panels spanning differing model families when deploying LLM-as-a-judge systems.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Zara Ansari, Shlok Natarajan, Vincent Yip, Cally Lin, Avi Udash, Mia Garvey, Aaron Fanous, Roxana Daneshjou. 2026-09-09. The Judge Is Not Impartial: Self-Preference in Medical LLM Evaluation. https://searcharxiv.com/abs/2611.18616
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.