Mitigating LLM biases toward spurious social contexts using direct preference optimization

Abstract

LLMs are increasingly used for high-stakes decision-making, yet their sensitivity to spurious context can introduce harmful biases. This is a critical concern when models are deployed for tasks like evaluating teachers' instructional quality, where biased assessment can affect teachers' professional development and career. We investigate model robustness to spurious social contexts about teachers using the largest publicly available dataset of U.S. classroom transcripts (NCTE) paired with expert evaluation scores. Evaluating seven frontier and open-weight models across seven categories of spurious contexts -- including teacher experience, education level, demographic identity, and sycophancy-inducing framings -- we find that irrelevant contexts can shift model-generated ratings by up to 1.48 points on a 7-point scale. Prompt-based mitigations and popular post-training methods, such as Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), fall short of addressing the issue. We propose Debiasing-DPO, which combines contrastive reasoning-augmented DPO with SFT on expert labels to reduce the effect of spurious context while avoiding mode collapse. Applied to Llama and Qwen Instruct models of 3-8B parameters, Debiasing-DPO reduces bias by 84% and improves predictive accuracy by 52% on average across models. Our findings from the educational dataset highlight that stronger models may exhibit greater sensitivity despite higher accuracy, and Debiasing-DPO can improve both accuracy and robustness in prompt-based prediction tasks.