Share the Judge, Learn the Deferral: Where Specialization Helps LLM Evaluation

Abstract

Agentic systems generate outputs faster than human review. We contrast two LLM evaluator specialization strategies: specialized judge weights, or rule-based deferral policies for safe judgment acceptance. On 99,952 rubric-conditioned samples, correct rubrics improve accuracy by 2.11 points, while incorrect rubrics reduce performance by 2.66 points. Splitting training data across eight criterion-specific LoRA experts lowers accuracy by 10.05 points and reduces 5% error-bound coverage from 24.44% to 5.43%. This loss is independent of model size and training settings, with most performance recoverable by warm-starting experts from a unified judge. Sweeping data budgets confirms scratch expert specialization yields no empirical gains. Warm-started splits appear competitive with unified models, yet under limited data, unified training outperforms split training, with specialization beneficial only after unified training plateaus. Results hold on HealthBench, where physician rubrics improve accuracy while flawed rubrics degrade performance. Unlike weight specialization, deferral policies enable efficient evaluation. On RewardBench 2, lightweight deferral heads form a 0.6B-4B-8B reward cascade with no core scoring modification. Across 20 splits, the cascade achieves 89.40% accuracy versus 84.75% for a standalone 8B judge at 41.5% compute, satisfying 95% risk constraints. Margin-based deferral matches accuracy at far higher compute cost. The design generalizes across models, improving Tulu-3-8B and Skywork-8B performance with a lightweight DeBERTa frontend. We derive simple, robust evaluator design rules: unify judgment training or warm-start split models, and use audited deferral cascades for low-cost, reliable LLM evaluation.