Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis

Abstract

Multimodal large language models (MLLMs) struggle with alignment due to the limitations of existing reward models (RMs), which are predominantly vision-centric, dependent on costly human labels, and provide opaque scalar scores that fail to capture nuanced reasoning, leading to brittle alignment. We present Omni-RRM, an \textbf{Omni}-modal \textbf{R}ubric-grounded \textbf{R}eward \textbf{M}odel that generates multi-dimensional reward signals across text, image, video, and audio. To overcome the high cost and inherent inconsistency of human-centric evaluation in multi-dimensional reasoning, we introduce \textbf{Omni-Preference}, a high-quality dataset constructed via automatic rubric-grounded preference synthesis. In this pipeline, teacher models reconcile raw preferences into explicit justifications, ensuring that the synthesized supervision is both high-fidelity and interpretable. Omni-RRM is trained using a progressive SFT + GRPO regimen, specifically optimized to sharpen reward discrimination on low-margin, hard preference pairs. It achieves state-of-the-art accuracy on video (80.2\% on ShareGPT-Video) and audio benchmarks (66.8\% on Audio-HH-RLHF and 65.0\% on TA2T), yielding a five-benchmark Overall accuracy of 70.4\% and a +17.0\% relative gain over its backbone. Furthermore, Omni-RRM effectively guides Best-of-$N$ selection and exhibits robust transfer to text-only alignment. All resources, including the dataset, training and inference code, and model checkpoints are available at https://tmfk418.github.io/Omni-RRM.