MR3: Multilingual rubric-agnostic reward reasoning models
A multilingual, rubric-agnostic reward reasoning model achieving the broadest language coverage in reward modeling to date.
Evaluation using Large Language Model (LLM) judges has been widely adopted in English and shown to be effective for automatic evaluation. However, their performance does not generalize well to non-English settings, and it remains unclear what constitutes effective multilingual training for such judges. In this paper, we introduce mR3, a massively multilingual, rubric-agnostic reward reasoning model trained on 72 languages, achieving the broadest language coverage in reward modeling to date. We present a comprehensive study of data and curriculum selection for training to identify effective strategies and data sources for building high-quality reward models, including the integration of target-language reasoning datasets. Our approach attains state-of-the-art performance on multilingual reward model benchmarks, surpassing much larger models (i.e., \textsc{GPT-OSS-120B}) while being up to nine times smaller, and its effectiveness is further confirmed through extensive ablation studies.
Latest publications
Alignment-weighted DPO
A DPO that targets the most problematic parts of an output by assigning different preference weights.
ICLRYour model diversity determines reasoning strategy
A framework decomposing reasoning uncertainty and deriving conditions where depth refinement outperforms parallel sampling. (ICLR)
ICLRGenARM: Reward guided generation with autoregressive reward model for Test-time Alignment
A test-time alignment approach that leverages the Autoregressive Reward Model.
ICLR