arXiv · 2611.17836
BLiMP-RO: A Benchmark of Linguistic Minimal Pairs for Romanian
Abstract
Evaluating grammatical knowledge in language models requires benchmarks that cover diverse linguistic phenomena and are validated against human judgments. We introduce BLiMP-RO, a Romanian minimal-pair benchmark covering 50 grammatical phenomena across ten domains. It comprises 493 linguist-constructed sentence pairs, evaluated by native speakers in five experiments, and 5,942 additional semi-automatically generated, human-validated pairs. We further investigate grammatical generalization under limited linguistic input by training GPT-2 and GPT-BERT models from scratch on Romanian BabyBabelLM data and a newly collected corpus of 10 million words from child-targeted video captions. Increasing caption training data yields approximately 9 percentage points of accuracy improvement per tenfold increase in corpus size. At matched data sizes, BabyBabelLM generally outperforms video captions, while GPT-BERT consistently outperforms GPT-2. Our findings highlight how training data quantity, composition, and model architecture shape grammatical generalization in Romanian.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Radu-Stefan Popa, Valentin Feroiu, Adina Camelia Bleotu, Sergiu Nisioi. 2026-11-13. BLiMP-RO: A Benchmark of Linguistic Minimal Pairs for Romanian. https://searcharxiv.com/abs/2611.17836
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.