Skip to contentDEEKOYA AI CoursesFLAT 20% OFFEnroll Now
Deekoya AI

BLiMP-RO: A Benchmark of Linguistic Minimal Pairs for Romanian

Radu-Stefan Popa, Valentin Feroiu, Adina Camelia Bleotu, Sergiu Nisioi · 13 Nov 2026 · arXiv

Abstract

Evaluating grammatical knowledge in language models requires benchmarks that cover diverse linguistic phenomena and are validated against human judgments. We introduce BLiMP-RO, a Romanian minimal-pair benchmark covering 50 grammatical phenomena across ten domains. It comprises 493 linguist-constructed sentence pairs, evaluated by native speakers in five experiments, and 5,942 additional semi-automatically generated, human-validated pairs. We further investigate grammatical generalization under limited linguistic input by training GPT-2 and GPT-BERT models from scratch on Romanian BabyBabelLM data and a newly collected corpus of 10 million words from child-targeted video captions. Increasing caption training data yields approximately 9 percentage points of accuracy improvement per tenfold increase in corpus size. At matched data sizes, BabyBabelLM generally outperforms video captions, while GPT-BERT consistently outperforms GPT-2. Our findings highlight how training data quantity, composition, and model architecture shape grammatical generalization in Romanian.

cs.CL

Comments

No comments yet. Share your own experience below.

Visitor posts are moderated before they appear and are the opinion of their authors, not DEEKOYA AI. Names are not verified. See our privacy policy.