Skip to contentDEEKOYA AI CoursesFLAT 20% OFFEnroll Now
Deekoya AI

FIDEL: Abugida-Aware Pre-tokenization for Ethiopic Languages

Gebreslassie Teklu Reda, Yuval Pinter · 16 Nov 2026 · arXiv

Abstract

Subword tokenizers assume that meaningful linguistic units can be recovered by splitting text at learned boundaries, an assumption better suited to concatenative morphology than to Semitic root-and-pattern systems. The challenge is acute for the Ethiopic languages, whose abugida script encodes consonant-vowel units as single characters and hides consonantal structure. We introduce FIDEL (FIdäl DEcomposition and Linearization), a family of reversible abugida-aware pre-tokenization schemes that expose consonant and vowel-order structure, extend SPLINTER reductions from abjad scripts to the Ethiopic abugida, and evaluate them on three Ethiopic languages under controlled conditions, holding corpus, architecture, budget and protocol fixed so that tokenization is the only variable. We report intrinsic tokenizer statistics for BPE and UnigramLM across vocabulary sizes alongside downstream NLP tasks, including transfer scenarios. We find that FIDEL reduction improves news classification consistently and named-entity recognition at larger capacity, while sentiment analysis performance declines under the more aggressive schemes, and extractive question answering performance is preserved.

cs.CL

Comments

No comments yet. Share your own experience below.

Visitor posts are moderated before they appear and are the opinion of their authors, not DEEKOYA AI. Names are not verified. See our privacy policy.