Post-Hoc Vocabulary Extension for Low-Resource Languages: Gains Beyond Fine-Tuning, a Pre-Tokenizer Limit, and Effects on Degenerate Generation

Source abstract

Open-weight language models segment many low-resource languages into three to ten tokens per word, which raises inference cost and latency and leaves less of the context window for content. Extending a pretrained model’s vocabulary after training is a known remedy, but recent reports of it do not separate the effect of the new vocabulary from the effect of the additional training it requires. We study post-hoc extension of Qwen2.5-1.5B for six low-resource and under-served languages in four scripts: Urdu, Sindhi, Pashto, Amharic, Hindi and Tamil. Each extended model is compared with a model fine-tuned under the same budget without new vocabulary. For Hindi and Tamil, extension alone gives little: the inherited pre-tokenizer splits words at every vowel sign, and plain extension already reaches 95 to 99 percent of the compression it allows. A script-scoped repair of the pre-tokenizer, applied to the pretrained model, removes this limit. With 2,000 new tokens, and with the repair for Hindi and Tamil, sequences become 45 to 72 percent shorter and generating the same text is 1.75 to 3.64 times faster. At equal compute, the extended models have lower held-out bits per character than fine-tuning alone and translate into English better on FLORES-200, in all six languages. Extension also reduces degenerate repetition when the model writes these languages. A vocabulary-size sweep, a length-matched analysis, forced fragmentation of English output, and an ablation that freezes the new embeddings all point to fragmentation as one cause of this repetition. We release the toolkit, tokenizers and adapters.