SPIN Unprocessed October 27, 2026 ai_technology research
Merge-Based Vocabulary Extension Stalls at the Pre-Tokenizer: An English-Preserving Repair for Nepali, Tested in Continued Pretraining
View original on arxiv.orgOverview
arXiv:2610.33016v1 Announce Type: new Abstract: Llama-3, Qwen3 and many other open LLMs pre-tokenize text with a regular expression whose word class admits only Unicode letters. Devanagari vowel signs are combining marks, so this regex splits Nepali words at every vowel sign, and models that use it spend 2.6 to 5.8 times as many tokens on Nepali as on the same English text. Merge-based vocabulary extension cannot join the split pieces: for Llama-3 and Qwen3 no number of new merges brings this pr
SpinGraph analysis pending — check back after processing.
Ask AI about this story
Opens with the SpinGraph .md URL and structured context — one click, prompt included.
More from arXiv Computation and Language
View all →- Each Insufficient in Its Own Way: Calibrating Language Models Across Varying Evidence States
- Position, Role or Label? A Controlled Study of Instruction Conflicts in LLM Agents
- Parametric Session States Recover Retrieval Intent without Replaying History
- EviGround-CT: Sentence-to-Volume Grounding for Verifying LLM-Generated Chest CT Reports
- A Benchmark for Controllable Speaking-Style Captioning in Audio-Language Models
- Conceptizer: Next-Token Prediction Is Not All You Need
Markdown (.md) · JSON-LD schema (.json) · Machine-readable for AI & GEO