New Research Tackles Tokenization Inefficiencies for Indic Languages, Cutting LLM Costs

For Llama-3 and Qwen3, processing Nepali text can require 2.6 to 5.8 times as many tokens as the same English text. This inefficiency, driven by how these models pre-tokenize Devanagari script, is a persistent challenge for developers building AI for Indic languages. New research, submitted to arXiv on October 8, 2026, details both a specific fix for this issue in popular open LLMs and a broader, script-aware tokenization approach for several other Indic languages.

Addressing the Pre-Tokenizer Problem for Nepali in Llama-3 and Qwen3

Aarjan Chaudhary’s paper, “Merge-Based Vocabulary Extension Stalls at the Pre-Tokenizer: An English-Preserving Repair for Nepali, Tested in Continued Pretraining,” identifies a core problem within the pre-tokenization step of widely used large language models. Models like Llama-3 and Qwen3 use a regular expression that categorizes only Unicode letters as word class characters. Because Devanagari vowel signs are combining marks, this regex inadvertently splits Nepali words at every vowel sign.

The consequence is a substantial token overhead: processing Nepali text demands between 2.6 and 5.8 times more tokens than equivalent English text. Chaudhary notes that traditional merge-based vocabulary extension, often seen as the solution for language-specific tokenization issues, largely fails here. For Llama-3 and Qwen3, even adding new merges cannot bring this token premium below approximately 2.2. Ten already-released vocabulary extensions achieve results within 4% of this pre-token floor. The problem is the initial split, not the vocabulary.

The proposed solution is a small, targeted edit to the deployed regular expression. This edit is designed to keep every token ID of text without combining marks or Devanagari characters unchanged, ensuring no adverse effects on English or other non-Devanagari content. When combined with 32,000 new merges, this adjustment brings Nepali token usage to 1.01 times that of English, representing about 54% fewer Nepali tokens compared to standard extension methods.

In small-budget continued pretraining experiments using Llama-3.2-1B and Qwen3-0.6B, the original tokenizer configuration ultimately yielded the lowest held-out Nepali bits per byte under evaluation. This suggests that while a fixed pre-tokenizer helps efficiency, the full optimization of a model for a language like Nepali involves both the initial tokenization strategy and further pretraining. The research, available on arXiv, offers a concrete path to significantly reduce the computational burden for Nepali in foundational LLMs.

AdaptTok-Indic: A Script-Aware Tokenizer for Multiple Indic Languages

Separately, researchers Binayak Mukherjeea, Moitreya Nandi, and Arindam Panja have introduced AdaptTok-Indic, an Indic-specialized extension of adaptive tokenization. Their paper, “AdaptTok-Indic: Script-Aware Adaptive Tokenization for Indic Languages,” also submitted to arXiv on October 8, 2026, addresses the broader problem of multilingual tokenizers trained predominantly on English corpora. These tokenizers frequently over-segment Indic languages, leading to longer token sequences and inconsistent tokenization across different scripts.

AdaptTok-Indic combines several key features: Unicode-normalized preprocessing, explicit handling of Indic conjuncts, script-aware routing, and learned byte-level boundary prediction. These components are jointly trained with a small Transformer language model.

The researchers evaluated AdaptTok-Indic on Bengali and Hindi, with additional assessments on Gujarati, Tamil, and Telugu. The results demonstrate lower token fertility compared to SentencePiece BPE, a common Indic-oriented tokenizer baseline, and an independently implemented MAGNET-style baseline. Across five recorded seeds, AdaptTok-Indic achieved a mean fertility of 1.432 for Bengali, significantly lower than SentencePiece BPE’s 2.156. For Hindi, the mean fertility was 1.388, compared to SentencePiece BPE’s 2.085. Downstream scores averaged 57.69 on evaluation tasks.

This reduction in token fertility directly translates to more efficient processing. Fewer tokens mean lower computational cost for inference and potentially better performance due to less fragmented representations of words.

A Continued Focus on Foundational Efficiency

Both papers confirm a consistent theme in Indian AI research: granular optimization of foundational language processing components. Our previous coverage, including “Indic Language AI: Tokenizer Efficiency and Tamil-First Models Advance Research” from October 14, 2026, has shown this ongoing effort to improve how AI models handle the unique structures of Indic languages. These new contributions fix fundamental issues in how text is broken down into tokens. They move beyond merely adding new vocabulary words. This includes bespoke tokenizers like AdaptTok-Indic and fixes to pre-tokenizer regex for existing LLMs.

The specific numbers, such as a 54% reduction in Nepali tokens and significantly lower token fertility for Bengali and Hindi, offer tangible benefits for developers. This granular work, detailed by Aarjan Chaudhary, Binayak Mukherjeea, Moitreya Nandi, and Arindam Panja, directly translates into lower inference costs and more accessible AI for the subcontinent’s diverse linguistic base.

Compiled by Launch91 Desk from the sources linked above. More about Launch91.