Using commercial language models for Indian languages can be significantly more expensive and slower than for English. New research on tokenizer efficiency quantifies this disparity. The findings, which include analyses of model performance, reinforcement learning complexities, and the need for tailored evaluation environments, detail specific challenges for AI development in India’s many languages.

The economic implications of language model use are directly tied to tokenization. Commercial APIs often bill per token, meaning an inefficient tokenizer makes a language more costly, increases inference time, and limits the amount of text that fits into a model’s context window. A study by Johin Johny Arimbur, Ankur Ashit Ghosh, and Shervin Sam, submitted on November 9, 2026, measured this “token tax” across nine public tokenizers, including those from GPT-4, GPT-4o, Qwen2.5, DeepSeek-V3, and Mistral-Nemo, as well as four Sarvam models. The researchers used 2,009 professionally translated sentences from FLORES-200, covering 25 Indian languages, and 13,732 human-translated Hinglish-English social media pairs from PHINC.

On GPT-4o’s tokenizer, the same content required between 1.57 times (Hindi) and 5.01 times (Odia) the tokens of English. For Santali in Ol Chiki script, this disparity grew to 13.7 times. GPT-4’s tokenizer showed similar inefficiencies, needing up to 12.8 times more tokens. The only third-party tokenizer that consistently stayed below 2.6 times the English token count across all languages, with a median of 1.53, was Sarvam-30B/105B. This research directly measures the “translation efficiency” and “script and code-switching challenges” that our earlier coverage noted for Indic language LLMs. The findings suggest that for cost-sensitive applications or those requiring large context windows in Indic languages, specialized tokenizers are necessary.

While large language models (LLMs) continue to gain prominence, their superiority for all tasks in Indic languages is not guaranteed. A comparative evaluation by Aishwarya Gunjal, Kanchan Khemnar, Pawan Poojary, Pranav Kulkarni, and Raviraj Joshi, submitted on November 14, 2026, examined the effectiveness of BERT-based encoders against LLM-based embeddings for Indic semantic textual similarity (STS). The team compared eight BERT-based and five LLM-based models on a 1,500-pair STS Benchmark development set translated into ten Indic languages, along with the native Marathi MahaSTS test set. IndicSBERT, a BERT-based model, achieved the highest Pearson correlation across all ten translated language sets, with a macro-average of 0.843, outperforming BGE-M3 at 0.792. Harrier-270M was the strongest LLM-based model in this setting, scoring a macro-average of 0.751. On the MahaSTS dataset, IndicSBERT and Bhasha-Embed-v0 both recorded strong Pearson correlations of 0.901 and 0.900 respectively, while EmbeddingGemma led the LLM-based models at 0.871. These results indicate that for tasks like semantic textual similarity in Indic languages, smaller, BERT-based architectures can still surpass larger, more general LLMs. This challenges the assumption that scale alone translates to superior performance across all linguistic contexts.

The process of fine-tuning models with reinforcement learning (RL) also presents specific challenges for multilingual models. Sutanshu Saksena Raj, Daksh Maan, Avinash Benki, Parijat Chatterjee, Pasunuti Prasanjith, and Purusharth Malik explored the gains and regressions in a supervised fine-tuned 30B-A3B mixture-of-experts model covering English and eight Indic languages after applying RL with GRPO. In their first evaluation campaign across 34 benchmarks, MATH performance improved by 36.07 points, and most Indic benchmarks saw gains. However, XQuAD-Indic F1 scores fell by 19.32 points, with English answers rising from 8.7% to 91.7%. IndicQA also saw an 8.64-point drop, even as English XQuAD gained 11.71 F1. This shows a common issue where RL optimization for one language or task can inadvertently degrade performance on others, particularly for less-represented languages. A second RL run, which incorporated an Indic environment along with language and repetition rewards, improved both passage-answering and instruction-following benchmarks for Indic languages, while preserving MATH performance. This second run, however, lost ground on reasoning and coding benchmarks. This demonstrates the complex trade-offs involved in balancing performance across diverse tasks and languages during RL.

Beyond model training, evaluating AI agents for real-world Indian workflows requires specialized environments. Daksh Bathla’s research, submitted on November 16, 2026, introduces Navital Gym, a set of six open environments with 321 public tasks in five task families: trip planning, UPI payments and disputes, GST invoicing and returns, form filling (including multi-document splits), and customer support. These environments specifically target workflows in Hindi, Hinglish, and three experimental Indian languages. A key innovation of Navital Gym is its use of named, deterministic rules to grade tasks by recomputing outcomes in a synthetic sandbox world, avoiding reliance on another language model for scoring. This approach directly addresses the “evaluation bias” previously noted for Indic language LLMs. The research reported one run per task for three commercial models: Claude Opus 5.5 solved 315 of 321 tasks, Claude Sonnet 5.5 solved 302, and Claude Haiku 4.5 solved 205. Gemini 3.1 Flash-Lite was also tested on two environments (UPI and Support core). This new benchmark provides a concrete way to measure agent performance on specific, high-stakes Indian applications, building on the need for specialized evaluation tools, similar to the Indic DiarBench for speech recognition that this publication reported on in August 2026.

The combined findings from these papers show that developing AI for India’s many languages requires targeted research and development, not just generic scaling of large models. Addressing fundamental inefficiencies in tokenization, navigating complex trade-offs in reinforcement learning, and building specific evaluation benchmarks all demand persistent, granular engineering efforts. The Navital Gym’s rule-graded environments, with 321 public tasks, provide a practical framework for assessing agent capabilities in critical Indian workflows.

Sources

Compiled by Launch91 Desk from the sources linked above. More about Launch91.