Custom Devanagari Tokenizer Improves Bilingual Model Performance, New Benchmarks Emerge for Indian Languages

A new 1.04 billion-parameter language model, Arkios, designed for bilingual English-Nepali text, employs a custom Devanagari-aware tokenizer to achieve competitive performance despite a smaller training footprint. Submitted by Sajal Regmi, Siddhartha Pudasaini, and Chetan Phakami Pun, the model was pretrained from scratch on 150 billion tokens of English-Nepali text using a bespoke C/CUDA training stack, according to a paper published on arXiv.

The authors report that Arkios exceeds three comparably sized open models—Pythia-1.4B, TinyLlama-1.1B, and OLMo-1B—on the ARC-Easy and ARC-Challenge benchmarks, despite having trained on an order of magnitude fewer tokens. This performance, the researchers note, likely stems from a close match between their educational-web-text pretraining data and ARC’s grade-school-science format, rather than indicating a general capability advantage. A key component of Arkios is its Devanagari-aware byte-level BPE tokenizer, built specifically for this project. This specialized tokenizer can reduce inference costs and improve processing efficiency for languages that use the Devanagari script, including Hindi, Marathi, and Nepali. The paper also highlights a limitation for small models in low-resource languages, noting that the standard multiple-choice-letter prompt format often results in chance-level performance on Nepali reading comprehension tasks.

Evaluating AI-Generated Indic Text Detection

The proliferation of large language models has increased the need for reliable AI-generated text detection, especially for languages beyond English. However, existing benchmarks for this task often neglect Indic languages and operate in idealized settings that do not reflect real-world usage. To address this, Bhaskar Ganesh Devalla, Junchao Wu, Nilesh Dokuparthi, Greeshma Yaluru, Tatiana Muniz Rodriguez, Lidia S. Chao, and Derek F. Wong have introduced IndicDetect, a new benchmark for AI-generated text detection in Hindi, Telugu, and Tamil.

According to their paper, also on arXiv, IndicDetect is designed to evaluate the reliability of detectors under realistic distribution shifts, including domain shift, generator shift, and adversarial perturbation. The benchmark consists of human-written texts paired with LLM-generated counterparts across various domains and generators. This work builds on a growing trend of specialized evaluation tools for Indian languages, following earlier efforts like the “Indian LLM Benchmarks” and “SurakshaEval” published in this outlet, which focused on factual knowledge, prompt injection, and safety. IndicDetect now provides a needed framework for assessing the authenticity of text in key Indian languages, which is critical for combating misinformation and maintaining trust in digital content.

Cross-Lingual Transfer in Low-Resource Legal Comprehension

For low-resource languages lacking extensive training corpora, using a related, higher-resource language as a scaffold for comprehension is a common approach. Sindhu Shetty, Spurthi Setty, and Natan Vidra investigated this strategy for Tulu, a low-resource Dravidian language, in the legal domain. Their research, submitted to arXiv, tested three models — Llama3, Hex-1, and Sarvam — on their ability to classify legal complaints written in Tulu.

The authors found that transliterating queries across Dravidian scripts allowed models to gain a preliminary understanding of Tulu speakers’ complaints without extensive training. However, the level of comprehension was heavily dependent on the script used for transliteration. Kannada, another relatively low-resource language, produced the strongest positive trend in comprehension. The use of a Retrieval-Augmented Generation (RAG) framework, retrieving from a corpus of Kannada legal papers, yielded mixed results, with some models showing only a weak positive trend in comprehension. This indicates that while cross-lingual transfer via transliteration holds promise, its effectiveness is highly nuanced and specific to the chosen script and retrieval mechanism.

These papers collectively illustrate the foundational work underway to make large language models effective and reliable across India’s many languages. From tailored tokenizers that improve efficiency for specific scripts like Devanagari, to benchmarks for detecting AI-generated content in Hindi, Telugu, and Tamil, and studies into cross-lingual transfer for languages such as Tulu, the focus remains on addressing the realities of data scarcity and the need for specialized rather than generic AI solutions. The Tulu legal comprehension study specifically highlights that while transliteration can provide initial understanding, the choice of a related script, such as Kannada, significantly impacts the degree of that understanding.

Compiled by Launch91 Desk from the sources linked above. More about Launch91.