Indic AI Moves to Pragmatic Evaluation, Nepali Speech Gains Accuracy

The evaluation of large language models for Indian languages is expanding beyond basic comprehension and safety to include more nuanced aspects of communication, while efforts continue to improve speech recognition in low-resource settings. A new benchmark, VakyArth, has been introduced to assess pragmatic competence in Hindi, Punjabi, Tamil, and Malayalam. Concurrently, new research has demonstrated significant improvements in Nepali financial speech recognition through domain-adaptive fine-tuning.

VakyArth: Evaluating Pragmatic Reasoning in Indic LLMs

Real-world language use often relies on implicit meanings, cultural context, and implied communication, a domain known as pragmatics. Until now, most pragmatic evaluation benchmarks for large language models (LLMs) have focused on English and other high-resource languages. A paper submitted on September 1, 2026, by Usneek Singh, Poorvaja Veera Balaji Kumar, Parth Nanda, Anand Madhusoodanan, Geyang Guo, Wei Xu, and Junyi Jessy L, introduces VakyArth, the first pragmatic benchmark specifically for Indic languages.

VakyArth provides a diagnostic evaluation across Hindi, Punjabi, Tamil, and Malayalam. It assesses LLMs on five pragmatic phenomena: deixis, speech acts, implicature, social pragmatics, and coherence. The benchmark uses multiple-choice questions, natural language inference tasks, and translation challenges, with all test items authored by native speakers of the respective languages. Initial findings from the VakyArth evaluation show “consistent failures” in pragmatic reasoning across multilingual LLMs of various architectures and sizes. This indicates that while LLMs may perform well on literal understanding, they struggle with the subtleties of human communication in these Indic languages.

This new benchmark builds on earlier work in evaluating Indian language LLMs. Previous efforts, such as the benchmarks for factual knowledge and prompt injection reported on August 18, 2026, and SurakshaEval for safety published on August 13, 2026, focused on foundational aspects of model performance. VakyArth pushes the evaluation deeper, moving toward how well models can interact in culturally and contextually rich linguistic environments. This suggests a maturing of the evaluation criteria for Indic language AI, shifting from basic functional capabilities to more advanced human-like understanding.

SpeakPay: Domain-Adaptive Speech Recognition for Nepali Financial Transactions

Separately, new research addresses the challenge of low-resource speech recognition for practical applications. Mobile payment systems in Nepal are often graphically mediated, creating barriers for visually impaired users. A paper submitted on September 1, 2026, by Biraj Subedi, introduces SpeakPay, a voice-first digital wallet designed to improve accessibility.

The core technical contribution of SpeakPay is a controlled study on domain adaptation for low-resource financial speech recognition in Nepali. The research introduces NepFinSpeech-403, a new dataset comprising 403 utterances of Nepali financial voice commands, covering operations like “send,” “load,” and “balance,” and spanning 237 unique numerals. The authors fine-tuned Whisper large-v2 using LoRA (Low-Rank Adaptation) on this dataset.

The results show a substantial improvement in accuracy. On the held-out test set, the domain-adapted model reduced the Word Error Rate (WER) from 129.95% for the zero-shot baseline to 42.58%, representing a 67.2% relative reduction. More critically for the application, Devanagari numeral recognition accuracy, which was 0.0% without adaptation, improved to 73.9%. The paper notes that word-level metrics alone do not fully capture the practical impact. The domain adaptation improved the Transaction Success Rate from 1.67% to 33.33%, a nearly 20-fold gain. This significant improvement was consistent across individual operations.

This work highlights the persistent challenge of low-resource languages, a theme previously seen in efforts to address the “tokenizer tax” for Indic scripts, as reported on August 6, 2026. While that work focused on efficiency, SpeakPay focuses on accuracy and practical usability in a specific, high-stakes domain. The improvements show that even small, domain-specific datasets can dramatically enhance the performance of large pre-trained models for critical applications, especially when dealing with languages that lack extensive training data. The ability to correctly interpret Devanagari numerals, a specific and crucial task for financial transactions, demonstrates the value of this targeted approach.

Compiled by Launch91 Desk from the sources linked above. More about Launch91.