Custom Devanagari Tokenizer Improves Bilingual Model Performance, New Benchmarks Emerge for Indian Languages
New research details Arkios, a 1.04B-parameter English-Nepali LLM with a Devanagari-aware tokenizer, and IndicDetect, a benchmark for AI-generated text detection in Hindi, Telugu, and Tamil. Another study explores cross-lingual transfer for Tulu legal comprehension.