Tamil AI Infrastructure Bolstered by New Tokenizer, End-of-Turn Detector, and Multi-Script Font System

A new grapheme-aware tokenizer for Tamil, a dataset and model for detecting speech turn boundaries in Tamil telephone calls, and a system for creating OpenType fonts across nine Indic scripts have recently been presented in research papers. These efforts address core infrastructure challenges in building artificial intelligence systems for Indian languages, from how text is processed by models to how it is displayed, and how spoken interactions are managed.

Tamil Tokenizer Addresses Grapheme Challenges

Tokenization, the process of breaking text into discrete units for language models, is a fundamental step in natural language processing. While standard multilingual tokenizers like Byte Pair Encoding (BPE), WordPiece, and SentencePiece perform adequately for many languages, they often struggle with the morphological richness and unique script characteristics of Indic languages. For Tamil, specifically, its grapheme-based writing system can represent a single visible character with multiple Unicode code points, complicating efficient segmentation.

Hari Krishnan K and Sudarsun Santhiappan propose a new grapheme-aware Indic tokenizer for Tamil, first reported on September 6, 2026. Their approach preserves complete grapheme clusters by applying a reversible Unicode mapping strategy before WordPiece vocabulary learning. This method aims to improve vocabulary efficiency, reduce sequence length, lower computational costs, and ultimately enhance the performance of downstream models. This work aligns with ongoing efforts to optimize tokenization for Indic scripts, including the Devanagari tokenizer discussed in this publication on September 1, 2026, and earlier reporting in August 2026 on the “tokenizer tax” faced by Indian language AI due to inefficient tokenization.

TamilEOT: Detecting End-of-Turn in Speech

For voice agents, accurately determining when a user has finished speaking is critical. Relying on a fixed silence timeout often leads to agents either interrupting the user too soon or introducing frustrating delays. While open semantic end-of-turn detectors exist, none, to date, has specifically covered a South Indian language.

Santhoshkumar V addresses this gap with TamilEOT, a new dataset and model for semantic end-of-turn detection in Tamil telephone speech, submitted on September 4, 2026. The dataset comprises 18,485 labelled turn boundaries extracted from 116 real Tamil telephone conversations. Using this data, two audio-only detectors were fine-tuned from Smart Turn v3. On a held-out split of 4,168 clips from 30 unseen calls, the models significantly improved accuracy from a 70.30% zero-shot baseline to 83.71% for the 8.7 MB model and 86.13% for the 21 MB model. The ROC-AUC score also increased from 0.751 to 0.921. Both models operate within 150 milliseconds single-threaded on a laptop CPU. The paper notes that rule-derived labels, verified by human listeners, were accurate 95.9% of the time for positive turn boundaries, but only 44.4% for negative instances, indicating a challenge in distinguishing non-turn pauses. This work contributes to the broader field of Indian language speech processing, following previous discussions on benchmarks like Indic DiarBench in August 2026.

Srijika: OpenType Font Restyling for Nine Indic Scripts

Accurate and consistent rendering of text is a foundational requirement for any language technology. For Indic scripts, this is particularly complex due to the hundreds to thousands of conjuncts, half forms, and matra variants that must remain consistent under OpenType shaping rules. Building these fonts from scratch is a labor-intensive process.

Anil Pai presents Srijika, a system for producing installable OpenType fonts for nine Brahmic scripts, submitted on September 4, 2026. Rather than generating fonts from the ground up, Srijika restyles glyph outlines from existing, shaping-complete template fonts. It preserves the template’s `cmap` and `GSUB` closure, along with its `GPOS` data, under a defined metric policy. This ensures every output is a complete font. The system covers Devanagari, Tamil, Bengali, Telugu, Kannada, Malayalam, Gujarati, Gurmukhi, and Odia. Srijika produces 66 TrueType Fonts (TTFs), including 57 curated presets and nine open-vocabulary showcase fonts. All generated fonts pass the OpenType Sanitizer, and testing with HarfBuzz and CoreText reproduced the template glyph-ID sequences on conjunct-heavy probes. A full-closure audit of 80,915 glyphs and 54,812 anchors quantifies the metric changes. This system provides a more efficient method for creating visually consistent and functional fonts across a wide range of Indian scripts.

These three research efforts, while distinct in their specific focus areas, illustrate a consistent pattern of addressing fundamental infrastructure needs for Indian language AI. From optimizing how language models process Tamil text to enabling more natural spoken interactions and ensuring accurate visual representation of nine scripts, the work moves beyond theoretical advances to build practical components for wider deployment. The TamilEOT models, for example, demonstrate an accuracy improvement from 70.30% to 86.13% in identifying end-of-turn in speech, making voice agents more responsive.

Compiled by Launch91 Desk from the sources linked above. More about Launch91.