All News

The latest from India's startup and technology ecosystem

New Research Tackles Tokenization Inefficiencies for Indic Languages, Cutting LLM Costs
TECHNOLOGY

New Research Tackles Tokenization Inefficiencies for Indic Languages, Cutting LLM Costs

New research from Aarjan Chaudhary, Binayak Mukherjeea, Moitreya Nandi, and Arindam Panja addresses tokenization inefficiencies for Indic languages. A regex fix cuts Nepali tokens in Llama-3/Qwen3 by 54%, while AdaptTok-Indic lowers token fertility for Bengali and Hindi, improving LLM performance and reducing inference costs.

Older →