Andrew Nickorgous
Verified Author 343 published articles

Andrew Nickorgous

Contributing Writer at Launch91 covering India's startup and technology ecosystem.

Articles by Andrew Nickorgous

New Research Tackles Tokenization Inefficiencies for Indic Languages, Cutting LLM Costs
TECHNOLOGY

New Research Tackles Tokenization Inefficiencies for Indic Languages, Cutting LLM Costs

New research from Aarjan Chaudhary, Binayak Mukherjeea, Moitreya Nandi, and Arindam Panja addresses tokenization inefficiencies for Indic languages. A regex fix cuts Nepali tokens in Llama-3/Qwen3 by 54%, while AdaptTok-Indic lowers token fertility for Bengali and Hindi, improving LLM performance and reducing inference costs.