In the rapidly advancing world of artificial intelligence, a hidden economic and technical disparity is emerging, one that places a significant "tax" on non-English speakers. Recent investigations into the architecture of leading large language models (LLMs) have revealed that the cost of processing text varies wildly depending on the language used. For speakers of Bengali, the penalty is particularly steep: using Meta’s Llama 3.1 to process a paragraph in Bengali requires 6.3 times more tokens than the exact same paragraph in English.
This discrepancy is not a result of the inherent complexity of the language or the laws of physics. Instead, it is a direct consequence of tokenizer design—the initial layer of an AI model that breaks human text into the numerical units the machine can understand. While some newer models, such as Google’s Gemma 3, have successfully narrowed this gap to a negligible margin, others continue to operate with tokenizers that treat common global languages as rare, expensive anomalies.
The implications of this "token tax" are far-reaching, affecting everything from the literal cost of API calls to the effective memory capacity of the models and the carbon footprint of global AI usage.
The Study That Exposed the Language-Energy Divide
The conversation around linguistic inequity in AI reached a fever pitch following the publication of a landmark study by researchers at the University of Michigan. Titled "The Language-Energy Divide: Measuring Energy Costs of Multilingual LLM Inference," the paper was presented at EMNLP 2026. The researchers utilized Michigan’s ML.Energy framework on the university’s Great Lakes computing cluster to measure the actual joules consumed during AI inference.
The team tested several prominent models, including Alibaba’s Qwen 3, Google’s Gemma 3, and Meta’s Llama 3.1. They used the Belebele dataset—a multilingual reading comprehension benchmark—to compare how much energy the models consumed when answering the same questions in 122 different languages.
The findings were staggering. The study revealed that these models use anywhere from 1.4 to 180 times more energy to process non-English text compared to equivalent English text. For widely spoken South Asian languages, the numbers were particularly concerning: Hindi required 7.9 times the energy of English, while Bengali reached a multiplier of 15.3x.
The primary mechanism identified by the researchers was token count. In the world of LLMs, every token represents a unit of compute. If a language is "shattered" into a high number of tokens by a model’s tokenizer, the model must perform more calculations to process that text, leading directly to higher energy consumption and increased latency.
Verifying the Arithmetic of Inequity
The claim that tokenization is the root cause of this energy disparity is one that can be verified outside of a high-performance computing lab. By examining the tokenizers of the models used in the Michigan study—Qwen 3, Gemma 3, and Llama 3.1—alongside OpenAI’s GPT-4o, the sheer scale of the disparity becomes clear through simple arithmetic.
In a controlled test, a single three-sentence paragraph was translated into eight languages: French, German, Spanish, Chinese, Japanese, Arabic, Hindi, and Bengali. By running these translations through the official tokenization files of each model, the "markup" for each language becomes visible.
When the paragraph was tokenized in English, the results were remarkably consistent across all models, hovering between 43 and 44 tokens. However, once the script shifted away from the Latin alphabet, the numbers diverged sharply. On Meta’s Llama 3.1, the Bengali version of the paragraph exploded to 278 tokens—a 6.32x multiplier over the English version. On Qwen 3, the same Bengali text cost 244 tokens (5.67x).
Even European languages, which share the Latin script, are not exempt from this tax, though their penalties are lower. French and German typically see a markup of 1.2x to 1.7x across most models. The real outliers remain the Indic scripts. Hindi, spoken by over 600 million people, faced a 4.72x multiplier on Qwen 3 and a 2.57x multiplier on Llama 3.1.
To ensure these results were not a fluke of a specific paragraph’s structure, a secondary check was performed using a simple, short sentence: "The library opens in eighteen months." The results were identical in shape. On Llama 3.1, the Bengali version required 45 tokens compared to just 8 for the English version—a 5.62x disparity.
The "Tax Bracket" of Bengali and Hindi
The data suggests that Bengali and Hindi speakers are effectively placed in a higher "tax bracket" by AI developers. The reason for this lies in the "vocabulary" of the tokenizer. A tokenizer is essentially a dictionary of fragments—words, syllables, or characters—that the model recognizes as single units. If a language is well-represented in the training data used to build the tokenizer, the model can assign single tokens to whole words or common phrases.
If a language is poorly represented, the tokenizer fails to recognize common patterns. Instead, it falls back to "byte-level" tokenization, where it must break a single character into multiple tokens. On Llama 3.1, Bengali text runs at approximately 0.463 tokens per byte. This means the model is spending nearly half a token for every single byte of text, a sign that the tokenizer is struggling to recognize the script as anything other than a collection of rare symbols. In contrast, English runs at a much more efficient 0.189 tokens per byte.
This is not a limitation of the Bengali language itself, but a design choice made during the training of the AI. When a model’s vocabulary is limited—such as Llama 3.1’s 128,000-token capacity—and that vocabulary is predominantly filled with English and European fragments, there is little room left for the scripts of the Global South.
Gemma 3: Proving the Tax is Optional
The most significant finding of the comparative test is that this linguistic disparity is entirely avoidable. Google’s Gemma 3 model serves as a proof of concept for a more equitable AI future.
In the same tests where Llama 3.1 showed a 6.3x markup for Bengali, Gemma 3 showed a markup of only 1.16x. For Hindi, Gemma 3’s multiplier was a mere 1.23x, compared to Qwen 3’s 4.72x. In fact, Gemma 3’s worst-performing language in the test was Arabic, and even then, the multiplier was only 1.41x.
The difference comes down to vocabulary allocation. Gemma 3 was built with a much larger vocabulary of 262,000 tokens, and crucially, that vocabulary was trained on a more balanced multilingual corpus. By dedicating more "space" in its internal dictionary to Indic scripts, Google ensured that Bengali and Hindi could be processed almost as efficiently as English.
This comparison proves that the "Bengali tax" is a training-data decision that ultimately shows up as a bill for the end user. It is a matter of priority, not technical impossibility.
The Hidden Costs: Accuracy and Energy
While token counts provide a measurable floor for the penalty non-English speakers face, they do not tell the whole story. The Michigan study found that energy costs (up to 15.3x for Bengali) were significantly higher than the raw token multipliers (maxing out at 6.3x).
This gap represents the "output side" of the problem. When an AI model is less "comfortable" in a language—meaning it has seen less of it during training—it doesn’t just take more tokens to read the input; it also struggles to generate the output. Models often become less accurate in high-cost languages, leading to answers that are longer, more repetitive, or heavily hedged with "AI-isms."
Furthermore, because these models generate text one token at a time, a Bengali response that requires six times as many tokens will take six times as long to generate and consume six times the computational power. This compounds the energy cost, leading to the massive 180x disparities observed in some of the more extreme cases in the Michigan study.
The Triple Penalty in Practice
For a developer or a business in a country like Bangladesh or India, these technical discrepancies manifest as a "triple penalty" that hampers the adoption of AI technology.
First, there is the literal cost. Most AI companies charge for their services on a per-million-token basis. Under this pricing model, a Bengali-speaking startup using a Llama-based API is effectively paying six times more for the same amount of information as an English-speaking competitor.
Second, there is the issue of the "context window." Every AI model has a limit on how much text it can "remember" at one time, known as its context window. For Llama 3.1, this window is 128,000 tokens. While this is enough to hold a massive amount of English text, the 6.3x multiplier means that for a Bengali speaker, that window effectively shrinks to just 20,000 tokens. The model’s "memory" is physically smaller for certain languages.
Third, the languages that are the most expensive to process are also the ones where the models are the least accurate. Users are paying more for a shorter memory and a worse result.
A Call for Balanced Tokenization
The solution to this divide is unglamorous but essential: AI developers must prioritize the training of tokenizers on balanced, multilingual corpora. As the success of Gemma 3 demonstrates, allocating vocabulary based on global speaker populations rather than training-data convenience can almost entirely erase the token tax.
As AI becomes the backbone of global infrastructure, the "receipt" for linguistic inequity is being printed in the tokenizer files of the world’s most popular models. Until more developers follow the lead of models like Gemma 3, hundreds of millions of people will continue to pay a premium for a technology that, by design, treats their native tongue as an afterthought. For now, the disparity remains a stark reminder that in the world of artificial intelligence, not all words are created equal.
