November 22, 2026

The Token Tax: Why Large Language Models Charge Bengali Speakers Six Times More for the Same Words

Even European languages, which share the Latin script, are not exempt from this tax, though their penalties are lower. French and German typically see a markup of 1.2x to 1.7x across most models. The real outliers remain the Indic scripts. Hindi, spoken by over 600 million people, faced a 4.72x multiplier on Qwen 3 and a 2.57x multiplier on Llama 3.1.

To ensure these results were not a fluke of a specific paragraph’s structure, a secondary check was performed using a simple, short sentence: "The library opens in eighteen months." The results were identical in shape. On Llama 3.1, the Bengali version required 45 tokens compared to just 8 for the English version—a 5.62x disparity.

The "Tax Bracket" of Bengali and Hindi

The data suggests that Bengali and Hindi speakers are effectively placed in a higher "tax bracket" by AI developers. The reason for this lies in the "vocabulary" of the tokenizer. A tokenizer is essentially a dictionary of fragments—words, syllables, or characters—that the model recognizes as single units. If a language is well-represented in the training data used to build the tokenizer, the model can assign single tokens to whole words or common phrases.

If a language is poorly represented, the tokenizer fails to recognize common patterns. Instead, it falls back to "byte-level" tokenization, where it must break a single character into multiple tokens. On Llama 3.1, Bengali text runs at approximately 0.463 tokens per byte. This means the model is spending nearly half a token for every single byte of text, a sign that the tokenizer is struggling to recognize the script as anything other than a collection of rare symbols. In contrast, English runs at a much more efficient 0.189 tokens per byte.

This is not a limitation of the Bengali language itself, but a design choice made during the training of the AI. When a model’s vocabulary is limited—such as Llama 3.1’s 128,000-token capacity—and that vocabulary is predominantly filled with English and European fragments, there is little room left for the scripts of the Global South.

Gemma 3: Proving the Tax is Optional

The most significant finding of the comparative test is that this linguistic disparity is entirely avoidable. Google’s Gemma 3 model serves as a proof of concept for a more equitable AI future.

In the same tests where Llama 3.1 showed a 6.3x markup for Bengali, Gemma 3 showed a markup of only 1.16x. For Hindi, Gemma 3’s multiplier was a mere 1.23x, compared to Qwen 3’s 4.72x. In fact, Gemma 3’s worst-performing language in the test was Arabic, and even then, the multiplier was only 1.41x.

The difference comes down to vocabulary allocation. Gemma 3 was built with a much larger vocabulary of 262,000 tokens, and crucially, that vocabulary was trained on a more balanced multilingual corpus. By dedicating more "space" in its internal dictionary to Indic scripts, Google ensured that Bengali and Hindi could be processed almost as efficiently as English.

This comparison proves that the "Bengali tax" is a training-data decision that ultimately shows up as a bill for the end user. It is a matter of priority, not technical impossibility.

The Hidden Costs: Accuracy and Energy