Navigating the "Lakh and Crore" Divide: New AI Benchmark Exposes Significant Errors in Indian Real Estate Mathematics

The results revealed a surprising hierarchy of competence. Gemini 3.7 Flash emerged as the top performer, achieving a perfect score across all 60 cases. It was followed closely by Gemini 3.1 Pro Preview, which scored 0.98, missing only a single question in the EMI and tax category. Claude Sonnet 4.6 secured a score of 0.95, showing minor slips in conversion and EMI math. GPT-5.4 mini, while performing admirably in area and formatting, struggled significantly with financial calculations, ending with an overall score of 0.92.

One of the most intriguing metrics tracked was "western comma drift." This phenomenon occurs when a model, despite being asked an Indian-specific question, reverts to Western numerical formatting in its internal logic or final output. For instance, a model might correctly calculate a value but write it as 7,200,000 instead of 72,00,000. Gemini 3.7 Flash exhibited the least drift at 5%, while Claude Sonnet 4.6 showed the most, drifting in 12% of its responses. While this might seem like a cosmetic issue, in the high-stakes environment of property investment, such a slip can lead to a buyer misreading ₹40 lakh as ₹4 crore during a late-night review of documents.

The EMI Trap: Where Arithmetic Fails

The most significant finding of the benchmark was the models’ collective struggle with compound interest. While the "Area" category—initially expected to be the most difficult due to the confusing nature of "loading" percentages—was aced by every model, the "EMI and Tax" category became a "bloodbath" for several participants.

The failure of the models in this area is particularly noteworthy because EMIs are calculated using a fixed formula that a standard calculator handles with ease. However, LLMs do not use a calculator by default; they perform "mental arithmetic" through token prediction. When faced with a ₹72,00,000 loan at an 8.4% interest rate for 25 years, the models frequently hallucinated specific but incorrect numbers. For example, while the correct monthly EMI was ₹57,492, Claude Sonnet 4.6 suggested ₹57,487, and Gemini 3.1 Pro Preview suggested ₹57,482.

GPT-5.4 mini demonstrated the most profound weakness here, missing five out of the 15 EMI-related questions. The danger, as noted by the benchmark’s creator, is that the models do not express uncertainty. They provide a clean, confident, yet wrong number. Over a 300-month loan period, even a small discrepancy of ₹10 per month can lead to significant errors in long-term financial planning and loan comparisons. This has led to a fundamental shift in how developers are advised to integrate LLMs into financial tools: the model should explain the formula, but a dedicated computational tool must handle the actual math.

Cultural Nuance and the "Right Value, Wrong Unit" Problem

The benchmark also highlighted the difficulty AI models have with adhering to specific unit requests. In one instance, when asked to calculate a 20% down payment on a ₹95 lakh flat and provide the answer in rupees, Claude Sonnet 4.6 responded with "₹19 lakh." While the mathematical logic was sound, the model failed the instruction to provide the answer in the specific unit of rupees.