Navigating the "Lakh and Crore" Divide: New AI Benchmark Exposes Significant Errors in Indian Real Estate Mathematics

The dream of homeownership in India is often a complex journey through a labyrinth of specialized terminology, unique numbering systems, and intricate financial calculations. For many, like the representative figure "Ananya"—a 31-year-old professional who has spent nearly a decade meticulously saving for her first property—the transition from browsing brochures to signing a registration deed is fraught with numerical uncertainty. When a developer’s glossy pamphlet advertises a 1,050-square-foot two-bedroom apartment, only for the legal registration to reveal a "carpet area" of 720 square feet, the buyer is left to navigate a world of "loading percentages" and "super built-up" discrepancies.

In an era where artificial intelligence is increasingly positioned as the ultimate personal assistant, many prospective buyers are turning to chatbots to decode these figures. However, a new study and benchmarking exercise conducted for the Kaggle Benchmarking Challenge suggests that while modern Large Language Models (LLMs) are becoming more sophisticated, they still struggle with the cultural and mathematical nuances of the Indian real estate market. The exercise, titled "Lakh, Crore and Carpet Area," reveals that even the most advanced AI models can falter when faced with the specific financial logic required by the Indian homebuyer.

The Foundation of the Benchmark

The developer of this new benchmark sought to quantify exactly how reliable these models are when tasked with "deterministic" questions—math problems with exactly one correct answer. The benchmark consists of 60 targeted questions designed to mirror the actual mental and digital arithmetic performed by Indian buyers. These questions were not merely theoretical; they were calculated using Python scripts to ensure absolute accuracy before being presented to the AI models.

The benchmark was divided into four distinct categories, each testing a different pillar of the Indian property-buying experience. The first category, "Conversion," focused on the fluid movement between lakhs, crores, and Lakhs Per Annum (LPA). In a market where a price might be quoted as ₹1.2 crore but a discount is offered in lakhs, the ability to subtract ₹45 lakh from ₹1.2 crore and return a result in lakhs is a fundamental requirement.

The second category addressed "Area," a notorious pain point in Indian real estate. This section tested the models’ understanding of the relationship between carpet area, built-up area, and super built-up area, including the concept of "loading" and the traditional unit of "gaj." The third category, "EMI and Tax," delved into the rigorous world of reducing-balance Equated Monthly Installments (EMIs), stamp duty, Goods and Services Tax (GST), and Tax Deducted at Source (TDS). Finally, the "Format" category tested the models’ ability to adhere to the Indian numbering system, which uses a 2,2,3 grouping (e.g., 9,87,65,432) rather than the Western 3,3,3 grouping (e.g., 98,765,432).

Performance Metrics Across Leading Models

To provide a comprehensive view of the current AI landscape, the benchmark tested several high-profile models across different price tiers and providers. These included Gemini 3.7 Flash, Gemini 3.1 Pro Preview, Claude Sonnet 4.6, and GPT-5.4 mini. The evaluation process was strict: each model was required to provide a final answer in a specific format, and numerical results were checked against the true values. For EMI calculations, a tiny margin of error (±₹2) was permitted to account for different rounding methods, while the formatting tests required an exact string match.

The results revealed a surprising hierarchy of competence. Gemini 3.7 Flash emerged as the top performer, achieving a perfect score across all 60 cases. It was followed closely by Gemini 3.1 Pro Preview, which scored 0.98, missing only a single question in the EMI and tax category. Claude Sonnet 4.6 secured a score of 0.95, showing minor slips in conversion and EMI math. GPT-5.4 mini, while performing admirably in area and formatting, struggled significantly with financial calculations, ending with an overall score of 0.92.

One of the most intriguing metrics tracked was "western comma drift." This phenomenon occurs when a model, despite being asked an Indian-specific question, reverts to Western numerical formatting in its internal logic or final output. For instance, a model might correctly calculate a value but write it as 7,200,000 instead of 72,00,000. Gemini 3.7 Flash exhibited the least drift at 5%, while Claude Sonnet 4.6 showed the most, drifting in 12% of its responses. While this might seem like a cosmetic issue, in the high-stakes environment of property investment, such a slip can lead to a buyer misreading ₹40 lakh as ₹4 crore during a late-night review of documents.

The EMI Trap: Where Arithmetic Fails

The most significant finding of the benchmark was the models’ collective struggle with compound interest. While the "Area" category—initially expected to be the most difficult due to the confusing nature of "loading" percentages—was aced by every model, the "EMI and Tax" category became a "bloodbath" for several participants.

The failure of the models in this area is particularly noteworthy because EMIs are calculated using a fixed formula that a standard calculator handles with ease. However, LLMs do not use a calculator by default; they perform "mental arithmetic" through token prediction. When faced with a ₹72,00,000 loan at an 8.4% interest rate for 25 years, the models frequently hallucinated specific but incorrect numbers. For example, while the correct monthly EMI was ₹57,492, Claude Sonnet 4.6 suggested ₹57,487, and Gemini 3.1 Pro Preview suggested ₹57,482.

GPT-5.4 mini demonstrated the most profound weakness here, missing five out of the 15 EMI-related questions. The danger, as noted by the benchmark’s creator, is that the models do not express uncertainty. They provide a clean, confident, yet wrong number. Over a 300-month loan period, even a small discrepancy of ₹10 per month can lead to significant errors in long-term financial planning and loan comparisons. This has led to a fundamental shift in how developers are advised to integrate LLMs into financial tools: the model should explain the formula, but a dedicated computational tool must handle the actual math.

Cultural Nuance and the "Right Value, Wrong Unit" Problem

The benchmark also highlighted the difficulty AI models have with adhering to specific unit requests. In one instance, when asked to calculate a 20% down payment on a ₹95 lakh flat and provide the answer in rupees, Claude Sonnet 4.6 responded with "₹19 lakh." While the mathematical logic was sound, the model failed the instruction to provide the answer in the specific unit of rupees.

This highlights a broader challenge in prompt engineering and model training. Real-world users often require data in a specific format to paste into tax forms, loan applications, or spreadsheets. A model that understands the math but ignores the formatting constraints fails to be a truly useful assistant in a professional or administrative context.

A Lesson in Benchmarking Integrity

Perhaps the most valuable lesson derived from this exercise was not about the models themselves, but about the methodology of benchmarking. During the initial testing phase, early data suggested that Claude Opus 5 had failed nearly every question, while GPT-5.5 had errored out entirely.

Upon closer inspection of the run logs, the researcher discovered that the failures were not due to the models’ lack of intelligence, but rather due to API quota limits. Because the cases were fired in parallel, the more expensive, high-resource models quickly exceeded their available quotas. The scoring script had initially counted these "Permission Denied" errors as wrong answers.

This discovery led to a total overhaul of the benchmarking process, moving to sequential calls with robust retry logic. It served as a stark reminder to the developer community: a benchmark score is only as honest as its error handling. If a leaderboard cannot distinguish between a model’s incorrect guess and a system-level failure, the results are essentially meaningless. Consequently, the most expensive models were excluded from the final public tally to ensure that the published numbers were beyond reproach.

The Path Forward for Indian Real Estate AI

As AI continues to permeate the Indian real estate sector, the "Lakh, Crore and Carpet Area" benchmark provides a roadmap for future improvements. The next phase of testing is expected to delve into even more localized complexities, such as the nuances of GST on under-construction versus ready-to-move-in properties, the calculation of Pradhan Mantri Awas Yojana (PMAY) subsidies, and the variations in stamp duty across different Indian states.

For developers and homebuyers alike, the message is clear: while AI is a powerful tool for summarizing brochures and explaining general concepts, it is not yet a replacement for a financial calculator or a legal expert. As long as "western comma drift" and EMI hallucinations persist, the human element—and a good Python script—remains an essential part of the home-buying journey in India. The benchmark is now open for public contribution, inviting others to fork the code, add city-specific tax laws, and continue testing the limits of these digital assistants.