Navigating the "Lakh and Crore" Divide: New AI Benchmark Exposes Significant Errors in Indian Real Estate Mathematics

The dream of homeownership in India is often a complex journey through a labyrinth of specialized terminology, unique numbering systems, and intricate financial calculations. For many, like the representative figure "Ananya"—a 31-year-old professional who has spent nearly a decade meticulously saving for her first property—the transition from browsing brochures to signing a registration deed is fraught with numerical uncertainty. When a developer’s glossy pamphlet advertises a 1,050-square-foot two-bedroom apartment, only for the legal registration to reveal a "carpet area" of 720 square feet, the buyer is left to navigate a world of "loading percentages" and "super built-up" discrepancies.

In an era where artificial intelligence is increasingly positioned as the ultimate personal assistant, many prospective buyers are turning to chatbots to decode these figures. However, a new study and benchmarking exercise conducted for the Kaggle Benchmarking Challenge suggests that while modern Large Language Models (LLMs) are becoming more sophisticated, they still struggle with the cultural and mathematical nuances of the Indian real estate market. The exercise, titled "Lakh, Crore and Carpet Area," reveals that even the most advanced AI models can falter when faced with the specific financial logic required by the Indian homebuyer.

The Foundation of the Benchmark

The developer of this new benchmark sought to quantify exactly how reliable these models are when tasked with "deterministic" questions—math problems with exactly one correct answer. The benchmark consists of 60 targeted questions designed to mirror the actual mental and digital arithmetic performed by Indian buyers. These questions were not merely theoretical; they were calculated using Python scripts to ensure absolute accuracy before being presented to the AI models.

The benchmark was divided into four distinct categories, each testing a different pillar of the Indian property-buying experience. The first category, "Conversion," focused on the fluid movement between lakhs, crores, and Lakhs Per Annum (LPA). In a market where a price might be quoted as ₹1.2 crore but a discount is offered in lakhs, the ability to subtract ₹45 lakh from ₹1.2 crore and return a result in lakhs is a fundamental requirement.

The second category addressed "Area," a notorious pain point in Indian real estate. This section tested the models’ understanding of the relationship between carpet area, built-up area, and super built-up area, including the concept of "loading" and the traditional unit of "gaj." The third category, "EMI and Tax," delved into the rigorous world of reducing-balance Equated Monthly Installments (EMIs), stamp duty, Goods and Services Tax (GST), and Tax Deducted at Source (TDS). Finally, the "Format" category tested the models’ ability to adhere to the Indian numbering system, which uses a 2,2,3 grouping (e.g., 9,87,65,432) rather than the Western 3,3,3 grouping (e.g., 98,765,432).

Performance Metrics Across Leading Models

To provide a comprehensive view of the current AI landscape, the benchmark tested several high-profile models across different price tiers and providers. These included Gemini 3.7 Flash, Gemini 3.1 Pro Preview, Claude Sonnet 4.6, and GPT-5.4 mini. The evaluation process was strict: each model was required to provide a final answer in a specific format, and numerical results were checked against the true values. For EMI calculations, a tiny margin of error (±₹2) was permitted to account for different rounding methods, while the formatting tests required an exact string match.