Navigating the "Lakh and Crore" Divide: New AI Benchmark Exposes Significant Errors in Indian Real Estate Mathematics

This highlights a broader challenge in prompt engineering and model training. Real-world users often require data in a specific format to paste into tax forms, loan applications, or spreadsheets. A model that understands the math but ignores the formatting constraints fails to be a truly useful assistant in a professional or administrative context.

A Lesson in Benchmarking Integrity

Perhaps the most valuable lesson derived from this exercise was not about the models themselves, but about the methodology of benchmarking. During the initial testing phase, early data suggested that Claude Opus 5 had failed nearly every question, while GPT-5.5 had errored out entirely.

Upon closer inspection of the run logs, the researcher discovered that the failures were not due to the models’ lack of intelligence, but rather due to API quota limits. Because the cases were fired in parallel, the more expensive, high-resource models quickly exceeded their available quotas. The scoring script had initially counted these "Permission Denied" errors as wrong answers.

This discovery led to a total overhaul of the benchmarking process, moving to sequential calls with robust retry logic. It served as a stark reminder to the developer community: a benchmark score is only as honest as its error handling. If a leaderboard cannot distinguish between a model’s incorrect guess and a system-level failure, the results are essentially meaningless. Consequently, the most expensive models were excluded from the final public tally to ensure that the published numbers were beyond reproach.

The Path Forward for Indian Real Estate AI

As AI continues to permeate the Indian real estate sector, the "Lakh, Crore and Carpet Area" benchmark provides a roadmap for future improvements. The next phase of testing is expected to delve into even more localized complexities, such as the nuances of GST on under-construction versus ready-to-move-in properties, the calculation of Pradhan Mantri Awas Yojana (PMAY) subsidies, and the variations in stamp duty across different Indian states.

For developers and homebuyers alike, the message is clear: while AI is a powerful tool for summarizing brochures and explaining general concepts, it is not yet a replacement for a financial calculator or a legal expert. As long as "western comma drift" and EMI hallucinations persist, the human element—and a good Python script—remains an essential part of the home-buying journey in India. The benchmark is now open for public contribution, inviting others to fork the code, add city-specific tax laws, and continue testing the limits of these digital assistants.