Google Unveils Android Bench 2.0 to Evaluate AI Performance on Complex, Long-Horizon Development Tasks

To address this, Android Bench 2.0 utilizes a "continuous scoring" system. This methodology provides a more granular and meaningful indication of a model’s performance. Instead of a simple "yes" or "no," the benchmark evaluates the progress made toward the final goal, rewarding models that successfully navigate the majority of a complex task even if they do not reach a perfect conclusion. This approach is intended to provide researchers and developers with a better understanding of the "near-misses"—areas where a model shows strong reasoning but perhaps lacks a specific technical execution.

This shift to continuous scoring is expected to be highly beneficial for AI researchers. It allows them to identify specific bottlenecks in a model’s logic. For instance, a model might be excellent at architectural planning but struggle with the specific syntax of Jetpack Compose, or it might be great at writing logic but fail at configuring the Gradle build system. By seeing a gradient of success, developers can more effectively fine-tune their models for the specific rigors of the Android ecosystem.

The Leaderboard: GPT-6 Astra Takes the Lead

With the launch of the new benchmark, Google has also released an updated leaderboard featuring some of the most prominent AI models currently in development. The results provide a sobering look at the current state of AI in software engineering, highlighting that while progress is being made, there is still a significant gap between AI capabilities and human-level proficiency.

At the top of the Android Bench 2.0 leaderboard sits GPT-6 Astra, which achieved a pass rate of 28%. While 28% may seem low in a traditional academic sense, within the context of long-horizon tasks that take humans days to complete, it represents a significant achievement in AI reasoning and persistence. GPT-6 Astra demonstrated a superior ability to maintain context over long periods and handle the architectural complexity required by the new LHT dataset.

Trailing behind the leader are several other high-profile models. GPT-5.6 Sol and Claude Fable 5.1 also made appearances on the leaderboard, showing competitive results in specific categories of development. Claude Opus 5 was also evaluated, continuing the trend of high-performance models being put to the test against the Android framework.