Claude Fable 5 dominates Google's revamped Android AI dev benchmark with 84.5% accuracy

Google revamps Android AI dev benchmark, adds Fable 5 and other agents

Code generation is emerging as one of the most popular applications for large language models (LLMs), but not all agents are equally good at all development tasks. Google created a benchmark earlier this year to evaluate how LLMs perform in Android app development, and Android Bench is getting a big update today. The leaderboard now includes a raft of new models, and Google has adopted a new framework that should be easier to use. Developers are invited to run their own tests and submit feedback that could shape the future of Android Bench. While they are popular coding tools, LLMs don't get everything right. Separating the useful outputs from straight-up slop means choosing the right tool. Android Bench aims to demonstrate which AI agents do best on a suite of 100 Android development tasks. After launching Android Bench in March, Google has added metrics like cost and efficiency, as well as open-weight models. To keep Android Bench relevant, Google is updating the test with eight new models, including all the latest heavy-hitters: Claude Fable 5, Claude Sonnet 5, Claude Opus 4.8, GLM 5.2, Kimi K2.7 Code, MiniMax M3, Qwen 3.7 Plus, and Qwen 3.7 Max. Even the initial release of Android Bench didn't have Google's AI models at the top -- OpenAI's latest LLMs were slightly in the lead. The story is worse for Gemini now that Google has expanded the lineup. In the new leaderboard, Gemini 3.1 Pro is in fifth place, behind GPT 5.4, Claude Sonnet 5, and Claude Fable 5. In fact, Fable 5 lives up to the hype with a sizeable lead at 84.5 percent accuracy in the test. However, Fable 5 and GPT 5.5 also have extremely high operating costs, chewing through more than $130 in tokens for the 100-problem, 10-run benchmark. Gemini 3.1 Pro didn't score as high, but it only costs $87 to run the test. Gemini 3.5 Flash, which is supposed to be cheaper to run than other models, has the highest cost on the leaderboard because it took so much longer to complete the benchmark: $165 per run and a 28-hour runtime. The Android coding performance gap for Google's models is a problem as the company shifts many of its projects toward agentic development. Obviously, Google would prefer that Android developers use Google's tools in their workflows, which may be why Google has reportedly been offering to buy application source code from developers for AI training. Community collaboration Android Bench is supposed to evolve over time, adopting new workflows to test models. Google hopes that developers will want to contribute to Android Bench by sharing benchmarks and development tasks. To make that more feasible, Google is switching to the Harbor framework. According to the company, this testing sandbox makes it easy for developers to run, evaluate, and share results for Android Bench. Google re-ran all its previous tests with Harbor to get a new baseline for LLM performance. So there's been some shift in the previously reported scores even though the underlying tests haven't changed (yet). The historical data will remain online in an archive. With the new, easier framework, developers can run their own development tasks against Android Bench and submit those for possible inclusion in the official test. The Android Bench GitHub has been updated with the new dataset and instructions on how to get involved.

Google is changing how it judges AI models for Android coding, updates list with Fable 5

Google is changing how it tests AI models for Android coding following new rankings led by Fable 5. Google released the "Android Bench" earlier in the year as a glanceable ranking system for the best AI models. Those rankings are all based around the model's ability to code for Android - a general ranking system it is not, but incredibly useful for developers. Google says there are a couple of changes coming to how it ranks those AI models for Android development. The core benchmarking system is being changed to the standardized Harbor framework. As it stood, models like GPT-5.5, Claude Opus 4.7, and Gemini 3.1 Pro Preview were ranked based on a mini-swe-agent v1 benchmark tool developed for general use. Switching tracks and opening the system up to the Harbor framework allows Android developers to use the same tools to analyze AI models for individual use cases. In fact, Google says it's opening up the Android Bench to users willing to submit Android development tasks. Those will be used to evaluate how models handle each scenario. Developers are also being invited to share their benchmark evaluations. From the beginning, we've valued an open and transparent approach, which is why we made our original methodology and test harness publicly available on GitHub. You've asked for a way to provide feedback on our dataset, so now we're taking collaboration a step further by giving you, the Android developer community, a chance to shape Android Bench. Android Bench adds Claude Fable 5 To top off the new approach, Google has refreshed the Android Bench list using the new framework for AI models. Each model between the closed-weight and open-weight variants has been reevaluated on the new testing bench. Unsurprisingly, Claude Fable 5 sits at the top of the list with a rather comfortable lead. Google gave it a score of 84.5, a healthy 4 points ahead of GPT-5.5, ranked at 80.2. Claude Sonnet 5 is nearly 10 points lower than Fable 5. Anthropic's power claims seem to be reasonable, and it's worth noting that Fable 5 still has heavy restrictions put in place. Most AI models that had a spot in the previous Android Bench rankings have received a new score, since the underlying rules have essentially changed. Here are the new rankings as Android Bench has them listed: