Visa's AI Shopping Benchmark Shows Models Fail 42.8% of Tests, Google and Meta Lead

Visa Launches Industry's First AI Shopping Benchmark

Visa has unveiled the Agentic Commerce Benchmark, the first standardized test measuring how well AI agents handle real-world shopping tasks1

1

2

2

. The benchmark tested 22 AI models from 12 labs across 15 distinct shopping scenarios, including purchasing TVs, laptops, and personal care appliances. Models had to identify customer needs through conversation, justify purchase decisions with sufficient information, and crucially, decide not to buy when no product met requirements. Across all tests, AI models failed 42.8% of their runs, with overall pass rates reaching only 57.2%3

3

.

Google and Meta Dominate AI Shopping Performance

Google's Gemini-powered Aragon 4 agent and Meta's Wonder agent tied for top performance, achieving 82% success rates in the AI models shopping test2

2

. Claude secured third position with a 76% success rate, while an OpenAI agent finished sixth at 64%. The performance gap between the best and worst models proved substantial—top performers passed 82.2% of attempts compared to just 28.9% for the lowest-ranked models1

1

. Cost correlated with performance: the five most expensive models per run achieved 67.1% pass rates, while the five cheapest managed only 50.2%3

3

.

AI Agents Make Wrong Purchases in 54% of Failures

The benchmark exposed a critical flaw in AI reliability in shopping: in 54% of failed runs, AI agents still completed purchases despite not meeting customer requirements or failing to uncover all necessary information1

1

. In the TV shopping scenario, customers mentioned a brand recommendation and a third-party supplier membership but withheld details like budget, HDR10+ support for family videos, and gaming features. AI agents that uncovered the most hidden buying factors passed 70.2% of the time, compared to 44.4% for those that discovered the fewest requirements1

1

. Even when models passed a scenario once, they often failed when repeating the same test.

Consumer Trust Remains Major Barrier to AI-Driven Commerce

Visa's Trust Index from September revealed that while 72% of US consumers had used an AI assistant, only 23% trusted AI agents to make payments on their behalf1

1

. "AI agents have the potential to power the next era of commerce, but trust is key to making it work," said Rubail Birwadker, Visa's global head of growth3

3

. Despite trust concerns, a NIQ survey found AI shopping adoption passed 50% among US consumers in September. Nearly 132 million U.S. adults have purchased products with AI assistance, though consumers maintain control over final purchase decisions3

3

.

Product Discovery Challenges and Future of Commerce

The complexity of product discovery emerged as a fundamental challenge for AI agents. "The motivations behind any given purchase are often complicated and, in many cases, consumers lack information and are not certain what they want," the study noted2

2

. Each model received three attempts at every scenario, yet all three tries passed only 34.2% of the time1

1

. Birwadker acknowledged that "we're not there yet from a trust or technology perspective. But we'll soon get into a world where consumers can trust and delegate"2

2

.

Payment Security and Legal Questions Ahead

Beyond product discovery, the future of commerce faces unresolved challenges around payment security and legal liability. Visa has integrated its services into ChatGPT and partnered with Mastercard and Ant International on identity verification for AI agents1

1

. Critical questions remain about who bears liability when agents make rogue purchases or commit fraud. Visa plans to expand the benchmark by adding scenarios, product categories, and extending tests to checkout, payment, and ordering processes. The company has shared results with leading model providers and made the test available to consumers on its Payments Frontier site1

1

2

2

.