Visa Agentic Commerce Benchmark Results, an Analysis

Analysis. This article reports Visa’s published benchmark results first, then gives EC4IM’s interpretation of what they mean for online sellers of restricted products. Sections labeled “EC4IM’s reading” are interpretation, not findings from Visa.

Visa’s first Agentic Commerce Benchmark put 22 AI models from 12 labs through 15 simulated shopping tasks, and the models passed 57.2% of their runs. Visa, which released the results on November 5, 2026, gave each model three attempts at every task. All three attempts passed only 34.2% of the time, and 54% of the failed runs still ended with the AI making a purchase.

That last figure matters most for merchants. In more than half of the failures, the shopping assistant did not walk away. It chose a product that missed something the customer needed, or chose before it had found out what the customer needed.

What Visa tested

Each test, which Visa calls an episode, is a conversation between two AI models. The model being tested acts as the shopping assistant. A second model plays the customer and holds a private file of requirements, budget and existing products. According to Visa’s October 2026 white paper, most of those facts reach the conversation only when the assistant asks the question that unlocks them.

The assistant searches a fixed, simulated market built largely from public product pages and reviews, not the live web. A run passes only if the assistant uncovers every requirement the task defines and then makes the right call, either buying a product that fits or declining when nothing does. A product that happens to match the answer key without the assistant having established why counts as a failure. The tasks included buying a TV, a laptop and a personal care appliance.

The benchmark ends at the decision. Visa’s white paper says payment, order placement and fulfillment are outside the current scope, so a “purchase” in these results means the customer committed to an offer, not that a card was charged.

The numbers behind the headline

Visa’s results page breaks all runs into three outcomes. Passing runs made up 57.2%. Runs that ended in a purchase that did not pass made up 23.1%. The remaining 19.7% ended with no purchase although a suitable product existed, usually because the assistant said there was none, or with no decision at all, usually because the assistant hit its tool-call limit or replied in the wrong format.

Spread between models. The best models passed 82.2% of attempts and the weakest 28.9%. Visa says the confidence intervals are wide and that no model clearly leads.
Asking questions paid off. The five models that uncovered the most hidden buying factors averaged a 70.2% pass rate. The five that uncovered the fewest passed 44.4%.
Cost helped, but not reliably. The five most expensive models per run passed 67.1% of runs, compared with 50.2% for the five cheapest.
Repeat runs diverged. A model that passed a task once often failed the same task on another attempt. No model passed all three runs on more than 67% of the episodes.

Visa lists five failure patterns the episodes are built to catch. Two stand out for merchants. One is meeting every stated requirement while never asking about unstated ones. The other is treating a value missing from a product listing as acceptable.

Where Visa is taking the test

Visa plans to add shopping scenarios and product categories and to extend testing to checkout, payment and order placement. It said it has shared the benchmark with model providers for feedback. Rubail Birwadker, Visa’s global head of growth, said in the release that “trust is key to making it work.”

Visa tied the benchmark to its Trust Index, released September 9, 2026. That survey, run for Visa by the Harris Poll among 2,065 U.S. adults from May 26 to 28, 2026, found that 72% had used an AI assistant but only 23% trusted generative AI to handle payment transactions on their behalf.

EC4IM’s reading, a wrong pick is still an order

Visa did not measure refunds, returns or disputes, and its tasks involved ordinary electronics and appliances. The following is interpretation. In the benchmark, a run that picks the wrong TV is a scoring failure. In a live store with checkout connected, the same behavior would produce a real order for something the customer did not want, and the merchant would be the one fielding the return request or the card dispute that follows.

The 34.2% consistency figure points the same way. If the same request can produce a correct choice on one run and a wrong one on the next, a merchant cannot assume that an agent which ordered correctly last week will order correctly this week.

EC4IM’s reading, restricted sellers carry the gap

The two failure patterns Visa singled out map closely onto restricted goods. An agent that never asks about unstated requirements may not ask whether the buyer is old enough for the product or whether it can legally ship to the buyer’s state. An agent that treats a missing listing value as acceptable may read a product page that says nothing about delivery limits as having none.

That suggests age and shipping checks belong in the merchant’s own checkout and fulfillment flow, run on every order regardless of whether a person or an agent placed it, rather than relying on the agent to have asked. EC4IM looked at one platform’s version of this question in What Age-Restricted Shopify Sellers Should Check as Shop MCP Opens Agent Checkout. Visa’s results add evidence on how often agents miss information they were never told to collect.

What the benchmark does not show

Checkout and payment. Visa has not yet tested them, so the results say nothing about how agents handle card entry, authentication or order confirmation.
Live conditions. Both the customer and the market are simulated, and the 15 episodes are private apart from one public demo.
Restricted products. None of the published task examples involves age-restricted or state-restricted goods.

A benchmark that stops one step short of the merchant

Visa’s first round shows AI shopping assistants passing a little more than half of realistic buying tasks, passing consistently about a third of the time, and choosing to buy in more than half of the runs they got wrong. The benchmark stops before checkout, so the cost of those wrong picks to a merchant is not measured yet. Visa says checkout, payment and order placement come next. For sellers whose products carry age or shipping rules, the results so far argue for checks the store runs itself.

Visa Benchmark Finds AI Shoppers Still Buy When They Fail