Questions about this study

It means that when you ask Claude the same buyer question twice and compare the two answers, the brand mentioned first in one answer usually is not the brand mentioned first in the other, or is not mentioned at all. Across 7,996 paired answers, the same brand held the first position in only 23.2% of pairs. The most common single outcome, at 31.4% of pairs, was the first-named brand vanishing from the other answer entirely. So "keeping the first pick" is the exception, not the rule, across this question set.

Both, but search makes it noticeably worse. Even with search off on both sides, asking Claude the identical question twice from memory only produced the same first-named brand 35.8% of the time, so there is baseline instability. But when one memory answer is paired against one web-search answer, that figure drops to 13.9%, and the first brand disappears entirely in 40.6% of those pairs. The honest reading is that memory alone is already shaky, and adding live search results widens the gap further.

The likely mechanism is that web search pulls in whatever pages rank or get retrieved at that moment, which is a different input than whatever the model learned during training. If those two sources disagree on which brands are even relevant, the model does not just resequence a fixed list, it can produce an answer that never surfaces the other source's top pick at all. This study cannot confirm that mechanism directly, since it only compares final answers, not retrieved pages or model reasoning. A test that logged the actual search results returned on each call would be needed to distinguish "different sources, different facts" from "same facts, different emphasis."

You don't, and this is a real limitation. The entire measurement window is 2026-10-19 to 2026-10-19, a single day. Web search results can shift day to day for reasons unrelated to any underlying instability in the model itself, so some of what looks like "Claude changes its mind" could instead be "the web changed overnight." Repeating this same design on multiple separate days and checking whether the same-brand-first rate stays near 23.2% or moves around would tell you how much of the effect is the model versus the day's search index.

Something else. The study sent questions through Anthropic's developer API, not the consumer Claude app. That matters because the API calls used here have no conversation history and no account-level personalization, while the app many people use day to day may behave differently for logged-in users with memory of past chats. The finding is specifically about Claude Sonnet 5 answering fresh questions through the API, with and without a tool-use web search capability, not a claim about what you'd see typing into claude.ai.

No. The method only checks whether the first-named brand in one answer matches, survives, or disappears in the paired answer. It has no independent way to judge which of the two answers is actually correct or better reflects current market reality. A brand disappearing from the search-based answer could mean the memory answer was outdated, or it could mean the search answer missed something real. Settling that would require a separate accuracy check against some ground truth for each question, which this study does not attempt.

Not entirely, though categories do vary a lot. Across 80 question categories, the share of pairs keeping the same first brand ranged from 5% up to 49%, which is a wide spread. But no single category dominates the dataset: the largest category accounts for only 1.3% of all rows, so the overall figures are not being driven by one oversized group. They are genuinely an average over quite different category-level behavior, which means a brand's experience could look very different depending on which category its questions fall into.

Treat a single good mention from Claude, especially one produced without search, as a point-in-time result rather than a stable ranking. Since the same brand repeats in first place in only 23.2% of paired answers overall, and even less often once search is involved, a team that spot-checks visibility once and stops is likely to overstate how secure that position is. A more useful practice is to query the same question repeatedly, with and without search enabled, and track how often you appear at all, since "one of the two answers names no brand" is itself common, at 26.9% of pairs, meaning sometimes the fix is getting named at all, not just getting named first.

The classification is built around whether a brand name identified in one answer appears in the paired answer, and whether it's the one named first there too. The four possible outcomes are mutually exclusive and cover every pair: the same brand comes first in both, a different brand comes first but the original is still mentioned somewhere, the original brand is absent from the other answer altogether, or one of the two answers names no brand at all. The fact table does not separately report how spelling variants or close name matches were handled, so if that's a concern for your use case, treat the categories as describing bottom-line outcomes rather than a guarantee of exact-string matching.

Each buyer question from Rankxa's question pool, spread across 82 categories, was sent to Claude Sonnet 5 through Anthropic's developer API four times: twice with no tools so the model answers from what it already knows, and twice with web search turned on. Those four answers per question were then combined into pairs, memory against memory, memory against search, and search against search, giving 7,996 pairs total across 7,996 rows, with 0 rows excluded from the analysis. Each pair was then classified into one of four outcomes based on whether the first-named brand repeated, moved, disappeared, or was never named at all.

It's in between. When both answers came from web search, the same brand held first position in 29.4% of pairs, compared with 13.9% when one answer came from memory and the other from search, and 35.8% when both came from memory alone. So two independent search calls agree with each other somewhat more often than a memory answer agrees with a search answer, which is consistent with the idea that mixing two different information sources, rather than search itself, is a big part of what drives instability.