Does turning on web search make Claude's brand picks less stable?

This study compared three kinds of answer pairs from Claude Sonnet 5: two memory-only answers to the same question, two web-search answers to the same question, and one of each, memory against search. All three pair types answer the same underlying question, does the brand named first in one answer also come first in the other, but they isolate different sources of change.

When both answers came from memory alone, the same brand came first again in 35.8% of pairs. When both answers came from web search, the figure was 29.4%, close to the memory-only rate. But when one answer came from memory and the other from web search, the same brand survived in only 13.9% of pairs, roughly half the memory-only rate. The denominators differ by design: the memory-against-search comparison drew on 3,998 pairs, about double the 1,999 pairs and 1,999 pairs in each same-condition comparison, because every question contributed two memory-vs-search pairings against one same-condition pairing on each side.

The mechanism this points to is retrieval, not just randomness. When Claude answers from memory twice, any instability comes from sampling variance alone, the model drawing a slightly different path through its own trained probabilities each time, with no new information entering the process. When Claude answers from web search twice, a comparable instability shows up, which suggests sampling variance affects search answers about as much as memory answers do on their own. But when memory is pitted against search, something additional happens: the brand missing from the other answer entirely jumped to 40.6% of those pairs, well above the 17.4% seen in memory-only pairs and the 27.1% seen in search-only pairs. That gap is the signature of a genuinely different information source, not just a second roll of the dice. A web search call pulls in current page content, rankings, and phrasing that simply are not available to a memory-only answer, so the two answers are drawing on different evidence, not just different random seeds from the same evidence.

An alternative explanation worth naming directly: maybe search answers are just noisier in general, and the elevated missing-brand rate in the mixed comparison is really a search-side effect that would show up in search-against-search pairs too, if given a larger sample. The data here argues against that reading, because search-against-search pairs showed a missing-brand rate of 27.1%, not far from the memory-only figure, and well below the mixed-pair figure. The instability specific to the mixed comparison looks like it comes from the mismatch between sources, not from search being inherently erratic.

What would distinguish these explanations further is outside what this single-window study can show. A model update, a different web search backend, or a longer measurement window than the single day covered here (2026-10-19) could each shift these numbers, and repeating the measurement over time is the natural next step.

For a company watching its position in AI answers, this matters concretely. If your brand is the first answer Claude gives from its trained knowledge, that position offers little protection against what the model says once it looks something up live. The two answers are different enough that confirming one tells you very little about the other. If search visibility matters to you, the memory-side answer, however favorable, cannot be treated as a proxy for what a user doing a web-search-enabled query would see.