Only about one in six brands Claude recommends shows up both with and without web search

If you ask Claude the same buyer question twice with web search turned off and twice with it turned on, how often does it name the same brand in all four answers? This study measures that directly, and the answer is: not often.

Claude Sonnet 5 was given a set of buyer questions drawn from Rankxa's industry question pool, spread across 82 categories. Each question was asked four times through Anthropic's developer API: twice with web search off, twice with it on, with the model, the question text, and the length limit held constant across all four runs. Claude Haiku then read every answer against a fixed rubric and extracted every brand name mentioned, producing 33,734 total brand mentions across 33,734 rows. No rows were excluded.

Every one of those 33,734 mentions falls into exactly one of three classes, depending on which condition it showed up under: named only when the model was working from memory, named only when it had web search, or named in both conditions. Pooling all four answers to each question, the breakdown is:

• Named only from memory: 35.1% of mentions (11,825 of 33,734)
• Named only with web search: 49% of mentions (16,525 of 33,734)
• Named with and without web search: 16% of mentions (5,384 of 33,734)

That last figure is the headline. Roughly one mention in six survives the switch between memory and search. The other five do not: they appear under one condition and vanish under the other.

It's worth being precise about what "overlap" means here, because it is easy to overstate. A brand counted in the with-and-without class does not need to appear in all four individual answers, only in at least one memory answer and at least one search answer, for that question, pooled. So 16% is already a generous estimate of consistency. A stricter test, requiring the brand to appear in every single one of the four answers, would produce a smaller number still. We did not measure that stricter version here, so we can't put a figure on it, but the direction is predictable: tightening the requirement can only shrink the overlap, not grow it.

Why would overlap be this low? Two mechanisms are plausible, and the data can't fully separate them. One is that web search genuinely changes the evidence Claude has in front of it: a search turns up recent reviews, current pricing, or a "best X of 2026" roundup that simply was not part of the model's training data, so new brands enter the running and some memory-only brands lose their edge. The other is noise: brand mentions are inherently somewhat unstable across any four independent generations, search or no search, because the model is sampling from a distribution of plausible answers each time, not retrieving a single fixed list. If noise alone were driving this, we'd expect the same instability even between two memory-only runs, or two search-only runs. This study did not isolate same-condition repeat-ask variance, so we can't rule noise in or out as a contributor, but it's unlikely to be the whole story, given how differently sized the memory-only and search-only buckets are. If it were pure sampling noise, you'd expect memory-only and search-only mentions to be roughly balanced; instead search-only mentions (49%) outnumber memory-only mentions (35.1%) by a wide margin, which looks more like search surfacing a genuinely different, and larger, candidate pool than like noise alone.

For a brand owner, the practical reading is this: ranking well in Claude's "memory" (meaning, being the kind of brand Claude would name without any tool use, presumably reflecting training-data presence, frequency of mention online, and general reputation signal) does not reliably translate into being named once a user's query triggers a web search, and vice versa. The two are closer to separate visibility channels than to one ranking viewed two ways. A brand optimizing only for one is leaving the other largely to chance.