Questions about this study

It means that when Claude Sonnet 5 answered the same buyer question twice from memory and twice with web search turned on, only 16% of all the brand mentions it produced appeared in both the memory answers and the search answers. The rest were split between brands named only when Claude relied on memory (35.1%) and brands named only when search was on (49%). So if you only ever check one mode, you're seeing a partial and lopsided view of which brands Claude is capable of naming for that question.

Memory answers draw on patterns baked into the model during training, which tend to favor well-established, frequently-discussed brands. Web search answers pull from whatever pages rank for that query on the day it's asked, which surfaces newer brands, regionally relevant ones, or ones with strong SEO, regardless of how often they show up in training data. These are fundamentally different retrieval processes, so there's no strong reason to expect them to converge on the same brand set, and the data bears that out: only 16% of mentions out of 33,734 total appeared in both.

No. Across the 81 categories measured, the share of brand mentions appearing in both modes ranged from 2.2% to 36.8%, with a median of 14.7%. So the one-in-six figure in the headline is the overall share across every category, not a number every category actually hits. A category near the low end has almost no stability between modes, while one near the high end is considerably more consistent.

Yes, somewhat. Brands that Claude named first at least once showed overlap between modes of 33%, compared to 13.2% for brands that were never named first. So top picks are a bit more stable across memory and search than brands further down the list, though the difference isn't huge, and neither segment comes close to full overlap.

That's a fair challenge, but the design actually works against that explanation. Each condition (memory and search) was asked twice, and a brand counted as 'named' if it appeared in either repeat. That's a generous counting rule that should inflate overlap, not shrink it, because it gives each mode two chances to produce any given brand. So the true overlap from a single memory answer versus a single search answer is likely lower than what's reported here, not higher. The low overlap isn't an artifact of too few repeats, if anything the method understates the gap.

It can't fully separate the two. The measurement window was a single day (2026-10-19), so the specific brands landing in the 'only with web search' class reflect whatever pages were ranking and getting pulled in on that date. A repeat of this study on a different day could shift which specific brands fall into which class, even if the overall three-way split looks similar. This study tells you that memory and search diverge substantially, but it can't tell you how much of that divergence is inherent to web search versus a snapshot of one day's search results.

No, this study only tested Claude Sonnet 5, accessed through Anthropic's developer API, on a single day. It says nothing directly about ChatGPT, Gemini, or even Claude's consumer app, which may retrieve and rank web results differently, or weight memory versus search differently in how it composes an answer. Anyone wanting to know whether this pattern holds elsewhere would need to run the same paired-question design on those other systems.

Based on this data, quite a lot. 49% of all brand mentions appeared only when search was on, meaning a memory-only monitoring setup would never see those brands at all. Combined with the fact that only 16% of mentions were stable across both modes, a company tracking just one condition is seeing only about half to two thirds of the brands Claude is actually capable of naming for a given question, not the full picture.

The practical implication is to check brand mentions in both modes rather than picking one, since memory-only or search-only monitoring each miss a large, different chunk of the brand set. If your visibility tracking tool only queries Claude with default settings, confirm whether web search is on or off for those queries, and understand that the number you get back under one setting is not a proxy for the other. It is also worth re-checking periodically, since the single-day measurement window here means the specific search-only brands could shift over time even if the overall three-way split stays similar.

The study drew on 33,734 total brand mentions across 33,734 rows, with no rows excluded (0 excluded). Questions were spread across 82 categories, and no single category accounted for more than 2% of the data, so the result isn't being driven by one oversized category. Each brand mention was classified into one of 3 classes: named only from memory, named only with web search, or named with and without web search.

A brand mention is an instance of Claude naming a specific brand in response to a buyer question, extracted by a second model, Claude Haiku, reading each answer against a fixed rubric. Claude Sonnet 5 generated the answers, drawing on buyer questions from Rankxa's industry question pool, and Claude Haiku did the extraction and classification into the three overlap categories. Using a model to do extraction at this scale means occasional misreads are possible, but the rubric was held fixed across every answer, so any such errors should apply fairly evenly rather than favoring one class over another.