Only about one in six brands Claude recommends shows up both with and without web search

Why this matters

When someone asks Claude which project management tool or which mattress brand to buy, the model answers in one of two modes. Either it draws on patterns learned during training, with no access to the live web, or it runs a search and reads current pages before answering. Brands care about this distinction because it determines which lever actually moves the needle. If a brand's visibility in Claude's answers depends mostly on web search, then the usual SEO and PR playbook (getting pages indexed, getting cited, getting fresh coverage) is what matters. If visibility depends mostly on what the model memorized during training, then the relevant work happened months or years ago, during whatever period shaped the training data, and current publishing activity may do very little.

This study asks a narrower, more measurable version of that question: when Claude names a brand, how often does that brand show up regardless of whether search is on, versus showing up in only one condition or the other? A brand that appears in both conditions is a brand with a redundant path into the answer. A brand that appears in only one is a brand with a single point of failure. If a company only ever shows up when search is off, a future model update, a change in training data, or a shift toward search-heavy defaults could erase it from the answer with no warning and no obvious cause.

The finding at the center of this study is that only 16% of all brand mentions, across 33,734 mentions pooled from 82 categories, appeared in both the memory condition and the search condition. The majority of mentions were specific to one mode or the other. That has a direct implication for anyone trying to diagnose why their brand does or does not appear in Claude's answers: the diagnosis depends on which condition you are even looking at, and a brand can look present in one audit and absent in another without anything about the brand having changed.

The rest of this page works through how that number was built, what it does and does not support, and how unevenly it holds across the 82 categories measured.

How the measurement works

The design is a controlled comparison, not a survey. Claude Sonnet 5 was given the same set of buyer-style questions, drawn from Rankxa's industry question pool, spread across 82 categories. Each question was asked four times: twice with web search turned off and twice with it turned on, with the model, the exact question text, and the response length limit held constant across all four runs. Holding those constant matters because it isolates one variable, whether the model could search, rather than letting prompt wording or response length differences confound the comparison.

A separate model, Claude Haiku, then read every one of the resulting answers against a fixed rubric and extracted the brand names mentioned. This is the step that turns free text into countable units. Each brand mention identified by the rubric is one row in the dataset. Across all four answers to all questions in all 82 categories, this produced 33,734 brand mentions spread across 33,734 rows, and 0 rows were excluded from the analysis.

What counts as one unit. The unit of analysis is a single named brand within a single answer. If the same brand appears in both of a question's two memory-condition answers, that is counted differently from appearing once in memory and once in search, because the classification tracks which condition(s) a brand appeared in, pooling the two repeated answers per condition per question. There are three possible classes a brand-question pairing can fall into:

Asking each question twice per condition, rather than once, guards against a single run's idiosyncrasy being mistaken for a stable pattern. A brand that shows up in one of two memory answers but not the other is a weaker case of "named from memory" than one that shows up in both, and the pooling approach treats presence in at least one answer per condition as the threshold for membership in that condition.

No rows were dropped from this analysis (0 excluded), and the largest single category accounts for only 2% of all mentions, so the headline figures are not being driven by one oversized group swamping the rest.

The headline result, and its limits

Pooling across all 82 categories, the 33,734 brand mentions split into three classes:

The class named in the title of this study, brands that show up in both conditions, is the smallest of the three. Just 16% of mentions fall into it. The largest class is brands named only when search was on, at 49%, followed closely by brands named only from memory, at 35.1%.

What this does mean. For a majority of brand mentions, whether Claude names a brand at all depends on whether it searched. This is a statement about overlap, not about which condition names more brands overall or which is more accurate. A brand appearing only with search is not necessarily a better or more legitimate answer than one appearing only from memory. It simply means that mode produced a mention and the other did not, in this measurement window, with this question set.

What this does not mean. The result does not say that brands named from memory are outdated or that brands named via search are more trustworthy. It does not say anything about the user's actual experience, since a real user's Claude session uses one setting at a time, not both at once; the comparison here is diagnostic, built to expose sensitivity to the search toggle, not a description of what any single user sees. It also does not establish causation about why overlap is low, only that it is low. A plausible mechanical explanation is that search retrieves current, often different sources each time, introducing brands that were simply not prominent in training data, while memory recall draws on a comparatively fixed, static set learned during training. Distinguishing that explanation from, say, instability in how Claude Haiku's rubric classifies borderline mentions would require auditing a sample of the rubric's judgments directly, which this study has not done.

Is this concentrated or widespread?

A headline percentage can hide a lot. It could be that the low overlap figure is driven by a handful of categories with near-zero overlap, while most categories look quite different, or it could be that something close to this pattern holds everywhere. These are different findings with different implications, so the study also breaks the classification out by segment, specifically whether a brand was ever named first in an answer versus never named first.

The pattern holds in both segments, but not identically. Brands that were named first at least once show higher overlap, 33%, than brands never named first, at 13.2%. That is a meaningful difference: being the brand a user actually reads about first seems to go along with a somewhat more durable presence across both conditions. It makes some mechanical sense. A brand prominent enough to be the leading answer under one condition is more likely to also be prominent, even if not leading, under the other. But the gap is one of degree, roughly one third versus roughly one in eight, not a reversal. The low-overlap finding is not an artifact of one segment; it shows up whether or not the brand was ever a leading answer, which is the first sign that this is closer to a property of the whole population of mentions than a quirk of a subgroup.

The next section checks the same question at the level of the 82 individual categories, which is a stricter test, since categories are a more natural unit for a brand manager to ask "does this apply to my industry" than a segment defined after the fact by first-mention status.

Spread across groups

The classification was recomputed separately within each of the 82 categories; 81 groups had enough mentions to compute a share for the "both" class. Rather than one national number, this produces a distribution, summarized here by its minimum, median, and maximum across those groups:

Two things stand out. First, the "both" class, the one in the study's title, never becomes the majority class in any category: even its maximum across 81 groups is 36.8%, well under half. So the one-in-six figure is not an average masking a mix of high-overlap and low-overlap categories, it is closer to a ceiling that individual categories approach but do not cross. Second, there is real range beneath that ceiling. The lowest-overlap category sits at 2.2%, meaning in that category brands named in both conditions are rare even by this study's already-low overall standard, while the highest sits at 36.8%, more than sixteen times higher.

That range matters for how the finding should be used. A category sitting near the 2.2% end has brand visibility that is almost entirely contingent on the search toggle; a company there should treat memory-only and search-only presence as close to independent problems. A category near the 36.8% end still has low overlap in absolute terms, but proportionally more of a shared core. This study does not report which specific categories sit at which end, so a company cannot look up its own industry's exact position from the figures given here. What it can take away is that the direction, low overlap, is consistent across essentially the full range of categories measured, even though the magnitude is not uniform.

What the method cannot see

No calibration or inter-rater reliability figure is reported for how Claude Haiku's rubric-based extraction compares to a human-labeled baseline, and 0 mitigations were applied to adjust for extraction error in this run. That is worth stating plainly rather than glossing over, because the entire study depends on Haiku correctly identifying which brand names appear in Sonnet's answers. If the rubric over- or under-counts certain kinds of mentions, for instance abbreviated brand names, brands mentioned only in passing, or close variants of a brand name used inconsistently across the four answers, the classification into the three overlap classes would shift without the underlying overlap in Sonnet's actual behavior having changed at all. A reader should treat the exact percentages as estimates produced by one rubric-and-extractor pairing, not as ground truth.

Several other limits apply. The measurement window is a single day, 2026-10-19 to 2026-10-19, so the result is a snapshot, not a trend; Claude's training data, search index, and even its default behavior can all shift over time, and this study cannot say whether the one-in-six figure was different a month earlier or will hold a month later. The question set is drawn from one source, Rankxa's industry question pool, and buyer questions phrased differently (more specific, more comparative, more casual) could plausibly shift how often any one brand gets named at all, which would mechanically shift the overlap figures too, since a brand named in only one of four answers behaves differently from one named in all four. The study also only covers one model, Claude Sonnet 5 answering and Claude Haiku grading, so nothing here speaks to how other assistants handle the same split between trained knowledge and live search.

Finally, pooling two answers per condition per question (rather than requiring a brand to appear in both repeats of a condition to count as a member of that class) is a specific methodological choice that leans toward classifying more brands as present in a condition, which in turn could modestly understate true overlap if a brand's appearance in a single answer out of two is closer to noise than signal. None of these limits overturn the direction of the finding, but they bound how precisely the 16% figure should be read, and they are exactly the kind of detail that should accompany the number whenever it gets cited.

What to do differently on Monday

The practical implication of low overlap is that a single audit of Claude's answers, run under one search setting, tells you about one half of the picture at best. A company checking whether it gets recommended by Claude and only testing with search on, or only testing with search off, is measuring a mode-specific outcome, not the brand's overall standing with the model. Given that 16% of mentions are shared between the two conditions, finding your brand under one setting gives you very little basis for assuming it also appears under the other.

Concretely, that suggests three changes to how a brand monitors or tries to influence its presence in Claude's answers:

1. Test both conditions, every time. Any monitoring process should run the same prompts with web search explicitly on and explicitly off, and report the two results separately rather than averaging or picking one. A brand that disappears when search is toggled is not a measurement error, based on this study it is the expected pattern for roughly five out of six mentions.

1. Separate the two kinds of work. Showing up in the search condition is plausibly tied to current, crawlable, citable content: pages that rank well, recent coverage, structured information a retrieval step can find and surface. Showing up in the memory condition is plausibly tied to whatever shaped the model's training data, which for an already-trained model is not something a current publishing effort can retroactively change. Treating these as one undifferentiated