Methodology

Engines and exact versions

A study that does not name the model version it ran against is not reproducible, because the answer changes when the model does.

Engine	Model version

Run window, UTC: [object Object].

Method

The design is a controlled comparison, not a survey. Claude Sonnet 5 was given the same set of buyer-style questions, drawn from Rankxa's industry question pool, spread across 82 categories. Each question was asked four times: twice with web search turned off and twice with it turned on, with the model, the exact question text, and the response length limit held constant across all four runs. Holding those constant matters because it isolates one variable, whether the model could search, rather than letting prompt wording or response length differences confound the comparison.

A separate model, Claude Haiku, then read every one of the resulting answers against a fixed rubric and extracted the brand names mentioned. This is the step that turns free text into countable units. Each brand mention identified by the rubric is one row in the dataset. Across all four answers to all questions in all 82 categories, this produced 33,734 brand mentions spread across 33,734 rows, and 0 rows were excluded from the analysis.

What counts as one unit. The unit of analysis is a single named brand within a single answer. If the same brand appears in both of a question's two memory-condition answers, that is counted differently from appearing once in memory and once in search, because the classification tracks which condition(s) a brand appeared in, pooling the two repeated answers per condition per question. There are three possible classes a brand-question pairing can fall into:

Class	Meaning
Named only from memory	Appeared when search was off, never when it was on
Named only with web search	Appeared when search was on, never when it was off
Named with and without web search	Appeared in both conditions

Asking each question twice per condition, rather than once, guards against a single run's idiosyncrasy being mistaken for a stable pattern. A brand that shows up in one of two memory answers but not the other is a weaker case of "named from memory" than one that shows up in both, and the pooling approach treats presence in at least one answer per condition as the threshold for membership in that condition.

No rows were dropped from this analysis (0 excluded), and the largest single category accounts for only 2% of all mentions, so the headline figures are not being driven by one oversized group swamping the rest.

Limitations we volunteer

Written by us, before anyone else found them.

Terms used in this study

Web search tool (on vs. off)

A setting in Anthropic's developer API that lets Claude issue up to two web searches per answer. With it off, Claude answers purely from what it learned during training; with it on, it can pull in current page content before answering.

Message Batches service

An Anthropic API feature for submitting many prompts together and collecting the results as a batch, used here so every question-condition combination was processed under identical conditions on the same day.

Named only from memory

A brand that appeared in at least one of the two web-search-off answers but in neither of the two web-search-on answers for that question.

Named only with web search

A brand that appeared in at least one of the two web-search-on answers but in neither of the two web-search-off answers for that question.

Named with and without web search

A brand that appeared in at least one answer from each condition, memory and web search, for the same question; this is the overlap class the headline figure describes.

Unit

The thing being counted: one brand named in response to one question, counted once regardless of how many of the four answers mentioned it.

Segment: named first

Whether a brand was the first one listed in at least one of the four answers, used to check if the overlap pattern differs between an assistant's top pick and the rest of its list.

Group

The buyer-question category a question belongs to; results are broken out by group to check that the overall pattern is not driven by one or two categories.

Rubric for counting a brand

The fixed rule Claude Haiku applied when reading each answer: a brand counts if it is presented as an option, recommendation, or notable player, but not if it only appears as a cited source, review site, or directory.

Name matching

The method for treating two different-looking names as the same brand: text is lowercased, punctuation and company suffixes are dropped, and a name that is another name plus extra words, such as a product line under a parent company, is folded into the parent.

References

Sources this study reads against. Every link was fetched and confirmed reachable at publication.

1. Anthropic Web Search Tool Cobus Greyling (Substack) Notes that Claude decides for itself whether to invoke web search and that Brave Search is the underlying provider, relevant to why our with-search condition is not deterministic.
2. Estimating LLM Consistency: A User Baseline vs Surrogate Metrics arXiv (Wu, Lin, Akgul, Bauer; Carnegie Mellon/RSAC Labs) Peer-reviewed paper (n=2,976 user study) showing common surrogate metrics for LLM consistency do not track how users actually perceive consistency; cited by SparkToro/Gumshoe as their methodological model.
3. We Asked 10 LLMs to Recommend Brands. They Gave Us 29 Different Answers. dev.to Adjacent benchmark reporting that Microsoft Copilot's brand list had zero overlap with nine other models, illustrating how search grounding can diverge sharply from base model output.
4. The New Rules of Brand Consistency in the Age of AI Search Progress.com Summarizes the SparkToro/Gumshoe data point that a user would need to repeat the same Claude query 1,429 times on average before seeing two identical, identically ordered brand lists.
5. NEW Research: AIs are highly inconsistent when recommending brands or products; marketers should take care when tracking AI visibility SparkToro/Gumshoe.ai Primary benchmark finding that repeated identical prompts to ChatGPT, Claude, and Google AI Overview rarely return the same brand list, based on 600 volunteers running 12 prompts across 3 tools 2,961 times.
6. LLM Mentions vs Citations: What They Mean and Why Both Matter AirOps Explains the general distinction between memory-based mentions and search-triggered, footnoted citations, which frames our memory-versus-search classes.
7. Search Is Everywhere. Winning Brands Are Too. Yext Industry benchmark finding Claude leans on reviews and social media 2 to 4 times more than competing models across seven sectors.
8. Estimating LLM Consistency: A User Baseline vs Surrogate Metrics ACL Anthology, EMNLP 2025 Confirms the peer-reviewed publication record of the Carnegie Mellon consistency paper.
9. Web search tool Anthropic, Claude Platform Docs Official documentation of how Claude's web search tool is invoked and billed, establishing the mechanism our with-search condition relies on.
10. AI Brand Recommendation Study: Why Intent Type Predicts AI Output Consistency Conductor Academy Independent large-scale benchmark (14,000 API calls, four LLMs, pairwise Jaccard similarity over 1,225 pairs per group) measuring run-to-run brand list consistency, used here as a methodological comparison point.
11. AI brand recommendations vary, marketers warned on tracking tools Communicate Online Reports the same SparkToro/Gumshoe study's finding that Claude was only slightly more repeatable than ChatGPT or Google, and that brand visibility (appearance frequency) looked more stable than ranking position.
12. LLM Brand Mentions: How To Track, Key Metrics & More Meltwater Represents mainstream industry practice for tracking brand mentions across models, used as a baseline framing our study refines.
13. Guide to LLM Tracking and AI Search Visibility for SEO Agencies Keyword.com Represents the standard industry framing that consistent LLM mentions signal authority, which our overlap findings complicate.
14. Your Brand Ranks #1 on Google. Claude Ignores It. Topify Industry analysis reporting Claude's web search results overlap with Brave Search's top 15 organic results at a rate of 86.7%, relevant to why our with-search brand lists differ from Google-indexed SEO data.
15. Claude Citation Study Reveals What Actually Gets Cited Otterly.ai Independent citation-pattern study finding company/product domains account for 64.0% of Claude citations and only 13% domain overlap between Claude and ChatGPT citations.
16. Web search with Anthropic Claude models Google Cloud Docs Explains that web search is a tool capability exposed through Anthropic's Messages API rather than a default behavior.

Data

The complete row-level dataset is published open and ungated under CC BY 4.0. Every number in this study can be recomputed from it.