Turn on web search and Claude rarely keeps its first pick

Why this matters

If you run marketing or SEO for a brand, you have probably already asked someone on your team whether turning on an AI model's web search tool helps or hurts your odds of being mentioned, and whether it helps the same way every time. That second question, the consistency question, is the one this study is built to answer.

Claude Sonnet 5, like other large language models offered through a developer API, can answer a buyer question two different ways. Left alone, it answers from what it already learned during training, a fixed and dated body of text. With its web search tool turned on, it can issue live queries and read back current pages before answering. Anthropic documents this as a tool the model may call during a conversation, not a separate product, so the same question can be put to the same model with the tool off or on and the two answers compared directly.

The practical stakes are straightforward. A brand's AI visibility strategy usually assumes some baseline of stability, that if a model names you first today for a given query, it will tend to keep naming you first tomorrow, with search just adding more current detail. This study tests whether that assumption survives contact with search. It also tests the narrower and more common claim, that an answer generated with search is basically the same answer with citations attached. Both claims are testable because the same question was sent to the same model twice with no tools and twice with search enabled, letting us compare answer to answer rather than guess from a single run.

The audience for this is anyone making decisions based on a single AI answer about their category: a brand tracking mentions, an agency reporting on share of voice, a founder who ran a prompt once and took the result as settled. The finding below says that single run is a much weaker signal than it looks, and it says specifically where the instability comes from, memory alone, search alone, or the mismatch between the two.

How the measurement works

The unit of analysis is a pair of answers to the same underlying buyer question. Each question came from Rankxa's industry question pool, spread across 82 categories, and none of the questions name a brand, so whatever brand appears first in an answer is the model's own choice, not an echo of the prompt.

Each question was sent to Claude Sonnet 5 through Anthropic's developer API four times: twice with no tools attached, so the model answers only from what it already knows, and twice with the web search tool turned on, so the model can issue queries and read pages before answering. Four answers per question let you form three kinds of pairs: a memory answer against another memory answer, a search answer against another search answer, and a memory answer against a search answer. Across the full dataset this yields 7,996 pairs spread over 7,996 rows, and no rows were dropped from the analysis (0 excluded).

For each pair, the question asked is simple to state and hard to fake: does the brand named first in one answer still come first in the other, or does it appear at all. Every pair gets sorted into exactly one of four classes:

This four-way split matters because it separates reordering from disappearance. A brand that drops from first to second is a different kind of event than a brand that vanishes outright, and lumping them together would hide which one search actually causes. No group supplied more than 1.3% of all units, so the headline numbers are not an artifact of one oversized category dominating the pool.

The headline result, and its limits

Across all 7,996 pairs, sorted into 4 classes, the outcomes were:

The single largest outcome, by a wide margin, is the first brand disappearing entirely. Add in the pairs where one answer names no brand at all, and roughly half of all pairs provide no basis for claiming the model kept its pick. Only 23.2% of pairs show the same brand coming first both times.

What this does not mean: it does not mean Claude is random, and it does not mean the model has no opinion about which brand is best. A brand missing from one answer in a pair may still be a reasonable answer to the question, just a different reasonable answer, and the model may be responding to real differences in what a live search turns up versus what it recalls from training. The class structure also cannot tell you whether the brand that replaces the original pick is better or worse for the asker, only that it is different.

What it does mean: anyone treating a single AI answer, with or without search, as a stable reading of brand position is making a claim the data does not support. The right comparison is not one answer, but the behavior of a pair, and here the pair agrees less than a quarter of the time. The next section asks whether this instability is a property of certain question types or of the measurement as a whole.

Where the instability comes from: memory versus search

The headline number blends three different comparisons, and they do not behave alike.

Two runs from memory alone agree on the first brand 35.8% of the time, the most stable of the three pairings. Two runs with search turned on both times agree 29.4% of the time, a bit lower but in the same range. The pairing that falls apart is memory against search: agreement drops to 13.9%, and the first brand goes missing entirely in 40.6% of those pairs, the highest missing-rate of any segment.

This points to a specific mechanism rather than generic noise. The model is reasonably consistent with itself when it answers the same way twice, from memory or from search. What breaks the pick is switching modes mid-comparison, because a live search surfaces current pages, recent reviews, and recency-weighted content that a training-data answer simply does not have access to, and the two source materials can lead to different top picks. That also means the common assumption, that turning on search just adds citations to the same underlying answer, does not hold here: search is changing which brand comes first, not decorating the same choice.

For a brand, this segment breakdown is the more useful number than the pooled headline. If your concern is specifically how a search-enabled answer compares to what the model would have said anyway, the relevant figure is the memory-against-search row, where the first pick survives only 13.9% of the time, not the blended figure that includes same-mode pairs.

Is this concentrated or everywhere?

A pooled percentage can hide a lot. If the instability were driven by a handful of unusual question categories, the fix would be narrow, know which categories to watch. If it is spread across the board, the fix has to be structural.

The dataset lets us check this because every pair carries a category, and the four classes were tabulated separately across 80 groups for each class. Here is the range:

No class sits at a single fixed value across groups, every one of them has real spread, and that spread is wide. 'The same brand comes first' ranges from a low where almost no category keeps its pick to a high where nearly half do. 'First brand missing' never drops below a meaningful floor in any group, meaning even the most stable categories still lose the original pick in a substantial minority of pairs.

The practical reading: the headline figures describe the pooled population honestly, this is not an artifact of one or two outlier categories dragging the average, since the instability shows up with a non-trivial floor in every group measured. But the position within that range varies enough that a brand in a low-volatility category is in a genuinely different situation than one in a high-volatility category. Knowing your own category's number, not just the pooled figure, is the actual decision-relevant piece of information, and that number is what the published dataset lets you look up.

What to do differently on Monday

None of this is an argument against using AI search features, or against tracking brand mentions in AI answers. It is an argument against treating any single answer, from any single mode, as a stable reading of where your brand stands.

A few concrete changes follow directly from the measurement:

Stop reporting a single screenshot as your AI visibility position. If the same question put to the same model twice can disagree about the top brand in roughly three out of every four pairs, one answer is a sample of one from a noisy process, not a measurement. Any report that shows "we are mentioned first by Claude" based on one query is showing you a coin flip, not a trend.

Separate your tracking by mode. Because memory-only pairs and search-only pairs are each more internally consistent than memory-against-search pairs, a tracking setup that mixes modes without labeling them will read as more volatile than either mode actually is on its own. If you want to know whether you are losing ground specifically because of what web search surfaces, run the memory-vs-search comparison deliberately rather than inferring it from mixed results.

Check your own category's spread before trusting the pooled number. The headline figures are a fair description of the whole pool, but individual categories range widely around them. A category near the low end for "same brand comes first" needs a different monitoring cadence than one near the high end.

Treat a disappearance differently from a demotion. The class structure here deliberately separates a brand dropping to second place, still named, from a brand vanishing from the answer entirely. These call for different responses: a demotion suggests a ranking or relevance problem, a disappearance suggests the live content search surfaces does not mention you at all, which is a content and citation gap, not a ranking one.

Re-run before you act. If a single answer currently shows a competitor ahead of you, the first useful step is not a strategy change, it is a repeat run to see whether the result holds. Given the base rates here, a one-off result has a real chance of simply being the pair disagreeing, not a change in the underlying landscape.

Reading the dataset yourself

The published dataset is organized around the same unit used throughout this study, the pair, so the fastest way to get oriented is to find the row count and confirm it matches what is reported here: 7,996 rows, 7,996 units, 82 categories, and 3 segment types (memory-vs-memory, search-vs-search, memory-vs-search). No rows were excluded (0), so the row count and the unit count should match exactly what you see in any full export.

Each row carries a category label, a segment label, and a class label drawn from the four-way scheme described above. To reproduce the headline table, group by class alone and divide by 7,996. To reproduce the segment table, group by segment first, then by class within each segment, dividing by that segment's own count rather than the pooled total, since the three segments are not equal in size: memory-vs-search alone accounts for 3,998 of the 7,996 total pairs, roughly twice either same-mode segment.

To check concentration, group by category and class together and look at the distribution of percentages within each class across all 82 groups, the same calculation behind the min-median-max table above. A category with an unusually low or high rate for any one class is worth a second look before you use the pooled figure to describe that category specifically. With an average of 97.51 units per group, most individual categories have enough pairs to support a percentage, but it is still a smaller denominator than the pooled total, so treat single-category figures as noisier than the headline ones and say so if you quote them.

No adjustments were applied to the classification before publication (0 mitigations applied), so the rates above are the raw tabulation, not a corrected or smoothed version. That is a deliberate choice: it means the numbers describe exactly what the four runs produced, with no attempt to filter out cases that looked like edge cases or classification ambiguity. If you build your own classifier on top of this kind of pairwise comparison, the categories worth stress-testing first are the ones with the widest spread in the table above, since that is where a borderline call between "different brand, still named" and "first brand missing" is most likely to flip the result.