Does Claude keep its first pick when you ask it twice?

If you ask an AI model a buyer question twice, hoping to confirm that the brand it recommends is stable, how often do you actually get the same answer?

This study paired up repeated answers from Claude Sonnet 5 to the same buyer questions, sent through Anthropic's developer API, and checked whether the brand named first in one answer also came first in the paired answer. The question pool came from Rankxa's industry question pool, spread across 82 categories, with no brand named in the questions themselves. In total, 7,996 pairs were built from 7,996 rows, and none were excluded from the analysis.

The result: the same brand came first in both answers of a pair only 23.2% of the time. That is the smallest of the four possible outcomes the study tracked. The most common outcome, by a wide margin, was that the brand named first in one answer was missing entirely from the other answer, which happened in 31.4% of pairs. A further 26.9% of pairs saw one of the two answers decline to name any brand at all, which on its own rules out a repeat. And in 18.5% of pairs, the original brand was still mentioned somewhere in the second answer, just not in the lead position.

Add those last three together and you get the inverse of the headline number: something other than a clean repeat happened in roughly three out of every four pairs. That is the core finding. A single answer from Claude is a sample from a distribution, not a fixed lookup. Ask once, and you have one draw. Ask again, and you are not confirming the first draw, you are taking a second, largely independent one.

Why would a language model behave this way? Two mechanisms are plausible, and this study design cannot fully separate them. One is genuine sampling variance: the model generates text probabilistically, and even with the same prompt, different token paths can lead to different brand choices, especially when several brands are roughly tied in the model's internal ranking. The other is context sensitivity: small differences in how the request is processed, or in what the model retrieves when search is involved, could push it toward a different brand each time. The paired design here, two memory-only answers and two web-search answers per question, lets later findings on this study separate these two mechanisms, because memory-against-memory pairs isolate sampling variance while memory-against-search pairs add retrieval differences on top. That comparison is treated on its own page. Here it's worth simply naming the overall rate.

The spread across the 82 question categories is also worth noting. The share of pairs landing in the same-brand-first class ranged from a minimum of 5% to a maximum of 49% across the 80 categories measured, with a median of 21%. So even the categories most favorable to repeat recommendations still saw repeats well under half the time, and some categories saw almost none. No category size dominates this result: the largest single question group accounted for only 1.3% of all pairs, so the overall rate is not being driven by one oversized or unusual category.

What should a company do with this. If you are tracking whether an AI assistant recommends your brand, a single check is close to meaningless, because the most likely outcome of asking twice is not agreement. You would need repeated sampling, ideally across both memory and search conditions, before treating any one answer as evidence of a stable position. The practical floor to remember is this: even in the most favorable comparison this study measured, memory against memory, the first brand still only came first again in 35.8% of pairs, a number covered in more detail elsewhere in this study.