Ask ChatGPT the same buyer question twice and it names a different brand first about three times in four

Why this matters

If you run a brand that depends on being recommended rather than searched for, you want to know whether a good answer today is a good answer next month. ChatGPT is increasingly where buyer questions like "best project management tool for a small agency" or "most reliable water filter brand" get asked, and the brand named first in that answer gets a disproportionate share of attention, the same way the top organic search result does. The practical question behind this study is simple: if you win that first mention today, can you expect to still hold it later, or is the leaderboard reshuffling underneath you on its own, with no change in your product, your reviews, or your marketing?

This matters to three groups differently. Brand and marketing teams who are starting to treat "AI answer visibility" as a metric need to know whether that metric is stable enough to act on, the way a monthly ranking report is, or whether it moves so much between asks that a single snapshot is close to meaningless. Agencies selling AI-visibility monitoring need to know what they are actually promising a client: consistent tracking of something durable, or a running tally of noise. And researchers studying how generative answers behave over time need a baseline rate of churn before they can say whether any particular intervention, like a PR push or a changed product review, moved the needle at all.

The finding itself is blunt. Across 172,624 pairs of answers to the same buyer question, one earlier and one later, the brand named first changed in 74.6% of units. Only 25.4% of the time did the same brand hold its position. That is not a small amount of drift, it is the dominant outcome. Before drawing conclusions from it, though, it is worth being precise about what was actually measured, because "ChatGPT named a different brand" can mean several different things, and the study had to pick one and hold to it consistently.

How the measurement works

The unit of analysis is a pair, not a question and not a company. Each pair consists of two answers to the identical buyer question, asked at two points in time, with everything else held constant: the same underlying model, the same web-search setting, the same prompt wording and version. If any of those three things changed between the earlier and later ask, that pair was not used, because the point of the study is to isolate drift in the named brand from drift caused by OpenAI shipping a new model version or the researchers phrasing the question differently. A question that gets asked repeatedly over the measurement window contributes one pair for each earlier-later comparison, so a frequently-tracked question shows up as many units, not one. This is why the headline denominator is 172,624 pairs rather than a smaller count of distinct questions, and it is also why the finding is correctly described as being about units, not about companies or categories directly.

Each pair was classified into exactly one of 2 classes: either "a different brand leads" (the brand named first changed between the two asks) or "the same brand still leads" (it did not). There is no partial-credit or near-miss category, the comparison is strictly about whoever is named first. 0 rows were excluded from the final set of 172,624, and 3 mitigations were applied during data collection to control for known confounds such as model version drift, which is the main reason the classification scheme can be trusted as measuring leader-change rather than measurement artifacts.

Pairs were also grouped by category, 172 in total, and segmented by the time gap between the earlier and later ask, in three bands: within a week, one to four weeks, and over four weeks. The largest single category contributes 1.7% of all units, which is small enough that no one category is dictating the overall result. On average each group contributes 1,003.63 units, giving most categories a reasonably sized sample to compute their own rate from, a point that matters later when looking at how the result varies by group.

The headline result, and its limits

Across all 172,624 pairs, 74.6% were classified as "a different brand leads" and 25.4% as "the same brand still leads." In plain terms, ask ChatGPT the same buyer question twice, separated by anywhere from a few days to over a month, and roughly three times out of four the brand it names first will not be the same brand both times.

What this does mean: the identity of the first-named brand in a ChatGPT answer to a buyer question is, on the whole, unstable across repeated asking. A single answer captured once is a weak signal of what the model will say tomorrow or next month. If a company's AI-visibility strategy rests on having gotten the top mention in one spot-check, this result says that spot-check tells you less than it feels like it tells you.

What this does not mean: it does not mean the model is choosing brands at random, and it does not mean that nothing a company does affects its odds of being named. A change rate of 74.6% is consistent with a model that is weighing a shifting mix of recent web content, review scores, and phrasing variance each time it answers, not necessarily with pure randomness. It also does not mean that every brand has an equal chance of appearing first. A brand could still be named first far more often than chance across many pairs while still frequently trading that top spot with one or two close competitors, which would show up as a high change rate without meaning the result is arbitrary.

It also does not tell us why the leader changes. Two different mechanisms could produce the same topline number: the model's output could be genuinely sensitive to small, meaningless variations in sampling, or the underlying web content the model draws on could itself be changing fast enough, through new reviews, new articles, new rankings, that a different brand legitimately deserves the top spot by the time of the second ask. The study design, which holds model version, search setting, and prompt fixed, rules out the simplest false explanation, a version change. It cannot by itself distinguish real-world-driven change from sampling noise, a point taken up in the next section.

Is this everywhere, or is it a few loud categories

A change rate of 74.6% could arise two very different ways. It could be a property spread fairly evenly across almost every category, meaning any brand in almost any market faces this churn. Or it could be an average pulled upward by a smaller number of categories where leadership changes constantly, while most categories are actually fairly stable. These two situations call for different responses, so it is worth checking which one this is.

The time-gap segmentation gives a first check. If instability were really about real-world content catching up over time, categories with long gaps between asks should show more churn than ones asked again within days.

The rate barely moves across these bands, from 76.1% at the shortest gap to 77% at one-to-four weeks to 74.1% beyond four weeks. If slow, real-world content change, like new reviews accumulating, were the main driver, the within-a-week rate should sit noticeably below the over-four-weeks rate, since there is less time for the world to change. It does not. That points toward the instability being present even at very short timescales, which is more consistent with the model's answer being sensitive to something other than slow-moving real-world shifts, such as how it samples among several plausible-seeming brands at the moment of answering, than with a story where brands are genuinely and gradually trading places on their merits. This is suggestive, not conclusive, since a week is still enough time for some fast-moving content changes, like a new viral review, to occur.

What the spread across groups shows

The time-segment view says the instability isn't concentrated in the longer gaps. The category-level view asks a different question: is the overall 74.6% rate a fair description of nearly every one of the 172 categories, or is it an average that conceals a lot of variation between, say, a software category and a consumer-goods category?

Across the 143 categories with enough data to compute a group-level rate, the "a different brand leads" share ranged from a low of 34.3% to a high of 92.8%, with a median of 77.3%. That is a wide range. A category at the low end, near 34.3%, behaves almost like a stable market where the same brand usually keeps its lead. A category at the high end, near 92.8%, is one where the named leader is close to a coin flip (or worse) each time the question is asked again.

The median, 77.3%, sits close to the overall headline rate of 74.6%, which is the important check. It means the headline is not an artifact of a few extreme categories dragging up an otherwise-low average, it is a reasonable description of a typical category, since half of all categories sit at or above a rate nearly identical to the topline figure. That said, the gap between the minimum (34.3%) and maximum (92.8%) is large enough that no single company should assume its own category sits at the median without checking. A brand competing in a category near the low end of this range has real reason to treat a top mention as somewhat durable, and a brand in a category near the high end should treat it as closer to noise.

What the published category-level breakdown lets a reader do, and what this summary cannot, is look up where their own market actually falls on that range rather than relying on the population-wide average.

What to do differently on Monday

Given a rate like 74.6%, the practical shift is to stop treating a single ChatGPT answer as a measurement and start treating it as one draw from a distribution. Concretely:

Stop reporting single snapshots as status. A screenshot showing your brand named first, taken once, describes that moment and nothing else with any confidence. If someone on your team is using that screenshot in a board deck as evidence of standing, the honest caveat is that re-asking the same question could very plausibly name a competitor first, with no change in anything you did.

Track at the pair level, not the point level. The unit that carries information here is not "ChatGPT said we're first," it is "ChatGPT said we're first both times we asked, including after a gap." That means repeated, scheduled re-asking of the same prompts is necessary to say anything about trend rather than a single state. A monitoring setup that asks once a month and reports the result as "current ranking" is implicitly claiming a stability that this data does not support.

Check where your own category sits before reacting. Because the group-level rate ranges from 34.3% to 92.8%, the right level of concern about losing a top mention depends heavily on which end of that range a category falls on. A category near the low end means a lost top mention is more likely to reflect something real, worth investigating. A category near the high end means losing the mention is closer to the expected background rate and doesn't by itself indicate anything went wrong.

Do not assume short gaps are safe. Because the within-a-week rate is close to the over-four-weeks rate, there is no evidence that asking again sooner buys meaningfully more stability. Spacing out monitoring checks to save effort will not noticeably improve the apparent consistency of the result.

Treat a single win as a hypothesis, not a result. If a brand wants to know whether a specific action, a new comparison page, a run of reviews, actually improved its odds of being named, the comparison that matters is the named-first rate across many repeated asks before and after that action, benchmarked against this background churn rate, not a before-and-after pair of single answers.

Reading the published dataset yourself

The dataset underlying this study is organized at the level of the pair, so the first thing to check when reading it directly is what a single row represents: one earlier-later comparison for one specific question, with the model, web-search setting, and prompt version recorded alongside the classification. A row is not a brand, a company, or a category, it is one instance of asking the same question twice.

A few checks worth running before trusting any cut of the data:

• Denominator size. The overall result rests on 172,624 pairs across 172 categories, averaging 1,003.63 units per category. Before trusting a category-specific number, check its own unit count rather than assuming it matches the average, some categories will have far fewer pairs behind them than others.
• Segment, not just headline. The time-gap segmentation (within a week, one to four weeks, over four weeks) is where the dataset earns its keep, since it is the piece that lets you check whether an observed change rate is driven by elapsed time or is present even at short gaps.
• Confounders controlled versus not. The study fixed model version, web-search setting, and prompt version across each pair and applied 3 mitigations during collection; 0 rows were dropped as a result. If you are extending this analysis to your own questions, replicate those controls, since a model version change between two asks will masquerade as leader instability when it is really just a different model answering.
• No brand-identity layer. This dataset classifies whether the leader changed, not which brand gained or lost most often. A category could have a high change rate while one brand still wins the top spot more often than any other competitor across the full set of asks. Answering "who tends to win" requires a different cut of the same underlying answers, tallying named-first frequency per brand within a category, which is a separate analysis from the stability question asked here.

The single most informative thing a reader with a specific brand or category in mind can do is look up that category's own row in the group-level breakdown, rather than apply the 74.6% population figure uniformly, given how wide the spread from 34.3% to 92.8% actually is.