Questions about this study

It tested whether the brand ChatGPT names first when answering a buyer question stays first if you ask the same question again later. Looking at 172,624 paired answers across 172 categories, the first-named brand changed in 74.6% of pairs, and stayed the same in only 25.4%. Every pair compared answers from the same model version, same web search setting, and same prompt version, so the brand swap was not caused by OpenAI updating the model underneath. Being named first once tells you very little about who gets named first next time.

A unit is one earlier answer and one later answer to the identical buyer question, compared to see if the first-named brand is the same in both. Because a question can be asked at multiple time gaps, it can contribute more than one pair to the dataset, so the 172,624 count is a count of comparisons, not a count of distinct questions. This matters for interpretation: a single volatile question asked repeatedly could contribute several unstable pairs, which is why the study also checks that no one category dominates the results.

Barely, and not in a clean increasing line. Within a week the leader changes 76.1% of the time, in the one to four week window it is actually highest at 77%, and past four weeks it settles at 74.1%. If this were purely markets genuinely shifting over time, you would expect instability to climb steadily with the gap. Instead the near-flat pattern, with a slight bump in the middle segment, suggests something closer to a constant background churn rather than accumulating real-world change.

That is a real possibility the study cannot rule out, but the timing pattern argues against it being the main driver. Real market share shifts should show up more as the gap between questions grows, yet instability within a week (76.1%) is close to instability past four weeks (74.1%). A week is rarely enough time for a brand to genuinely overtake a category leader. The honest limitation is that this measurement cannot separate three things that look identical in the data: sampling variance inside the model, changes in what the web search retrieves, and actual market movement.

No. The study checked instability separately across 143 categories, and the share of pairs with a different leader ranged from 34.3% up to 92.8%, with a median of 77.3%. On top of that, no single category supplied an outsized chunk of the data, the largest one accounts for only 1.7% of all 172,624 units. So while some categories are noticeably more stable than others, the general pattern of frequent change is broad, not an artifact of one heavily-sampled market.

No, that specific explanation is controlled for. Every pair in this dataset was matched to hold the model version, the web search setting, and the prompt version fixed between the earlier and later answer. If a pair involved a model update, it would not have been compared as a matched pair at all. So the instability measured here reflects variation within a constant model and settings, not the more obvious story of the underlying system changing versions.

Not directly. This study used a fixed, API-style instrument with controlled settings so every comparison was apples to apples, which is also why it cannot speak to the consumer ChatGPT app. That app can draw on conversation history and account-level personalization, both of which could make answers more consistent for a given user, or could introduce their own sources of variation. The finding here is best read as a property of the underlying model and search behavior, not a claim about what any specific logged-in user would see.

Stop treating a single first-place mention as a stable asset to report to leadership, and start tracking it the way you'd track any noisy metric, with repeated measurement over time. Given that 74.6% of pairs showed a different leader, one favorable screenshot is close to meaningless on its own. A more honest approach is to sample the same question repeatedly across a window, look at the rate at which your brand holds first place rather than a single instance of it, and expect that rate to vary by category since the spread across categories ran from 34.3% to 92.8%.

No rows were excluded, 0 out of 172,624 total pairs. The study did apply 3 mitigations during construction, which is a separate step, these are methodological controls such as fixing model version, web search setting, and prompt version across each pair, rather than data cleaning that removed observations. So the 74.6% figure reflects the full set of qualifying pairs collected in the window, not a filtered subset.

These are the two possible outcomes for each pair of answers. 'The same brand still leads' means the brand named first in the earlier answer was also named first in the later answer to the identical question, under identical model, search setting, and prompt version. 'A different brand leads' means the first-named brand changed between the two answers. Across the full dataset these split 25.4% to 74.6%, and that split is the central measurement the rest of the study's breakdowns, by time gap and by category, are built from.