Methodology

Engines and exact versions

A study that does not name the model version it ran against is not reproducible, because the answer changes when the model does.

Engine	Model version

Run window, UTC: [object Object].

Method

The unit of analysis is a pair, not a question and not a company. Each pair consists of two answers to the identical buyer question, asked at two points in time, with everything else held constant: the same underlying model, the same web-search setting, the same prompt wording and version. If any of those three things changed between the earlier and later ask, that pair was not used, because the point of the study is to isolate drift in the named brand from drift caused by OpenAI shipping a new model version or the researchers phrasing the question differently. A question that gets asked repeatedly over the measurement window contributes one pair for each earlier-later comparison, so a frequently-tracked question shows up as many units, not one. This is why the headline denominator is 172,624 pairs rather than a smaller count of distinct questions, and it is also why the finding is correctly described as being about units, not about companies or categories directly.

Each pair was classified into exactly one of 2 classes: either "a different brand leads" (the brand named first changed between the two asks) or "the same brand still leads" (it did not). There is no partial-credit or near-miss category, the comparison is strictly about whoever is named first. 0 rows were excluded from the final set of 172,624, and 3 mitigations were applied during data collection to control for known confounds such as model version drift, which is the main reason the classification scheme can be trusted as measuring leader-change rather than measurement artifacts.

Pairs were also grouped by category, 172 in total, and segmented by the time gap between the earlier and later ask, in three bands: within a week, one to four weeks, and over four weeks. The largest single category contributes 1.7% of all units, which is small enough that no one category is dictating the overall result. On average each group contributes 1,003.63 units, giving most categories a reasonably sized sample to compute their own rate from, a point that matters later when looking at how the result varies by group.

Limitations we volunteer

Written by us, before anyone else found them.

Terms used in this study

unit

One comparison between an earlier and a later answer to the same buyer question. A question answered ten times produces nine units, one for each consecutive pair, so questions asked more often do not automatically dominate the results more than their actual number of comparisons warrants.

leader

The brand recorded in the lowest position number among the businesses named in an answer, meaning the one ChatGPT listed or implied first. When two businesses would tie, the business id is used to break the tie so the same underlying data always produces the same leader.

class

The outcome of one comparison: either the same brand still leads in the later answer, or a different brand leads. An answer where no leader could be determined contributes no unit at all, rather than being counted as a change.

segment

The time gap between the earlier and later answer in a pair, grouped into within a week, one to four weeks, and over four weeks. Segments let the study check whether instability grows with elapsed time, which would suggest real market movement, or stays flat, which would suggest the model itself is the source of the churn.

group

A product or service category, such as a type of software or service. Results are checked across groups so a single volatile category cannot be mistaken for a general pattern.

instrument

The specific combination of model version, web search setting, and prompt version used to generate an answer. Two answers are compared only when they share the same instrument, so an OpenAI model update is never counted as the brand ranking changing.

gpt-4o-mini / gpt-5.4-nano

The two model versions behind the API calls used in this study, the latter introduced with web search access from mid September 2026. Comparisons never mix the two.

API versus consumer ChatGPT app

This study queried OpenAI's API directly, a plain question-and-answer interface without login, memory, or personalisation. The consumer ChatGPT app that most buyers use can factor in account history, which this study cannot observe or rule out as a source of different behavior.

References

Sources this study reads against. Every link was fetched and confirmed reachable at publication.

1. Estimating LLM Consistency: A User Baseline vs Surrogate Metrics arXiv preprint / EMNLP 2025 (Wu, Lin, Akgul, Bauer, Carnegie Mellon) Methodological backbone for operationalizing LLM output consistency, including a large user study on perceived consistency.
2. Evaluating the Application of ChatGPT in Outpatient Triage Guidance: A Comparative Study arXiv Clinical-domain study measuring consistency of the top recommendation across repeated identical questions, the same metric class used here.
3. GEO: Generative Engine Optimization arXiv (Aggarwal, Murahari, Rajpurohit, Kalyan, Narasimhan, Deshpande) Foundational paper defining visibility measurement for brands and content in generative-engine answers.
4. Prompt Tracking Works, But Not as a One-Prompt Measurement System: Prompt Wording and Brand Visibility in AI Answer Engines SSRN (Ehrlinspiel, Landwehr, Rudzki / Peec AI) Formal working paper behind the Peec AI benchmark, with two studies isolating prompt wording as a source of brand-visibility variance.
5. The key to prompt tracking Peec AI Quantifies variance in branded versus non-branded prompt visibility, offering a plausible statistical explanation for high churn in unbranded buyer queries.
6. What Matters In An AI Prompt? Intent or Keywords? Search Engine Journal (Peec AI study) Large-scale companion benchmark testing whether rewording a prompt changes which brands surface, across five sectors and multiple answer engines.
7. Semrush Releases Expanded 2026 AI Visibility Index, Analyzing 126 Million AI Search Prompts Semrush Industry-standard large-scale benchmark showing month-over-month brand stability at the top-100 level, a contrasting frame to single-prompt-pair instability.
8. NEW Research: AIs are highly inconsistent when recommending brands or products; marketers should take care when tracking AI visibility SparkToro / Gumshoe.ai Closest direct precedent, framing the same question (how many different answers across repeated identical prompts) and citing the Carnegie Mellon consistency baseline as a model.
9. Evaluating the accuracy and consistency of ChatGPT for the management of type 2 diabetes: A cross-sectional study medRxiv Reinforces the general pattern that LLM responses to identical questions vary across users, attributed to the probabilistic nature of the models.
10. Evaluating the Appropriateness, Consistency, and Readability of ChatGPT in Critical Care Recommendations PMC (National Center for Biotechnology Information) Adjacent professional-domain benchmark finding both ChatGPT models lacked consistency across repeated identical clinical questions.
11. Same Question, Different Brands: How ChatGPT and Google's Gemini Recommend Different Companies for the Same Query BrightEdge Cross-engine disagreement baseline, quantifying how much two engines differ on the same query.
12. Why AI Engines Cite Different Sources but Recommend the Same Brands BrightEdge Contrasting finding used to check whether instability is a property of sourcing rather than brand naming itself.

Data

The complete row-level dataset is published open and ungated under CC BY 4.0. Every number in this study can be recomputed from it.