Methodology

Engines and exact versions

A study that does not name the model version it ran against is not reproducible, because the answer changes when the model does.

Engine	Model version

Run window, UTC: [object Object].

Method

The unit of analysis is a pair of answers to the same underlying buyer question. Each question came from Rankxa's industry question pool, spread across 82 categories, and none of the questions name a brand, so whatever brand appears first in an answer is the model's own choice, not an echo of the prompt.

Each question was sent to Claude Sonnet 5 through Anthropic's developer API four times: twice with no tools attached, so the model answers only from what it already knows, and twice with the web search tool turned on, so the model can issue queries and read pages before answering. Four answers per question let you form three kinds of pairs: a memory answer against another memory answer, a search answer against another search answer, and a memory answer against a search answer. Across the full dataset this yields 7,996 pairs spread over 7,996 rows, and no rows were dropped from the analysis (0 excluded).

For each pair, the question asked is simple to state and hard to fake: does the brand named first in one answer still come first in the other, or does it appear at all. Every pair gets sorted into exactly one of four classes:

Class	What it means
The same brand comes first	Both answers agree on the top brand
A different brand comes first but the first is still named	Top brand changed, but the original pick still appears somewhere in the other answer
The first brand is missing from the other answer	The original top pick does not appear at all in the other answer
One of the two answers names no brand	At least one answer in the pair names no brand, so first-place cannot be compared

This four-way split matters because it separates reordering from disappearance. A brand that drops from first to second is a different kind of event than a brand that vanishes outright, and lumping them together would hide which one search actually causes. No group supplied more than 1.3% of all units, so the headline numbers are not an artifact of one oversized category dominating the pool.

Limitations we volunteer

Written by us, before anyone else found them.

Terms used in this study

Message Batches service

An Anthropic API feature for submitting many requests together and getting results back as a batch, used here so all four answers to a question were generated under identical conditions on the same day.

Web search tool

An optional capability Anthropic's API can give the model, letting it issue search queries, here capped at two per answer, and read results before composing its response; without it the model answers only from what it learned during training.

Pair

The unit of analysis: two of the four answers to the same question compared against each other, for example a memory answer against a search answer.

Segment

Which kind of pair is being looked at: memory against memory, search against search, or memory against search, the first two serving as control comparisons.

Class

The four possible outcomes for what happened to the brand named first in the first answer of a pair, from keeping that position to disappearing entirely.

Group

The category a question belongs to, used to check whether a result holds broadly or is driven by a few markets.

Rubric (brand reading)

The fixed rule a separate model, Claude Haiku, used to decide what counts as a named brand in an answer: options, recommendations, and notable players count, while review sites, directories, and cited sources do not.

First brand missing

The outcome where the brand named first in one answer of a pair does not appear anywhere in the other answer, not even further down the list.

Local service question

A buyer question that names a city, common in this question pool, which means a large share of results describe local services rather than national software categories.

References

Sources this study reads against. Every link was fetched and confirmed reachable at publication.

1. Self-Consistency Improves Chain of Thought Reasoning in Language Models arXiv, 2022 Foundational method for measuring answer stability across repeated generations via majority vote, the baseline notion of consistency our memory-against-memory segment is closest to.
2. FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation arXiv, 2023 Standard benchmark showing search augmentation changes model correctness and hallucination rates, and that retrieved-evidence order affects the answer; the closest prior measurement of how search changes LLM output, though it scores correctness rather than brand-identity consistency.
3. Architectural Vulnerability and Reliability Challenges in AI Text Annotation: A Survey-Inspired Framework with Independent Probability Assessment arXiv, 2026 Defines and reports 'flip rate' as the share of instances where a model's top label changes under an intervention, the methodological template we adapt for brand-identity flips.
4. Same Agent, Different Answers arXiv, 2026 Independent study documenting answer flips in a retrieval-augmented QA system after an index change, holding model and prompt fixed, with a small reported flip count; closest methodological analogue to our pairing design.
5. SycEval: Evaluating LLM Sycophancy arXiv, 2025 Quantifies sycophantic answer-switching specifically in Claude-Sonnet alongside ChatGPT-4o and Gemini-1.5-Pro, giving a directly comparable prior figure for how often Claude revises an answer.
6. How You Ask Matters! Adaptive RAG Robustness to Query Variations arXiv, 2026 Shows flip rates stay high even at near-deterministic decoding temperature, ruling out sampling noise as the sole explanation for answer instability, a confound our fixed-temperature design must also address.
7. AI Sycophancy: Why Chatbots Agree With You - IEEE Spectrum IEEE Spectrum, 2025 Science-journalism synthesis of cross-lab replication finding models from multiple labs more sycophantic than crowdsourced baselines, used to show the phenomenon is not Claude-specific.
8. Sycophancy to subterfuge: Investigating reward tampering in language models Anthropic, 2024 Anthropic's own research page on related answer-revision behavior under pressure, cited for context on why first answers are not always stable commitments.
9. Tool use with Claude - Claude Platform Docs Anthropic, 2026 Describes the server-side mechanics of the web search tool as a citation-bearing retrieval step, the same configuration varied in our memory-versus-search pairs.
10. Web search tool - Claude Platform Docs Anthropic, 2026 Official spec for how Claude decides to search and how results are loaded into context; establishes that the practice under study (toggling web search on or off) is a documented, user-controllable setting.
11. Tool reference - Claude Platform Docs Anthropic, 2026 Version history of the web_search tool definition, relevant to reproducing this measurement on later Claude releases.
12. Towards Understanding Sycophancy in Language Models arXiv / alphaXiv, 2023 Seminal Anthropic study documenting models reversing a correct answer once a user expresses doubt, the closest prior account of why a model might abandon a first pick.
13. The Latest on Click-Through Rate by SERP Position Bruce Clay, 2025 Aggregates multiple CTR-by-position studies establishing the human first-pick-is-sticky intuition our finding contradicts in the AI-answer setting.
14. SERP CTR by Position & Feature: 2026 Benchmarks Outerbox Design, 2026 Confirms the consistent shape of the position-one click advantage across datasets, used as the reader's existing mental model of 'first pick' behavior.

Data

The complete row-level dataset is published open and ungated under CC BY 4.0. Every number in this study can be recomputed from it.