How often does the first-named brand disappear entirely from the second answer?

Across the full set of paired answers in this study, four outcomes were possible when checking whether a brand named first in one Claude answer also appeared in a paired answer to the same question: the same brand comes first again, a different brand comes first but the original one is still mentioned somewhere, one of the two answers names no brand at all, or the first brand is missing from the other answer entirely. That last category was the largest of the four, covering 31.4% of all 7,996 pairs, built from 7,996 rows with none excluded.

This is a stronger finding than simple reordering. A brand dropping from first place to second or third place is a ranking problem. A brand disappearing from the answer entirely is a visibility problem, and a sharper one, because there is no partial credit for a brand a user never reads. Compare this to the 18.5% of pairs where the original brand was demoted but still mentioned, a softer outcome where the brand at least stayed in the conversation. The full-disappearance outcome was nearly double that rate.

The rate of full disappearance was not uniform across the three pair types this study measured. It was lowest when both answers came from memory alone, at 17.4%, similar when both came from web search, at 27.1%, and sharply higher when one answer came from memory and the other from web search, at 40.6%. That last figure, drawn from 3,998 pairs, is the single largest number in this entire study broken down by segment, and it says something specific: a brand that Claude's trained knowledge favors has a meaningfully worse chance of surviving contact with a live web search answer than it has of surviving a second draw from memory alone.

It is worth being honest about what this class does and does not capture. A brand missing from the second answer does not necessarily mean the model ranks the brand poorly, or has negative information about it. It could mean the model's memory-side training data covers the brand but its web search results for that particular query, on that day, surfaced a different set of pages, pulling in different candidate brands entirely. It could also mean the opposite: that the brand is well covered online but thin in the model's trained weights, so memory answers skip it while search answers would, in a different pairing, surface it. This study's design, checking only whether the first-named brand reappears, cannot tell these two situations apart. What it can say is how often the gap opens up, not why in any individual case.

The spread across question categories also deserves a mention here, since it bears on how worried any single company should be. Across the 80 categories measured, the share of pairs in this missing-brand class ranged from 12% at the low end to 59% at the high end, with a median of 31%. Even the best-case category still saw a meaningful share of brands disappear between paired answers, which suggests this is a property of how the model handles paired memory and search queries generally, not an artifact of one unusual question category pulling the average up.