Why ChatGPT recommends different brands every time — and why there's no “ChatGPT ranking”

If you've ever checked whether ChatGPT recommends a client, refreshed, and got a different answer, nothing went wrong. That's how these systems work. ChatGPT doesn't look up a stored ranking; it writes a new answer every time, and several things change from one answer to the next. The result: there is no stable "position in ChatGPT" to track. There is a stable frequency — and that's what to measure.

What the research shows

In January 2026, SparkToro (Rand Fishkin) and Gumshoe.ai (Patrick O'Donnell) published one of the largest public tests of this. Around 600 volunteers ran 12 prompts through ChatGPT, Claude and Google's AI — nearly 3,000 responses, with each prompt run roughly 60–100 times. The findings:

• ChatGPT and Google's AI returned the same list of brands less than once in a hundred runs; Claude was only slightly more consistent.
• The same list in the same order came up roughly once in a thousand.
• How often a brand appeared was far more stable. One cancer hospital appeared in 69 of 71 ChatGPT answers (97%) but was the first name in only 25 of them.

The authors' recommendation: treat any single AI list as unreliable, ask many times, and use visibility percentage — not ranking position — as the metric. (Gumshoe sells AI tracking; the authors disclose it.)

Why answers change

1. Generation is probabilistic

Language models produce text one piece at a time, choosing among likely continuations. Unless a system is configured to always pick the single most likely option, small differences early in an answer — which brand is named first — change everything that follows.

2. Search results differ between runs

When ChatGPT searches the web before answering, it runs its own search queries, which can differ from run to run, and reads a selection of the results. A different set of pages read means a different set of brands available to recommend.

3. The person asking

The consumer app can use location, language, conversation history and memory. Two people asking the "same" question are rarely asking it in the same context. The SparkToro team also found that people phrase the same need very differently — human-written prompts for the same intent were barely similar to each other.

4. The product changes

Models are updated, search integrations change, and instructions behind the app are adjusted. An answer from March and one from September may come from meaningfully different systems.

5. API and app aren't identical

Tracking tools either call OpenAI's API with web search, or collect answers from the consumer app. The app adds instructions and features of its own, and developers have reported that API web search results can differ from the app's. Neither method reproduces what one particular person sees; both can measure patterns.

What this means for reporting

• Don't report a rank. "#3 in ChatGPT" describes a single answer. Clients will check, see something different, and lose trust in the report.
• Don't report from screenshots. A screenshot is one draw. It's fine as an illustration, not as evidence of change.
• Do report frequency over many answers. "Mentioned in 71 of 240 answers this month, up from 54" is honest, comparable over time, and survives a client checking for themselves.
• Do keep the evidence. Store the answers behind each number so you can show what the engines actually said.

How to explain it to a client

A version that works in meetings:

Then show the numbers with counts, and open two or three answers. Clients understand frequency immediately; it's the screenshot that confuses them.

How many answers is enough?

More answers make a rate more reliable, and a change between two months more believable. As a rough guide, a few dozen answers per engine per month shows the pattern; a few hundred across engines lets you take moderate month-to-month changes seriously. The details are in how often to track AI visibility.

Ranxii is built on this principle: no rankings, no blended scores. Every prompt is asked on every engine every week, every answer is stored, and every number is shown as "X of Y answers" with the answers one click away. The method is described in our guide to measuring AI visibility.

Sources

• SparkToro: AIs are highly inconsistent when recommending brands or products (January 2026)
• SparkToro: Why everyone prompts AI differently
• OpenAI Developer Community: API web search vs ChatGPT — developer reports of differing results