Someone checks whether ChatGPT recommends their business. It does not. They panic, or they book an agency. A week later they check again and there they are, third on the list. Nothing changed on their website.
We wanted to know how much of that is real movement and how much is noise. So we measured it.
On 22 August 2026 we ran 20 Australian buying searches through Google AI Mode twice each, seconds apart, with identical settings. Same query, same location, same device, same moment.
The two answers shared only 24% of their sources on average. Not one of the 20 pairs was word for word the same.
That is the same question, asked twice, thirty seconds apart. Before anyone optimises anything.
Key findings
- Average source overlap between two identical searches: 24%.
- 0 of 20 pairs returned identical text.
- Best case was 44% overlap. Worst case was 0%.
- The number of sources cited swung wildly. One search returned 7 sources on the first run and 22 on the second.
- One search returned no sources at all on the first run and 22 on the second.
What the pairs looked like
| Search | Sources run 1 | Sources run 2 | Shared | Overlap |
|---|---|---|---|---|
| best physiotherapist sydney | 8 | 5 | 4 | 44% |
| emergency plumber adelaide | 8 | 3 | 3 | 38% |
| how much does invisalign cost australia | 13 | 14 | 7 | 35% |
| best personal injury lawyer sydney | 9 | 11 | 5 | 33% |
| how much does seo cost in australia | 13 | 26 | 9 | 30% |
| best electrician perth | 9 | 6 | 3 | 25% |
| best dentist melbourne | 7 | 14 | 4 | 24% |
| custom home builder brisbane | 7 | 22 | 4 | 16% |
| best accountant for small business australia | 10 | 15 | 3 | 14% |
| best removalists melbourne | 12 | 7 | 2 | 12% |
| best seo agency australia | 0 | 22 | 0 | 0% |
Look at “custom home builder brisbane”. Seven sources, then twenty-two. Four in common. If you were one of the businesses that appeared in only one of those runs, your visibility that day was a coin toss.
Why this happens
It is not a bug and it is not personalisation. We used the same location and no account.
Two things are going on.
The model picks its own searches. Before writing an answer, these systems run their own background queries. We could see ChatGPT’s in our data, and they varied between runs. Different background searches return different pages, and different pages produce a different shortlist.
The writing step is not deterministic. Language models sample from probabilities. Two runs over the same source material still produce different sentences, and a different sentence can mean a different business named.
So there is no fixed “position” to hold. There is a probability that you get named, and it is somewhere between zero and one.
What this means for anyone checking their AI visibility
Three practical consequences.
One check tells you almost nothing. If you ask ChatGPT once and you are not there, that is one draw from a random process. Ask five times and count. If you appear in four of five, you are strong. If you appear in zero of five, that is real.
A single before-and-after screenshot is not evidence. If an agency shows you a screenshot from before the work and one from after, that comparison is inside the noise. Ours moved 24% on its own with no work at all. Ask for a rate across repeated checks over time instead.
Being cited once is not the goal. Being reliably citable is. You cannot influence which pages the model happens to fetch on a given run. What you can influence is whether, once it lands on you, there is a fact worth quoting. That is the part that carries across every run.
How to measure it properly
- Pick 10 to 20 real customer questions. The ones people actually type when they are ready to hire, not your keyword list.
- Run each one at least five times. Record whether you were named, not where.
- Report a rate. “Named in 12 of 50 runs” is a number that means something. “We appear in ChatGPT” is not.
- Repeat monthly. Anything shorter and you are measuring randomness.
- Track your competitors in the same runs. The useful comparison is your rate against theirs on the same day.
None of this is complicated. It is just more sampling than people are used to doing, because ranking checks never needed it.
The thing that does not move
One finding held steady. The kind of source barely changed even when the specific sources did. Business websites were 82% of citations across the first runs and 84% across the second. Directories, forums and social sat within two points of each other both times.
So the target is stable even though the results are not. The systems consistently want a business website with something quotable on it. Which sites they happen to grab on a given run is noisy. What they are looking for is not.
That is the useful thing here. You cannot control the draw. You can control whether you are in the pool.
We covered what belongs on the page in our study of 500 Australian homepages, and the wider AI Mode findings in this study of 40 Australian searches. If you would rather have the sampling done for you across your real customer questions, that is what our AI SEO work tracks.
Common questions
Why does ChatGPT give different answers to the same question?
Two reasons. It runs its own background searches before answering, and those vary between runs. And the writing step samples from probabilities rather than repeating a fixed output. In our test, two identical Google AI Mode searches shared only 24% of their sources.
Does that mean AI visibility cannot be tracked?
It can, but as a rate rather than a position. Run each question several times and record how often you are named. One check is a single sample from a random process.
How many times should I check?
At least five runs per question, and at least ten questions. Fewer than that and normal variation will look like a result.
My business appeared last week and not this week. Did I get penalised?
Almost certainly not. Our runs moved that much within seconds, with nothing changed. Look at your rate across repeated checks before assuming anything went wrong.
Can an agency guarantee I will appear in ChatGPT?
No, and it is worth being wary of anyone who does. The output varies run to run on its own. What can be improved is how often you are named, not whether you are named on a given day.
What actually improves the odds?
Being fetchable by AI crawlers, and having specific checkable facts in plain text on your pages. Those are what the systems reach for regardless of which pages they happen to pull on a given run.
Is Google AI Mode more stable than ChatGPT?
We measured repeat runs on AI Mode only, so we cannot rank them. What we can say is that AI Mode drew 40% of its sources from Google’s own top 10, the highest of any AI surface we tested, so it stays closer to normal search results than the others.
Method: 20 Australian commercial and research searches were each run twice through the Google AI Mode SERP API on 22 August 2026, with identical location, language and device settings and no user account. Overlap is the proportion of shared domains between the two runs of a pair, measured against the combined set of domains from both. Google’s own internal preview links were excluded.
