The most common piece of evidence in AI visibility work is a screenshot of ChatGPT naming a business, and it is close to worthless. Not because the answer is fake, but because the next run of the same question may not include the business at all. When we ran identical queries through Google AI Mode seconds apart with identical settings, the cited domains overlapped an average of 24 per cent. Not one of twenty pairs produced the same answer text. One query cited zero domains on the first run and 22 on the second.
That volatility is not a flaw you wait out. It is how these systems work, and it means AI visibility can only be measured the way you would measure anything probabilistic: as a rate across repeated trials. This article is the full measurement system, the one we run for clients, laid out so you can build it yourself in a spreadsheet for nothing.
The core idea: rates, not runs
A single AI answer is weather. What you want to measure is climate: across many runs of many buyer questions, how often is your business named, how often is your site cited as a source, and how are both moving over time. Three numbers, tracked monthly, against a baseline that never changes. Everything else in this system exists to make those three numbers trustworthy.
The discipline this requires is the part most businesses and, frankly, most agencies skip: the yardstick has to stay still. Same prompts, same engines, same method, every month. The moment you swap prompts mid program, add engines, or count differently, your trend line becomes fiction. We put fixed yardstick clauses in client agreements because the temptation to improve the panel is constant and always wrong.
Building the prompt panel
The panel is 20 to 25 questions a real buyer would ask, and its quality decides the value of everything downstream. Build it in four blocks.
Shortlist prompts, about ten. The best X in city questions, phrased the way customers actually phrase them, including the sloppy versions. Who should I get to do my switchboard upgrade in Brisbane. Best solar installer near Penrith. These are the money prompts, because assistants answer them with named shortlists and the shortlist is the whole game, as we showed in how AI recommends builders.
Consideration prompts, about six. Cost, worth it and comparison questions in your category. How much does a home battery cost installed. Is it worth using a broker for a first home loan. You will rarely be named in these, but your site can be cited as a source, and citation here builds the association that feeds shortlists.
Situation prompts, about five. The specific scenarios you most want to win. Best builder for a sloping block in the eastern suburbs. Accountant who understands ecommerce inventory. These reveal whether your specialisation pages are doing their job.
Brand prompts, two or three. Your business name directly: is X reputable, X reviews. What the assistant says when asked about you outright is worth knowing, and wrong facts here point straight at what needs publishing.
Write the panel once, argue about it once, then freeze it. If you must evolve it later, add a second panel alongside rather than editing the first, so the original trend line survives.
Running it: engines, cadence, recording
Engines. Run every prompt through ChatGPT with web search on, Google AI Mode, and Perplexity. They behave differently enough that they are effectively three channels: in our cross engine testing ChatGPT averaged 3.7 sources per answer with heavy register checking, AI Mode 8.2 with a business site skew, Perplexity 17.9 with a recency and comparison content bias. Track them separately. Perplexity moves first and is your leading indicator, AI Mode has the widest reach, ChatGPT is the hardest and most valuable to win, per our Perplexity playbook and our AI Mode playbook.
Cadence. Monthly, and run each prompt twice per engine within the measurement window, on different days. Doubling the runs is the cheapest way to dampen the volatility, and the difference between your two runs is itself useful data: it tells you whether your presence is anchored or lucky. A business named in both runs of a prompt has durable visibility; named in one of two, it is on the bubble, and the fix is more published facts and citations for that topic.
Recording. One spreadsheet row per prompt per engine per run: date, named yes or no, cited yes or no, who else was named, which sources were cited. The who else column matters more than it looks, because over months it maps your real competitive set as the assistants see it, which is routinely different from your competitive set as you see it, and it surfaces the directories and listicles doing the naming, which become your earned media targets.
The three metrics and what moves them
Naming rate: the percentage of shortlist and situation prompt runs where you are named. This is the headline number. It moves when your facts become verifiable, your register entries align, your reviews accumulate, and you appear on the third party sources assistants read. Expect it to move in months, not weeks.
Citation rate: the percentage of all runs where your site is cited as a source. This moves faster than naming, because it responds to content quality and structure directly. Perplexity citation rate is usually the first thing in the whole system to move, often within weeks of publishing answer shaped, dated content.
Fact accuracy: from your brand prompts, is what the assistants say about you correct. Score it simply: correct, minor errors, major errors. Wrong pricing, dead locations and outdated claims trace directly to what your own site and profiles say, and fixing the source usually fixes the answer within a crawl cycle or two. The six facts that drive this are laid out in our homepage study.
Set the baseline before any optimisation work starts, and resist the urge to skip it because the numbers are embarrassing. A baseline of zero is the most persuasive number in marketing: it makes every later month legible.
Tools versus spreadsheet
A paid AI visibility tracking market has appeared quickly, and some tools are genuinely useful: they automate the runs, increase the sample size, and remove the discipline problem of actually doing it monthly. Costs run from about $100 to $500 and up per month, and several general SEO platforms have bolted on AI tracking modules of varying depth, some of which we covered in our AI SEO tools roundup.
Our honest guidance: start with the spreadsheet. Two hours a month, zero dollars, and running it manually for a quarter teaches you how the answers behave in a way dashboards never do. Move to a tool when the manual routine is genuinely established and the limitation is sample size rather than discipline, because a tool automating a measurement habit you do not have automates nothing. The one thing never to accept, from a tool or an agency, is single run reporting presented as truth. Ask any vendor how they handle run to run volatility, and if the answer is a blank look, keep your spreadsheet.
Reading the results without fooling yourself
Three traps catch nearly everyone. Celebrating or panicking over one month: with a 25 prompt panel run twice across three engines, you have 150 data points, enough for a trend over a quarter but noisy month to month, so judge direction over three months. Chasing a competitor’s single appearance: they are subject to the same 24 per cent overlap you are, and their screenshot is as meaningless as yours. And moving the yardstick: every prompt edit resets your history, which is why the panel freezes at baseline.
What a working program looks like in the numbers, in order: Perplexity citation rate moves first, then AI Mode citations, then naming rates, then ChatGPT naming last, because its verification habits are the strictest. If nothing has moved anywhere by month three, the work is wrong, usually at the crawler access or published facts layer, and the fix sequence is the audit again.
The monthly report that keeps everyone honest
Measurement only changes behaviour if it gets read, so the last piece of the system is a one page monthly report format that owners actually look at. Ours has five lines, and the discipline is that it never grows.
Line one: naming rate this month versus baseline, by engine. Three numbers and three deltas. Line two: citation rate the same way. Line three: fact accuracy from the brand prompts, flagged only when something is wrong, with the wrong claim quoted so it can be traced to its source and fixed. Line four: the competitive note, which businesses appeared most in your prompts this month and whether the sources naming them changed. Line five: what shipped and what ships next, three bullets maximum, because a measurement report that cannot name the work it is measuring is theatre.
Two habits make the report worth its page. First, annotate events on the trend line: the month you fixed crawler access, published the pricing page, aligned the register entries. Attribution in this field is probabilistic, but a citation rate that bends the month after a specific change, across enough prompts, is as close to causation as this discipline offers, and the annotations are what let you see it. Second, report the boring months honestly. Flat is information: three flat months at the foundation stage means the foundation is wrong, while three flat months after strong gains usually just means volatility, and the difference is visible only because the yardstick never moved.
If an agency runs this for you, this report is what you should be receiving, and every number on it should trace to raw run data you can inspect. If you run it yourself, the report is also your discipline mechanism: the months you do not want to write it are reliably the months the routine was slipping. Either way, after six months you own something genuinely rare in this field, a defensible record of whether AI visibility work produced AI visibility, and that record is worth more than any individual tactic it measured.
Frequently asked questions
Why do AI answers change between identical queries?
Because generation is probabilistic and retrieval varies run to run. We measured 24 per cent average citation overlap between identical back to back queries, with zero identical answer texts in twenty pairs. This is normal operation, and it is why rates across repeated runs are the only meaningful measurement.
How many prompts do I need in a panel?
Twenty to 25 covers a single location service business: about ten shortlist prompts, six consideration, five situation, and two or three brand prompts. Run each twice per engine per month and you have around 150 monthly data points, enough for quarterly trend reading.
Which AI engines should I track?
ChatGPT with web search, Google AI Mode and Perplexity, tracked separately because they behave like different channels: 3.7, 8.2 and 17.9 average sources per answer respectively in our testing, with different source preferences. Perplexity is your early indicator, AI Mode your reach, ChatGPT your hardest win.
What is a good AI naming rate?
There is no universal benchmark yet, and anyone quoting one is inventing it. What matters is your movement against your own frozen baseline. Most businesses we baseline start between zero and 15 per cent on shortlist prompts, and a program that doubles the starting rate inside six months is working.
Do I need paid tools to track AI visibility?
No. A spreadsheet, two hours a month and discipline produce a trustworthy trend. Tools earn their fee later by automating runs and increasing sample size. Whatever you use, reject single run reporting: any measurement that cannot explain how it handles volatility is decoration.
How long before tracking shows improvement?
Citation rates, led by Perplexity, can move within weeks of publishing structured, dated, answer shaped content. Naming rates follow over two to four months as facts, registers and third party citations align. Flat numbers across all engines at month three mean a foundation problem, almost always crawler access or missing published facts.
