One screenshot from ChatGPT is not a measurement. A real AI visibility report runs a fixed set of prompts for your trade and city, repeats some of them, logs the date and login state, and reports how often you were named, assistant by assistant, as “named in X of N.” If a report cannot show you the prompts and the repeats, you cannot tell progress from noise.
Search Console now has a Generative AI performance report that shows whether your pages appear in Google’s AI features. It is first-party data, so read it next to any prompt test. And Google’s guide to optimizing for generative AI features says no special AI file or markup is needed to appear, so a report that sells one as the fix is selling the wrong thing.
This page is for owners who were just sent an AI visibility report, or are about to pay for one. Below is a real test we ran on October 14, 2026, laid out the way a monthly report should look, with notes on what each part means and what to be suspicious of. If you want to know what gets a business named in the first place, that is a different question, covered in our guide to AI search visibility for local service businesses.
Why One Screenshot Proves Nothing
AI assistants do not keep a fixed ranking the way a map pack roughly does. Ask the same question twice in two fresh chats and the list of names can change. Ask a different assistant and you may get a different set of companies entirely. So a screenshot of your business inside one ChatGPT answer tells you that it happened once, on one day, to one session. It does not tell you how often it happens to the homeowner searching tonight.
Our own test below shows both effects. Repeated prompts came back with different tails, and two of the five most named companies were never named by Gemini at all. That is why the only useful number is a rate over many answers, split by assistant.
The Sample Report: A Real Scottsdale Pool Remodeling Test
We ran this for the market “pool remodeling company in Scottsdale, Arizona.” It was not run for a client, so there is no “you” in it. Read it as if you were one of the companies listed. Names are hidden on purpose: the five most named businesses appear as Company A through Company E, and everyone else is “other local companies.”
1. Run log
| Field | What we logged |
|---|---|
| Date and time | Wednesday, October 14, 2026, prompts sent between about 10:22 and 10:35 AM ET |
| Login state | Logged out on every assistant. One new chat per prompt. |
| Assistants | ChatGPT (logged out, temporary chat), Gemini (logged out), Google Search AI Overview (logged out, not AI Mode) |
| Location | City written into every prompt |
| Answers | 24 primary answers (3 assistants, 8 prompts each), plus 2 ChatGPT repeats |
What it means: anyone can rerun this next month under the same conditions and compare. Red flag: a report with no date, no login state and no location. Signed-in answers are shaped by the account’s history, so they say more about the person testing than about the market.
2. Prompt panel
Every prompt, exactly as sent. None of them names a business.
- Who are the best pool remodeling companies in Scottsdale, AZ?
- I want to resurface my pool in Scottsdale. Which companies should I call?
- Pool remodel contractors near Scottsdale Arizona with good reviews
- Who does pebble tec pool resurfacing in Scottsdale?
- Best company to remodel an old pool and add a baja shelf in Scottsdale AZ
- How much does a pool remodel cost in Scottsdale, and who should I get quotes from?
- Recommend a pool renovation company in North Scottsdale for new tile and coping
- Which Scottsdale pool remodelers are licensed and insured?
What it means: these are written the way buyers ask: by service, finish brand, neighborhood, cost and license. Red flag: a panel made of branded prompts like “Is [your company] good?” An assistant will usually talk about you when you put your name in the question. That proves nothing about whether it names you when a homeowner asks for options.
3. Mention rate: named in X of 24
Each cell counts primary answers that named the company in the answer itself. Companies that only showed up in a source link were not counted.
| Business | Total (of 24) | ChatGPT (of 8) | Gemini (of 8) | Google AI Overview (of 8) |
|---|---|---|---|---|
| Company A | 14 | 3 | 7 | 4 |
| Company B | 14 | 3 | 6 | 5 |
| Company C | 7 | 6 | 0 | 1 |
| Company D | 7 | 6 | 0 | 1 |
| Company E | 6 | 0 | 1 | 5 |
Behind them, other local companies were named in 3 or 4 of the 24 answers. Adding the two ChatGPT repeats does not change the order of the top five: A 15, B 14, C 9, D 9, E 6.
What it means: “named in 14 of 24” is a number you can track month to month. Red flag: a single “visibility score” with no N behind it. If you cannot see how many answers the score came from, you cannot tell 1 of 2 from 50 of 100.
4. How the assistants differed
- ChatGPT favored Company C and Company D, naming each in 6 of 8 answers. Gemini named neither of them once.
- Google AI Overview favored Company E, in 5 of 8 answers. ChatGPT never named it.
- Gemini favored Company A, in 7 of 8 answers.
- Only A and B were strong in more than one assistant.
What it means: “AI” is not one channel. If Company C got a report with one blended number, it would look fine and never learn that it is missing from Gemini entirely. Red flag: results reported for “AI” as a whole, with no split by assistant.
5. What changed on reruns
- ChatGPT, prompt 1, rerun in a new chat: 3 of the original 6 names came back, 3 dropped out and 2 new names appeared. Companies C and D stayed at the top.
- ChatGPT, prompt 2, rerun: 3 of the original 4 names came back, 1 dropped and 1 new one appeared.
- Gemini, prompt 1, rerun: the same two leaders, in swapped order (B first, then A).
- Google AI Overview also changed between runs of the same prompt.
What it means: in this test the names at the top held fairly steady while the bottom of each list moved around. We only repeated two prompts, so we are not putting a repeat rate on it. Red flag: a report built from one run per prompt that celebrates a new name near the bottom of a list. That may be gone tomorrow.
6. Sources the answers pulled from
- The state license board. ChatGPT cited the Arizona Registrar of Contractors (roc.az.gov) in several answers, most clearly on the “licensed and insured” prompt.
- Review and lead sites. Birdeye, Angi and HomeAdvisor (ChatGPT, Gemini), Houzz (Gemini, Google AI Overview) and Yelp (Google).
- The companies’ own websites. All three assistants used them. Google shows them as small source chips with a “+N” count.
- A manufacturer’s installer locator. pebbletec.com came up in ChatGPT and Google on the pebble tec prompt.
- Google Maps. Gemini often showed map pins and no visible citations at all.
What it means: this row tells you where to earn a listing, not just whether you were named. For a pool company here, that points to an active license record, review profiles, a manufacturer locator listing and service pages that say city, finish and license. For why an assistant can quote a tiny directory review count while you have hundreds on Google, see where your reviews actually show up. Red flag: a report that shows names but never the sources behind them, because then there is nothing to act on.
7. Competitors named next to you
In a client report, this row lists which companies sit beside the client in each assistant’s answers. Here, with no client, it is the table above: if you were Company E, your real rivals in Google’s AI Overview are not the same as your rivals in ChatGPT.
8. Google AI impressions by page
Not part of this test, because there was no client website. In a client report, this row comes from Search Console’s Generative AI performance report: [impressions by page, for the month]. Red flag: a report that claims AI queries or AI clicks the data source does not provide.
Method notes for this test
- Perplexity is not in the results. Logged out, in a private window, it answered prompts 1 and 2 with “Sign up and repeat your request” and showed no sources, so we stopped. We did not run it signed in, because a signed-in answer is shaped by that account. If a report shows logged-out Perplexity numbers, ask how it got past that wall, or whether it was signed in and labeled that way.
- Google location. The Google page header showed “Scottsdale, AZ,” but the footer showed a Georgia IP location. The city was in every prompt, but Google can still mix in where the tester is sitting, so set the location on purpose and write it down.
- Only one answer asked where the property was (ChatGPT, prompt 3). Every other answer took the city from the prompt.
- Names were merged only for obvious variants, such as a company listed under a short name and a long name. One Google result card had no name we could capture, and it was left out of the counts.
How We Run a Prompt Panel Each Month
The Scottsdale test is one run of the method we use for clients. In a client report, the setup stays fixed so this month can be compared with last month:
- 5 to 10 prompts per trade and city, written the way buyers ask, with no business names in them.
- Logged out, in a fresh browser session, with the location written into the prompt and logged.
- Repeats, so we can see which names hold and which ones come and go.
- Reported monthly as “named in X of N”, split by assistant, with the same prompts each month.
- A log for every answer: date, assistant and mode, the names in order, and the sources shown.
- Next to Search Console, so the prompt test sits beside Google’s own AI impressions for the same month.
As run in Scottsdale, the test covered three assistants (Perplexity would not answer logged out) and repeated two prompts on ChatGPT. A full client panel repeats more of the set.
Three Checks Before You Trust This Month’s Numbers
Added October 19, 2026. Two things happened in the week after we published that change how an October report should be read, and one check belongs at the start of every panel.
1. ChatGPT traffic “dropped” in October? Check the tag first.
After GPT-6 rolled out (October 7 for paid plans, October 8 for free), several analytics vendors reported ChatGPT sessions in GA4 falling by roughly half while Direct traffic rose. The reported cause: links inside ChatGPT’s answer text stopped carrying the utm_source=chatgpt.com tag, and the visits landed as Direct. Adobe’s Cedric Huesler says OpenAI confirmed to him on October 14 that it was a bug and that tagged clicks were back to roughly normal by the morning of October 15 (Search Engine Journal, October 15, 2026). As of October 19, OpenAI has made no public statement, so treat that as reported, not confirmed.
- Annotate October 8 to 15 in GA4 so nobody reads that week as lost demand.
- Count the tag, not the referrer. Huesler says most ChatGPT clicks still arrive with no referrer, so a report that relies on referrers undercounts.
- Judge on calls and forms. If booked jobs from your top ChatGPT landing pages held steady, the “drop” was a labeling problem.
2. Score the business, not the exact page.
A research paper posted October 11 (a preprint, not peer reviewed, with data collected before GPT-6) watched which pages ChatGPT actually cited and compared them with two stand-ins you can check without opening ChatGPT: Google’s results and OpenAI’s API. Together, the stand-ins found only 26.2% of the pages ChatGPT cited in the GPT-5.5 Instant tests, but 48.7% of the sites. And 44.5% of the distinct pages for a question showed up in just one of 20 runs (Search Engine Journal, October 18, 2026). That is why our panel is run in the assistants themselves, repeated, and scored as “named in X of N” at the business level. A report that tracks one URL’s exact citation, through an API, is measuring the noisiest thing available.
3. Make sure the AI knows which business you are.
Before counting mentions, starting with our next panel, we ask each assistant to describe the business by name and check what comes back: the right trade, the right city, the right phone and address. If an assistant mixes you up with a similar name, or splits one company into several names, mention counts for that assistant are not reliable until the business details are fixed. Consistent name, address, and phone across your site, Business Profile, and directories is the first repair.
Red Flags in Agency AEO Reports
- Hidden prompts. If you cannot read the exact questions, you cannot judge them.
- Single runs. One pass per prompt cannot separate a real gain from a lucky answer.
- Branded prompts only. Asking about your company by name is not the same as being recommended.
- One blended score. Our test shows companies that are strong in one assistant and absent from another.
- Guaranteed placement in ChatGPT. Answers change on rerun, so no one can promise a fixed spot.
- An llms.txt file or “AI schema” sold as the fix. Google’s own guide says no special file or markup is needed for its AI features.
- AI clicks with no named source. Ask where the number came from.
- A panel run only through an API or a rank tracker. Those stand-ins miss most of the pages ChatGPT actually cites. Ask whether the answers were collected in the assistant itself.
Our rule: if a report fails two of these, treat its trend line as noise until the method is fixed.
Questions to Send Before You Pay for One
- Can I see every prompt you run, word for word?
- Are the answers collected logged out, and where is the location set?
- How many times do you repeat each prompt, and will you show me what changed?
- Will you report “named in X of N” for each assistant separately?
- Which sources did the answers cite, and which of them are we missing from?
A good agency answers all five without hesitating. Still choosing the agency itself? See the full sales call script for choosing an agency. For how this looks in a single trade, see how AI names pool companies in ChatGPT and law firms in ChatGPT.
Want Your Own Prompt Panel?
We are a Google Partner agency. Book a call and we will run a logged-out prompt panel for your trade and city, show you who gets named in each assistant and which sources come up, and tell you where to start. See how our AI search visibility work runs.
Book a Strategy CallSources, read October 14, 2026, with the October 19 update read that day. Ostmen Marketing test log, Scottsdale pool remodeling, October 14, 2026 (ChatGPT, Gemini and Google AI Overview, logged out). Search Console Help: Generative AI performance report (Search). Google Search Central: Optimizing your website for generative AI features on Google Search. Search Engine Journal: ChatGPT traffic drop appears to be a tracking problem (October 15, 2026) and OpenAI’s API only partly matched ChatGPT’s citations (October 18, 2026), on the arXiv preprint 2610.15382.
