How often should you track AI visibility? Cadence, sample size and noise

For a monthly client report, asking every prompt on every engine once a week is enough — about 240 answers per client per month at 20 prompts and three engines. Track daily when you're testing a change on a handful of prompts and need to see movement within days. Checking once a month is too little to say anything about change.

The reason is arithmetic, not taste. AI answers vary from run to run, so every visibility number is an estimate from a sample. The more answers behind it, the smaller the margin of error.

How precise is a mention rate?

If a client is mentioned in 30% of answers, here's roughly how far the true rate could be from what you measured (95% confidence, treating answers as independent):

Twenty answers — one per prompt per month — leaves you unable to tell 15% from 45%. Two hundred and forty narrows it to roughly 24–36%.

How many answers to call a change real?

Detecting a change between two months needs more data than estimating one month. Roughly, to reliably detect (80% power, 5% significance):

Two practical consequences:

1. Large changes show up quickly; small ones don't. A jump from 30% to 45% is visible with a weekly cadence on one client. A five-point move needs far more data than most agencies collect per client — so don't headline it.
2. Pool where you can. Across three engines, or across all prompts in a topic group, you get more answers per comparison. Report engine-level and prompt-level numbers too, but expect them to be noisier.

Daily, weekly or monthly?

Daily data isn't wrong; it's often more than a monthly report needs. A day-over-day chart of a small prompt set mostly shows randomness, and clients read meaning into every wiggle. Aggregate to weekly or monthly before you show it.

A sensible default for agencies

• Every client: ~20 prompts × 3 engines, weekly, reported monthly with counts.
• Flagship client or a live experiment: add daily tracking on the 5–10 prompts you're actively working on, for the duration of the test.
• Always: keep the prompt set fixed between periods, show "X of Y answers", and flag changes smaller than the margin as "no clear change".

This is the cadence Ranxii runs: every active prompt on every engine once every seven days, about 240 stored answers per client per month at the default 20 prompts, with month-over-month change calculated on the same prompt set.

How to talk about noise with clients

Say it once, in plain words, in the first report: "AI answers vary, so these are measured rates from many answers. Changes of a few points month to month are within normal variation; we'll call out changes that are bigger than that." Then hold yourself to it. The agency that doesn't celebrate noise is the one whose real wins get believed. See why ChatGPT gives different answers every time for the background, and the AI visibility report template for how to present it.

Notes on the numbers

Margins use the normal approximation for a proportion; sample sizes use the standard two-proportion formula (two-sided, α = 0.05, power 0.8). They're guides for planning, not guarantees — related answers and changing models both add uncertainty.