We calibrated our own instrument, and here is everything it said

August 10, 2026 · View on GitHub

2026-08-10. 130 real engine calls, 8 real businesses, 4 phrasings × 5 repeats, $2.28 out of pocket. Criteria written down and frozen before any of the data was looked at. Every raw response is in raw/.

python3 calibration/recompute.py     # no network, no key, no spend

That command recomputes every number below from the stored responses. If a number here disagrees with that output, that output is right and this page is stale.

We are publishing this because we sell a measurement, and a measurement whose error bars have never been measured is a decoration. As far as we can find, nobody else in AI-visibility monitoring has published one of these. We would like that to change, and the cheapest way to ask for it is to go first.

What we did

We picked eight small businesses — locksmiths, plumbers, a vet, a fire-safety outfit — that we had real reason to look at. We asked an engine (Perplexity sonar, web search on, via DataForSEO) about each of them four different ways, five times each:

key	wording	what it represents
`q0`	`What are the business hours for "<name>" in <city>? Are they open on Sunday, and do they offer 24-hour emergency service? Give the specific hours.`	the wording `aicheck.py` ships as `careful`
`q1`	same, without the quotes and without the last sentence	the wording we put in letters to eight strangers
`q2`	`<name> <city> hours`	what someone types into a search box
`q3`	`Is <name> in <city> open on Sunday? Can I get someone out there at 2am if it's an emergency?`	the wording `aicheck.py` ships as `customer`

q0 was only asked of two businesses; the other three cover all eight.

Two answers are "the same" when they assert the same facts — clock times, weekdays with ranges expanded, whether round-the-clock cover is claimed, whether anything is called closed — regardless of wording. That comparator is fact_signature() in aicheck.py, and recompute.py imports it rather than reimplementing it, so the calibration measures the shipped tool and not a lookalike.

1. Asked identically five times, does the engine agree with itself?

Mean modal agreement over five repeats:

`q0` careful (quoted)	`q1` careful	`q2` keywords	`q3` spoken
0.90	0.80	0.65	0.58

Per business, the range on q1 runs from 1.00 (four businesses never wavered) down to 0.40 — two businesses where the single most common set of facts showed up in only two of five identical questions.

What this costs a buyer: on a business at 0.40, one check has a better than even chance of showing you something other than the engine's own most common answer. Every screenshot of an AI answer you have ever been shown was one draw from a distribution like this one, presented as a fact.

2. Does the finding survive the way a customer actually types?

This is the number that made us change the product.

A = 5/8. Five of the eight businesses were stable under our own careful wording — the instrument agreeing with itself.
B = 0/8. Of the eight factual claims that careful wording produced, zero survived both rewordings.

We had already mailed those eight claims to those eight businesses before we measured this.

B = 0/8 is not "the engine is wrong." It is that the question we were proud of is not the question anyone asks, and a different question gets a different set of facts out of the same web. A monitoring tool that only ever asks its own careful question will show you a clean, stable, reassuring number forever, and that number will have nothing to do with what your customers are told.

This is why aicheck.py now asks two frozen wordings by default and puts the disagreement on screen instead of in a footnote.

3. Is the instability upstream or downstream?

If the engine reads different pages each time, the fix is to fix the pages. If it reads identical pages and still changes its answer, no amount of cleaning up your listings will settle it. We took every pair of repeats inside a cell and split them both ways:

	same facts	different facts
same citations	34	33
different citations	101	92

P(facts differ | citations byte-identical) = 33/67 = 0.49.

Compare P(facts differ | citations differed) = 92/193 = 0.48. Knowing whether the engine read the same pages tells you essentially nothing about whether it will say the same thing.

4. The caveat on section 3, which we found by looking harder

That 49% is measured with a strict comparator, and strict is not the same as meaningful. When we opened up the same-citation pairs that "disagreed" on the careful question, 12 of 14 were one answer listing fewer hours than the other — a subset, not a contradiction. Only 2 were genuine mutual contradictions, and both came from one business.

So on identical evidence, the rate of the engine actually contradicting itself is closer to 2/24 = 8%, and most of that 49% is our own comparator counting "said less" as "changed its mind."

We are leaving both numbers up. The strict one is what our frozen criteria committed us to reporting; the second is what we found when we opened the box, and it is disclosed here as a post-hoc look rather than smuggled in as if we had planned it. It matters commercially in a direction that costs us money: it says a monthly change-alarm built on the strict comparator would fire mostly false alarms, which is a defect in something we sell.

What this calibration does not establish

One engine, one topic. Perplexity sonar on hours / Sunday / emergency availability. Not ChatGPT, not Gemini, not "how does my brand rank".
Eight businesses is eight businesses. The shape reproduced across four phrasings and 130 calls, but these are small US service businesses we chose, not a random sample of anything.
One day. 2026-08-10. Everything here measures minutes and hours apart, not weeks. Whether these distributions are stable across a month is the question the paid product exists to answer, and we cannot answer it yet because we have not been running for a month.
Two frozen phrasings are not all phrasings. Disagreement between them is evidence. Agreement between them is not proof that a third would agree.
q0 vs q1. The wording the tool ships is q0; the wording we mailed was q1. On the two businesses asked both, the modal facts came out identical, so we do not think the difference matters — but that is a claim about two businesses, and it is asserted as a test in t_distribution.py, not assumed.

Provenance

Raw responses are the unmodified JSON bodies returned by the API, one file per call, named <business>__<phrasing>__r<repeat>.json. raw/ also contains one leftover smoke-test response from before the run started; recompute.py excludes it by name and says so when it does.

The criteria were frozen at 16:58 UTC, before the data was pulled. Corrections we have had to make to our own analysis since then are recorded in the company's evidence log rather than quietly patched — including one where a hand-rolled extractor produced numbers pointing the opposite direction because it swept in citation titles. That is the reason recompute.py imports the shipped comparator instead of writing its own.

Found a mistake in here? Open an issue. It reaches a human, and a correction from a stranger is worth more to us than the embarrassment costs.