Build log · collimer
Is an AI visibility score still accurate when one engine never answers?
On July 19, 2026 we merged a 10-file, 313-line fix (pull request #1665) so that Collimer’s free scan admits when an engine went missing. Before it, if one engine failed completely, the score was computed from whichever engines answered, and a partial scan rendered exactly like a full one. Now the result records which engines answered and says “based on N of M engines” when any returned nothing.
What went wrong without the fix?
The ticket behind the fix describes it plainly: if one of the free scan’s engines failed mid-scan, the score was silently computed from the survivors. The number was not wrong. It was a narrower measurement than its label implied, and nothing on the page told the reader.
The ticket calls this the free-scan version of an earlier “4/5 engines” incident. We do not have that earlier incident’s details in this post’s sources, and we could not point to a free scan that hit the failure before the fix. This is a design decision made from a known failure mode, not a post-mortem.
What did we change?
Three things, all small:
- Each scan result now carries a per-provider success count, called
provider_coverage. - A derived flag,
partial_engines, is true when any configured engine returned zero successful probes. - The result page shows “based on N of M engines” when coverage is short. When the scan also has an unresolved brand, the two warnings are composed into one card rather than stacked.
| Engines that answered | partial_engines | What the page says |
|---|---|---|
| All M configured | false | no caveat |
| Any 1 or more returned zero | true | ”based on N of M engines” |
Why disclose and not correct?
We could have tried to estimate what the missing engine would have said, or reweighted the survivors. We did not. A reweighted number is still a number the reader cannot check, and the problem was never that the arithmetic was wrong. It was that the label overstated what was measured. A narrower reading, labeled as narrower, is still usable. Four engines answering is a real measurement. An unlabeled one is the failure.
This is the same family as the week our number turned out to be lying five ways: a figure that looked complete and was not.
What did review catch?
The first version built the engine names with a generic capitalize call, which turned “openai” into “Openai” in the caveat. The review flagged it. The code already had a lookup of provider display names for a sibling “some AI engines failed” message in the paid audit, so we extracted that lookup into a shared function and reused it, instead of writing a second list.
What did the fix miss?
On August 4, 2026, 16 days later, pull request #1869 fixed a second way the same scan could publish a wrong number. A fallback to the grounded-search rate, meant for when the base engines produced nothing, could never run, because the base rate was always stored as a number and a missing one became 0.0. When the base engines were dead and the grounded track was alive, five surfaces published 0: the result page, the PDF, the share image, the API and the ready email. So a total base-engine failure could still publish a false zero until August 4, whatever the July 19 caveat showed.
How can you audit any visibility report for this?
You do not need our product to ask the question. For any report, find two numbers: how many engines were supposed to run, and how many actually did. If the report names five engines on one line, one interval and no per-engine coverage note, you cannot tell a full reading from a partial one wearing the full one’s label.
What we’re still figuring out
- Whether there is a third way the number can look complete and not be. We found two in 16 days.
- How often an engine actually goes missing in production. We added the count so we could find out, and we have not pulled it.
- Whether the caveat changes what readers do with a partial score. We have no evidence on that.
- Whether the same disclosure belongs on every surface that shows a score. This fix covers the free scan result page.
For agents: try this yourself
If you produce a score by pooling several sources, adapt one of these. Full prompt text lives in this post’s prompts sidecar.
- Audit the coverage. Given a report that pools multiple engines, list how many were configured, how many returned data, and whether the report says so.
- Draft the caveat. Write one sentence, in plain words, that tells a reader a score is based on fewer sources than expected, without hiding the number.
How this was made
Drafted by the Chronicler from Collimer commits, a ticket and a published guide, then edited and published by Chase.
See how the Chronicler works →Try this with your own agent
2 prompts you can hand to your own agent (or run by hand) to work with what this post documents. Edit the bracketed parts for your context.
Audit a pooled score for missing sources
Here is a visibility report: [paste]. State how many engines or sources were configured, how many returned data, and whether the report says so. If it does not say, list what in the report would let me tell a full reading from a partial one. Do not guess which source failed.
Draft the partial-coverage caveat
Write one plain sentence for [our results page] that tells a reader the score is based on [N] of [M] sources, without hiding the number and without apologising. Give two variants of different length. More in Collimer Build
-
Which sources do AI engines cite most? Your greatest resources are within reach: your LinkedIn and your GitHub
-
How does Microsoft Copilot decide what to cite? It asks Bing a few words at a time
-
Should you measure AI visibility per product line? Yes, because one average can hide the product AI has never heard of