Skip to main content

Methodology · Reviewed

How we measure AI visibility

Ask ChatGPT the same buyer question twice and you can get two different answers. Most tools report whichever one they happened to catch. Collimer, our engine, asks dozens of real buyer questions across four assistants, scores every answer the same way, and reports the interval around the result, so you know what is signal and what is noise.

The instrument: every answer from every engine goes through the same scoring lens, and the reading always carries its interval.

The instrument, in numbers

Question types
5
one per buyer stage
Prompts per type
4
in the audit panel, written for your brand
Engines
4
ChatGPT, Claude, Gemini, Perplexity
Runs per prompt
×2
the same engine answers differently twice
Confidence interval
95%
on every score we report
Free scan
Up to 100 answers: 25 prompts, asked of ChatGPT and Claude, twice each, plus a live-search check on Gemini
Full audit
The same panel asked of all four engines, every category
Prompts
Generated for your brand, category, buyers and competitors; never a generic list
Refresh
Models drift; we re-scan and report the delta against its interval

What we ask

Five kinds of buyer question, one for every stage

A buyer's intent is one beam. We split it into the five ways people actually ask an assistant for help, then write four prompts of each kind for your brand, your category and your competitors. Every category counts equally: the score is a plain average of every answer, so no stage is quietly weighted up to flatter the result.

  1. 01 · Top of funnel

    Problem-led

    Buyers describe the pain, not your name. Absent here, you never make the shortlist.

    “What helps a B2B SaaS team see churn coming before renewal?”

    4 prompts · 20%

  2. 02 · Exploring

    Category-aware

    They know the kind of tool they need. You have to show up as a credible option in it.

    “What are the best customer analytics platforms for mid-size SaaS?”

    4 prompts · 20%

  3. 03 · Switching

    Alternative-seeking

    Someone is unhappy with a rival. Appearing here turns their churn into your pipeline.

    “What are good alternatives to Rival Y for churn tracking?”

    4 prompts · 20%

  4. 04 · Deciding

    Comparison

    Head-to-head questions carry the highest intent. The answer often is the decision.

    “Acme Analytics vs Rival Y: which is better for a 50-person team?”

    4 prompts · 20%

  5. 05 · Checking you

    Branded

    Does the assistant actually know what you do, and say it correctly? This is the baseline.

    “What does Acme Analytics do, and who is it for?”

    4 prompts · 20%

Example prompts are illustrative, for a fictional brand. Real prompts are generated per scan.

How we score

How every answer gets scored

Every answer is scored on its own, by the same rule, before anything is averaged. These are the exact multipliers the engine uses today.

answer score = 100 × mention × sentiment × position

Mention

Does the answer name your brand at all? If not, the answer scores zero and nothing else matters.

named
1.0
absent
0.0

Sentiment

How the assistant talks about you. A recommendation lifts the answer; a warning cuts it.

positive
1.2
neutral
1.0
negative
0.7

Position

Your rank among the brands an answer names, by first appearance. The decay follows click-through falloff in ranked lists. If no rank can be read, no penalty.

Two honest footnotes. A positive first-place mention scores 120, above the 100 base; the final average is clamped to 0 to 100. And a citation multiplier (does the answer link to your site?) is built into the formula but held at 1.0 until its measurement is wired in; we report citations separately for now.

Score one answer yourself

Pick what the assistant said. The math updates as you go.

Mentioned?
Sentiment
Rank in the list
100 × 1.0 × 1.0 × 0.72 = 72

How answers become your report

Your report is built from the bottom up, and nothing is weighted by hand. Below is a small, fictional scan: five question types, each asked of two engines, so ten answers. Follow the numbered steps to see where each number in the report comes from.

  1. 1 Score each answer Each cell is one answer, scored with the formula above.
  2. 2 Average down each engine That is the "By engine" breakdown in your report.
  3. 3 Average all answers That is your headline visibility score.
  4. 4 Measure the spread That is the ± interval printed beside it.
Example scan: answer scores by question type and engine, with engine averages
Question type ChatGPT Claude
Problem-led 100 #1 · neutral 102 #2 · positive
Category-aware 72 #3 · neutral 0 not named
Alternative-seeking 120 #1 · positive 0 not named
Comparison 42.7 #4 · negative 50 #5 · neutral
Branded 0 not named 85 #2 · neutral
2By engine 66.947.4

3Headline score

57 ±28

571.7 total ÷ 10 answers = 57.2

4Why ±28?

Ten answers that range from 0 to 120 leave a lot of doubt. A free scan collects up to 100, which narrows this to about ±9. More on that below.

Reading it: this brand wins problem-led questions on both engines but is missing from three answers entirely. Each blank cell pulls the score down hard, so getting named where you are absent usually moves the score most.

Show the math for every answer
Worked example: ten answers scored and averaged
Prompt Engine Rank Sentiment Math Score
Problem-led #1 ChatGPT 1 neutral 100 × 1.0 × 1.00 100
Problem-led #2 Claude 2 positive 100 × 1.2 × 0.85 102
Category #1 ChatGPT 3 neutral 100 × 1.0 × 0.72 72
Category #2 Claude n/a n/a not mentioned 0
Alternatives #1 ChatGPT 1 positive 100 × 1.2 × 1.00 120
Alternatives #2 Claude n/a n/a not mentioned 0
Comparison #1 ChatGPT 4 negative 100 × 0.7 × 0.61 42.7
Comparison #2 Claude 5 neutral 100 × 1.0 × 0.50 50
Branded #1 ChatGPT n/a n/a not mentioned 0
Branded #2 Claude 2 neutral 100 × 1.0 × 0.85 85
mean = 571.7 ÷ 10 57.2

Why the interval is the point

Every score comes with its error bars

Assistants are not deterministic, so a single number is never the whole truth. We report a 95% confidence interval around the mean of the answer scores, using the standard normal approximation:

interval = mean ± 1.96 × s ÷ √n

s is the spread of the answer scores, n is how many answers we collected.

Why we ask so many times

The ten-answer example above has a spread of s = 45.8. With n = 10 that is 57 ±28, too wide to act on. The same spread across the 100 answers of a free scan narrows to ±9. More answers, tighter reading.

When a change is real

We call a move between two scans real only when their intervals do not overlap. Anything else is labelled "within measurement noise", with the ± beside it. Citation rates get the same discipline through a two-proportion z-test at p < 0.05.

With fewer than two answers there is no spread to measure, so we show no interval at all rather than a falsely precise one. We will never report a single visibility number as if it were exact.

Verification

When a fix ships, we don't take anyone's word for it, including our own. We re-fetch the published page, confirm the change is actually live, and re-probe the same question set against the baseline. You see the delta against its interval, or an honest "no measurable change." Outcomes are reported in one of five states, from verified-moved to moved-against; nothing is collapsed into a thumbs-up.

Limitations

Limitations we'll say out loud

  • Model providers change behavior without notice; your score drifts and we show how much.
  • Engines ground differently: Perplexity always cites; ChatGPT triggers web search on a minority of queries. We report per engine, not a false average.
  • Citation overlap between engines is low; being cited by one is not being cited by all.
  • Sentiment and rank are read from each answer by a model, so they carry their own error. The interval absorbs some of it; it cannot absorb a systematic misread, which is why we quote the answers.
  • A scan measures presence in answers, not downstream conversion. It's the leading indicator, not the sale.

Want the evidence behind the recommendations? The playbooks cite the research each lever is built on. Questions about the method? hello@sandcastlelabs.ai