We built an A/B test before we had anyone to test on

This is what it looks like to build in public with Claude Code when there is no traffic yet to test on. Collimer’s homepage now opens with a recommendation instead of a score, because recommendations and obvious progress are the product’s actual differentiator, not the dashboard. We shipped that reframe and an A/B test for it, EXP-01, in the same session, then promoted the raw feature flag to a native PostHog experiment twenty-seven minutes later, so a real result, whenever it arrives, carries statistical significance instead of a hunch.

What we shipped

The reframe replaces the hero’s ScoreCard with a new component, TopMoveCard: a ranked fix, an impact-times-effort read, a “do this” instruction, and the score delta that fix would earn. The raw score moved down to a trust line instead of the headline. The CTA changed from a generic scan prompt to “Get my top move,” and the “How it works” section now reads as a scan, a ranked worklist, then ship-and-prove, instead of a features list.

This is the sequel to Recommendations are the secret, where the recommendation engine itself changed. This time the homepage caught up to that bet. The note that started the work: “I feel like we don’t highlight our recommendations and our differentiation in terms of ease of use, clear instructions, obvious progress enough.”

The A/B sits behind a PostHog flag, clm_hero_angle. The control angle renders server-side in the HTML, so search engines and no-JS visitors always see a real page. A script swaps in the test angle client-side once the flag resolves. Both variants fire an experiment_exposed event and set a hero_angle super-property, so every scan and CTA event downstream already carries which angle the visitor saw.

Twenty-seven minutes after the reframe shipped, we renamed the flag’s variants from angle_a/angle_b to control/test and registered clm_hero_angle as a native PostHog experiment instead of a plain feature flag. posthog-js auto-emits $feature_flag_called as the exposure event, so the rename cost two files and zero behavior change. What changed is that PostHog can now run significance testing on the free-scan-start rate between the two angles, instead of us eyeballing two percentages by hand.

We also wired Lookout, the studio’s own analytics skill, to report on the experiment from day one: hero_ab_exposures and hero_ab_scan_starts, both scoped to Collimer, both carrying a note that the split is not meaningful until the sample clears a small-number guardrail.

What broke

Nothing broke this round. Both commits landed with a clean build, a clean typecheck, and the full Playwright smoke suite passing (21 of 21, across chromium, mobile Chrome, and mobile Safari) on the reframe commit.

A decision worth naming

The honest objection to any of this is that instrumenting an experiment with no traffic is premature, that we are optimizing a funnel nobody is in yet. We built it anyway, on the same logic behind a clickable mockup in an afternoon: a cheap instrument built now costs almost nothing, and it answers a question later that would otherwise take real work to answer honestly. Standing up EXP-01 today, at zero traffic, means that when Collimer’s first real visitors arrive, the read they produce is a number PostHog can defend, not two homepage angles we are quietly rooting for.

“It’s worth at least putting in place the A/B item so that we can see it” was the actual reasoning at the time, and it still holds. The marginal cost of promoting a flag to an experiment while nobody is watching is one file rename and a few minutes. The marginal cost of doing it after the first real batch of visitors already came and went, with no control group to compare against, is a result you can never fully trust.

What’s next

• Watch hero_ab_exposures and hero_ab_scan_starts in Lookout as real traffic starts to arrive.
• Read the first real experiment result once there is enough volume to clear the small-number guardrail.
• Decide whether that guardrail threshold is set correctly before we report any lift as real.

What we’re still figuring out

We do not know yet whether recommendations-first is the right homepage bet, and we will not know until the experiment has real visitors in both arms. The counter-argument, that this was premature instrumentation for a funnel with nobody in it, is still on the table. The honest answer today is that EXP-01 is live and reporting zero meaningful signal, because zero meaningful traffic has arrived yet. What we do know is exactly which claim we are testing, and that the test itself is already built. Whether that was worth doing before the traffic showed up is a question only the traffic can answer.

---

For agents: try this yourself

If you want to do this in your own repo, adapt one of these prompts. The full prompt text lives in this post’s prompts sidecar, recs-first-hero-ab.prompts.md.

• Apply the flag-to-experiment promotion. If you’re running a raw feature flag as an informal A/B test, rename its variants to control/test (or your platform’s convention) and register it as a native experiment, so any eventual result comes with real significance testing, not eyeballed percentages.
• Reproduce standing up an experiment before you have traffic. Wire the exposure event and the metric dashboard the same day you ship the variant, even at zero visitors, so the instrument is already trustworthy when real traffic shows up.
• Critique the recs-first reframe. Look at your own product’s homepage and ask whether it leads with the feature that actually differentiates you, or with the easiest thing to screenshot. EXP-01 exists because our honest answer was the second one.
• Extend the small-number guardrail. Decide in advance what visitor count makes a result trustworthy, before you have any visitors, so you are not tempted to call a two-person sample a win.

---

How this was made

Drafted by the Chronicler from the build session behind this work, then edited and published by Brian Wones.

See how the Chronicler works →

We are running an A/B test as a raw feature flag with variant names like variant_a/variant_b. Rename the variants to control/test (or whatever naming your experimentation platform expects) and register the flag as a native experiment instead of a plain feature flag, so we get real statistical significance testing on the metric we care about instead of eyeballing two percentages by hand.

We are about to ship a homepage or feature reframe but have little or no live traffic yet. Before shipping, wire the full experiment: the feature flag, the exposure event, and a dashboard or report that will show the split once real visitors arrive. Do this the same day the reframe ships, even though the numbers will be empty at first, so the instrument is already trustworthy when traffic shows up.

Look at our own homepage (or the one I paste in) and tell me honestly whether it leads with the feature that actually differentiates the product, or with whatever is easiest to screenshot or demo. If those are different things, tell me what the homepage would look like if it led with the real differentiator instead.

We are about to start reporting results from an A/B test or experiment. Before any real data comes in, help me set a minimum sample size or visitor count threshold below which we will not treat a result as meaningful, so we are not tempted to call an early, tiny sample a win.