Statistical Significance for Marketing Experiments

AI made producing test variants nearly free, which means marketers now run more experiments than ever, and I review more broken test readouts than ever. The variants got cheap. The math that decides whether a result means anything did not, and most of the “winners” I am shown in audits are noise wearing a green badge. This post is the working understanding I think every marketer who touches an A/B test needs, with the arithmetic kept to what fits on a napkin. It sits in my measurement series for the same reason everything there does: a wrong number acted on confidently costs more than no number at all.

What significance actually means

Statistical significance answers one narrow question: if the two variants truly performed identically, how often would chance alone produce a gap at least this large? When a tool reports significance at the 95% level, it is saying that a gap this size would arise from pure luck less than 5% of the time. That is all. It does not say the effect is large, it does not say the effect will persist, and it does not say the test was designed well. I have watched teams treat “statistically significant” as a synonym for “true and important”, and the phrase means neither. It means “probably not luck alone, under the assumptions we made”.

The companion concept, the one the tools surface less, is power: the probability your test would detect a real effect of a given size if one exists. An underpowered test that ends “no significant difference” has not shown the variants are equal. It has shown the test was too small to tell, which is a different and much less useful finding.

The napkin math for sample size

Here is the arithmetic I actually do in meetings, using the standard shortcut for the usual settings (95% confidence, 80% power). Take your baseline conversion rate as a decimal, call it p. Decide the smallest lift you would care about detecting, in absolute percentage points, call it d. The visitors you need per variant is roughly 16 times p times (1 minus p), divided by d squared.

A worked example with round numbers. Baseline conversion 3%, and you want to detect a lift to 3.6%, which is a 20% relative improvement and a genuinely good result for a landing page change. So p is 0.03, d is 0.006. The formula gives 16 × 0.03 × 0.97 / 0.000036, which is roughly 13,000 visitors per variant, about 26,000 in total. At 500 visitors a day, that is nearly two months for one two-way test. When I want the precise version rather than the napkin version, I put the same numbers into Evan Miller’s sample size calculator, which is free and asks for exactly the three inputs above.

The lesson generalizes brutally: the smaller the effect you are hunting, the sample grows with the square of the shrinkage. Halve the detectable lift and you need four times the traffic. This single fact explains most bad testing programs I see. They are sized to detect miracles, run on traffic that could only ever confirm miracles, and miracles are rare. The same napkin formula sizes a tool holdout too, and my incrementality testing walkthrough shows it applied to an AI subscription end to end.

The three ways tests lie

Peeking. Watching the dashboard daily and stopping the moment significance appears. The 5% false positive guarantee holds for one look at a predetermined sample size. Check twenty times and stop on the first green badge, and your real false positive rate is several times higher, because you gave randomness twenty chances to fool you and took the first offer. My rule is decided before launch: the sample size and the end date are written down, and the test runs to them. Reading along is fine. Acting early is not.

Too many variants. Test five subject lines against control at 95% confidence and the chance that at least one clears the bar by luck alone is far above 5%. Every added variant is another lottery ticket for noise. I keep tests to the comparisons I would actually act on, and when a tool crowns a winner from a large field, I ask for the runner-up gap before I believe it.

Partial weeks and clipped seasons. Conversion behavior differs by weekday, payday and season. A test that runs Tuesday to Friday samples a different population than one that runs a full week, so I run tests in whole-week blocks, minimum two, and I do not launch tests across events I cannot repeat, like a sale.

What to do with low traffic

Most companies asking me about testing do not have 26,000 visitors to spend on one experiment, and the honest advice is to stop pretending otherwise. Three moves work.

Test bigger swings. The formula rewards it quadratically: a change bold enough to plausibly move conversion by half again needs a fraction of the sample. Offer, headline promise and page structure are testable on modest traffic. Button shades are not.

Move the test upstream to where the volume is. Email lists and ad impressions often carry ten or a hundred times the sample of a bottom-funnel page, so subject lines and ad creative can be tested properly even when your checkout cannot.

And when the volume is not there at all, drop the A/B framing rather than faking it. A before and after comparison with a stable baseline, read with appropriate humility, beats a fake experiment read as gospel. This is the same trade I recommend for attribution in the zero-click measurement work: less precision, more truth.

Where AI helps, and the one place it must not

I use ChatGPT constantly around experiments: to critique a test design before launch, to translate a readout into plain language for stakeholders, and to interrogate a vendor’s claim of significance. A prompt that earns its keep: paste the raw numbers, visitors and conversions per variant, plus how long the test ran and how many variants existed, and ask “what conclusions does this data support, what conclusions does it not support, and what would you check before acting”. The model is good at spotting the failure modes in this post, because they are well documented.

The one prohibition, the same one I apply across AI-assisted reporting: the model never supplies a number. It reasons about your arithmetic, it does not perform the arithmetic unchecked, and it absolutely does not estimate a significance level from vibes. A model handed an ambiguous readout will produce a fluent, confident, wrong interpretation. Numbers in, narrative out.

FAQ

Is 95% confidence mandatory? No, it is a convention. What matters is choosing the threshold before the test and understanding the trade: at 90% you accept a doubled false positive rate in exchange for smaller samples. For a reversible landing page change I will sometimes take that trade knowingly. For anything expensive to reverse, I will not.

My tool says the test reached significance in three days. Can I stop? If three days was your pre-registered plan, yes. If the plan was two weeks and the badge just appeared early, that is peeking, and the readout is exactly the kind that reverses in week two. The tool cannot tell the difference between those situations. You can.

Do these rules apply to email subject line tests? Especially there, because list sizes make big samples easy but subject line effects are small, so tests are underpowered anyway, and send-time splits add the partial-week problem. The napkin formula works the same: open rate as p, the difference you would act on as d.

What should I report to stakeholders from a test? The absolute numbers, the relative lift, the confidence level, and what the test cannot say. I pick the metrics with the same framework I use for choosing AI marketing KPIs: report what someone will act on, not what looks impressive.

This guide is part of my measurement series. The reporting workflow that keeps models from inventing numbers is in AI reporting automation, and the aggregate-first measurement philosophy continues in attribution in the zero-click era.