Incrementality Testing for AI Marketing: A Walkthrough

An incrementality test answers the only question about an AI tool that a dashboard cannot: what would have happened without it. You hold the tool out of one comparable slice of the work, run it everywhere else, and read the gap. The idea fits in a sentence. The execution has about six decisions in it, and getting any of them wrong quietly turns the test into theater.

I have made the case elsewhere that holdouts are the strongest proof an AI program can produce, and the industry is coming around to the same view. In TransUnion’s July 2025 survey of 196 US marketers with EMARKETER, 67% said they now prioritize incremental ROI, ahead of every other measurement goal. This post is the part that usually goes missing between the intention and the readout: the actual walkthrough. Six steps, one worked example with the arithmetic shown, and the ways I have seen tests go wrong.

What the test measures, and what it does not

Incrementality is the difference between your result with the tool and your result without it, measured at the same time on comparable slices. That last clause is the whole method. A before-and-after comparison shares its timeline with everything else that changed that month. A holdout shares nothing, because both groups live through the same market, the same season and the same news cycle, and the only difference between them is the tool.

This is also why I trust a small holdout over a big dashboard number. Platform-reported and vendor-reported results have well-documented mechanisms that inflate them, and every one of those mechanisms dies in a holdout, because the control group does not care who commissioned the study. What the test does not measure is everything outside its slice. A subject-line test says nothing about your content program. Scope the conclusion to the scope of the test and you will stay honest.

Step one: write the claim as a sentence with numbers in it

Before touching any tool settings, I write one sentence: “Using [tool] for [task] will improve [metric] by at least [amount] within [period].” The vendor’s pitch or your own hope goes into that template, and the template forces the two decisions people love to defer: which single metric counts, and how big a lift would actually justify the subscription.

The “at least” number should come from the tool’s price, worked backwards. If the tool costs $200 a month, work out what lift in your chosen metric pays for it, using the same logic as my AI ROI calculation, then set the bar somewhat above break-even. A test designed to detect any difference at all is a test designed to run forever. You want the smallest difference worth acting on, because that number sets the size and length of everything that follows.

One claim per test. If the tool allegedly improves subject lines and send times and segmentation, that is three tests, and the discipline matters because a test that changes two things can attribute its result to neither.

Step two: pick the split

The split is the unit you randomize, and the right one is whichever of these three you can keep clean. My order of preference:

Split the audience. Half the email list gets the AI-assisted version, half gets your normal process, assigned randomly by your email platform. This is the cleanest option a marketer can run without help, because every mainstream email tool has random splitting built in. The same logic works for any channel where you control delivery per person or per account.

Split the geography. When the tool touches something you cannot split per person, like a campaign’s creative, run the AI version in most regions and hold out one or two comparable regions as control. Comparable is the hard word: pick holdout geos that historically track the rest of the country in your metric, and if you do not know whether they do, this is a fifteen-minute question for whoever owns your analytics. Ask them for a chart of the candidate geo’s metric against the national line for the last six months.

Split the time. Alternate weeks on and off the tool. I treat this as the last resort because time never splits clean, one arm always catches the payday week or the product launch, and it takes more whole-week cycles to average that noise out. Use it only when audience and geo splits are impossible, and run at least three on-off cycles.

What I never do is let the split be chosen by convenience of implementation, like giving the tool to the team that volunteered. Volunteers differ from non-volunteers in exactly the ways that corrupt a comparison.

Step three: size it and schedule it

Now the claim from step one earns its keep. The smallest lift worth detecting, together with your baseline rate, determines how many people or geos or weeks the test needs. I keep the full arithmetic in my statistical significance guide, napkin formula included, and I will not re-derive it here. The one-line version: sample size per group is roughly 16 times the baseline rate times one minus the baseline rate, divided by the square of the detectable lift.

Two scheduling rules sit on top of the math. Run whole weeks, minimum two, because weekday behavior differs and a partial week samples a skewed slice. And write the end date down before launch. The test ends on the date or at the sample size you pre-registered, and the first person to say “it looks significant, can we stop early” gets pointed at the peeking section of the significance guide.

The split ratio is a business decision, and I set it by what the test is risking. If the AI version is the unproven thing, expose half. If the AI version is already live everywhere and you are testing whether it earns its renewal, hold out 10 to 20% and accept the longer runtime, because the sample math punishes small holdouts.

A worked example, start to finish

This example is hypothetical, with round numbers chosen to make the arithmetic easy to follow. It is the shape of the test I would design, not a test I am reporting results from.

Imagine a retailer with a 40,000-subscriber list paying $200 a month for a tool that writes AI-personalized subject lines. The claim, written per step one: “AI subject lines will lift open rate by at least 2 percentage points over four weekly sends.” The baseline open rate is 25%, and the 2-point bar was set because at this list size a smaller lift would not change any downstream number enough to notice.

Sizing, per step three: 16 × 0.25 × 0.75 = 3, divided by 0.02 squared (0.0004) = 7,500 subscribers per group. The list split of 20,000 per arm clears that on a single send, so the four-send window is there for coverage across weeks, not for sample. Pre-registered plan: four weekly sends, same offers and send times in both arms, judged on pooled open rate, with click rate and unsubscribe rate watched as guardrails.

Hypothetical results after four sends: the control arm opens at 24.8%, the AI arm at 27.1%. The lift is 2.3 percentage points, above the pre-set 2-point bar. Clicks tell a quieter story: 2.9% control against 3.1% AI, a 0.2-point gap. Unsubscribes are flat. My reading of that panel: the tool genuinely moves opens, and barely moves clicks. In money, 0.2 extra clicks per hundred recipients on a 40,000-person send is 80 extra clicks per week. Whether 320 monthly clicks justify $200 depends entirely on what a click is worth in this shop, and that is the conversation the test was designed to force. A 2.3-point open lift sounds like a win. The click column decides whether it is one.

Step four: read the result like an adult

Three outcomes are possible, and I decide what each one means before launch, in writing, because a result read after the fact bends toward whatever the reader wanted.

The lift clears the bar. The tool stays, and the readout says exactly what was tested: “AI subject lines lifted opens 2.3 points in a randomized four-week split.” Resist upgrading that sentence to “AI improved our email program” in the retelling.

No meaningful difference. This is a real answer, with one caveat: a null from an undersized test is not evidence of no effect, it is evidence of an undersized test. If the sizing from step three was honest, a null means the tool’s effect at your scale is smaller than the smallest lift worth paying for, and the subscription goes. That is the test working.

The AI arm loses. It happens, and it is the most valuable outcome per dollar of test cost, because it catches a regression the dashboard would have dressed up as a win. Roll back, write down what happened, and test the next claimed use case rather than concluding AI is over.

Whatever the outcome, the result travels in the one-line-per-claim format from my proof framework: claim, evidence type, confidence label. A holdout earns the high-confidence label. That is the entire reason to go through this instead of quoting the vendor’s number.

The failure modes I actually see

Contamination. The AI arm’s subject lines get reused in the control arm’s sends “because they performed well”, or the holdout geo sees the AI creative through national channels. Once the treatment leaks into the control, the gap shrinks toward zero and the tool looks worse than it is. Brief everyone who touches the campaign that the control arm is load-bearing.

A second change mid-test. A price change, a redesign, a new segment added to one arm. The test only isolates the tool if the tool is the only difference, and my fix is procedural rather than clever: the test gets an owner, and every change to the channel during the window goes through them.

Novelty dressed as lift. AI-written variants are often different in style from what your list is used to, and different gets opened once. A four-week window catches some of this decay and a one-send test catches none of it, which is one more reason the runtime is pre-registered rather than cut short at the first good week.

Testing the tool instead of the workflow. The thing you actually run in production is the tool plus the human review around it. Test that. An arm where the AI output ships unedited measures a workflow you would never use, and its result, good or bad, transfers to nothing.

FAQ

How is this different from an ordinary A/B test? Mechanically it is one, and the sizing and significance rules are identical. The difference is what sits in each arm: an A/B test compares two versions of an asset, while an incrementality test compares your process with the tool against your process without it, so the readout prices the tool itself. That framing, plus a pre-set bar derived from the tool’s cost, is what makes the result a renewal decision instead of a curiosity.

Do I need a data scientist to run this? Not for the audience-split version, which is the one I would start with. The two moments to borrow an analyst are geo selection, where you want someone to confirm the holdout region historically tracks the others, and any readout you are about to defend in a budget meeting. Both are short, specific requests.

The tool’s own dashboard already reports incremental lift. Is that enough? A platform measuring its own incrementality is grading its own homework, and the measured gap between self-reported and experimental results is large. I treat platform lift numbers as an upper bound and run my own holdout before any renewal large enough to argue about.

What should I test first? The most expensive claim in your stack, on the channel where you control the split. For most teams that is an email or lifecycle tool, precisely because the list split is the one design a marketer can run alone, end to end, this month.

This walkthrough is part of my measurement series. The proof framework it feeds sits in proving AI ROI without attribution, the sizing math lives in statistical significance for marketing experiments, and the cost side of the renewal decision is in how to calculate AI marketing ROI.