Why AI Tools Report Higher ROI Than Reality
Four mechanisms that inflate AI marketing ROI claims - commissioned studies, self-graded attribution, perception gaps, survivorship - and how to verify.
Every AI tool in your stack has a number about itself. A 5x content velocity, a 30% CPA improvement, an ROI multiple with a decimal point for credibility. Meanwhile S&P Global’s enterprise survey found 42% of companies abandoning most of their AI initiatives in 2025, up from 17% a year earlier. Both things are simultaneously true, and the gap between them is not fraud. It is four specific measurement mechanisms, each of which inflates the reported number without anyone lying outright.
I wrote this framework because I kept watching the same meeting: a vendor’s deck says 40%, the team’s own quarter says nothing visible, and everyone concludes the team implemented it wrong. Sometimes they did. More often the 40% was never a number their business could have produced, and knowing the four mechanisms tells you which is which before the renewal, not after.
Mechanism one: the study was commissioned
A large share of the ROI documents in vendor decks are commissioned research, and the researchers say so themselves. Forrester’s own Total Economic Impact policy states that “all TEIs are clearly marked as ‘commissioned’ and cannot be used in Forrester’s syndicated content.” That second clause is the revealing one. Forrester itself firewalls these studies out of its independent research, and my rule follows the same logic: a commissioned study is the vendor’s best case with a methodology section, useful for understanding where value might come from and inadmissible as a forecast of your result. The customers interviewed were chosen by the vendor, and the composite organization was assembled to showcase the product working.
The tell I check first in any TEI-style document is the interview count and the word “composite.” Four happy customers averaged into a fictional company is a case study wearing the costume of a sample.
Mechanism two: the tool grades its own homework
When an AI ad product reports its results through its own attribution, the measurement and the sales pitch share an author. The scale of the resulting bias has been measured. Gordon and Zettelmeyer’s study of fifteen Facebook ad experiments, across 500 million user-experiment observations, found that observational methods, the kind platform dashboards run on, generally overestimate ad effectiveness relative to randomized experiments, off by a factor of three in half the studies. The famous eBay paid search experiments found the same shape earlier: returns “a fraction of conventional non-experimental estimates,” with brand-keyword ads showing “no measurable short-term benefits” at all.
Even the good numbers carry this asterisk. Meta’s economic impact report cites a randomized controlled trial showing $4.52 of return per advertising dollar with its AI tools against $3.71 without, and an RCT is genuinely the right method. It is still Meta’s RCT, on Meta’s selection of campaigns, published because it flattered. I take platform experiments as an upper bound and my own holdout tests as the estimate.
Mechanism three: people report the speedup they expected to feel
The subtlest inflator is honest self-reporting. METR’s 2025 developer study randomized experienced developers into working with and without AI tools across 246 real tasks. With AI they took 19% longer, yet they expected a 24% speedup going in, and after living through the slowdown they still believed the AI had made them 20% faster. The perception gap survived direct personal experience.
Marketing’s own numbers rhyme with that. In Orbit Media’s long-running blogger survey, 92.4% of bloggers now use AI in some part of the process, while the share reporting strong results from their content sits at 13.9%, the lowest in the survey’s history, and AI use correlates with none of the success measures. Adoption is nearly universal, felt productivity is high, and the outcome metric went the other way. Any ROI figure built on “how much time do you feel this saves you” surveys inherits this bias at full strength, which is why my own ROI method only credits saved time that provably became other output.
Mechanism four: the failures write no case studies
The 42% of companies in the S&P data who abandoned their AI initiatives will never appear in a vendor’s results page, a webinar, or a conference talk. The population you hear from is filtered for success before you hear from it. This is ordinary survivorship, and I only note the marketing-specific twist: our industry’s case-study machine is unusually good, so the filter is unusually strong. When every visible data point is a survivor, “typical results” is not a concept the evidence in front of you can support, which is the same reason vendor benchmarks make bad yardsticks for a rollout’s first quarter.
The verification framework
Four mechanisms, four checks. I run these in order on any claim that might influence a budget.
Ask who paid for the number. Commissioned study, platform self-measurement, vendor survey of its own customers, or independent research with named methodology. The answer does not make a claim false. It sets how much weight the claim can bear, and a renewal decision should rest on the last category or on your own data.
Demand the denominator and the baseline. “Saved 10 hours a week” against what process, measured how, for whom. A claim that cannot name its baseline is a feeling with a percent sign, and the pre-AI baseline you capture yourself in week one is worth more than any external figure.
Re-derive it from your own arithmetic. Take the claimed percentage, apply it to your actual volumes, costs and conversion rates, and see if the resulting money is even large enough to detect. A tool promising 20% faster briefs in a team that writes six briefs a month cannot produce a visible quarter no matter how honest the 20% is. Most vendor numbers fail here before any deception is required.
Test small with a control. The only claim that matters is the one your own holdout confirms: a team, a campaign set or a time period without the tool, compared honestly, with the full costs counted, review time and rework included. The full design, from writing the testable claim to reading the result, is my incrementality testing walkthrough. If the effect is real at your scale, it survives this. If it only exists in the dashboard, you have learned which mechanism you were looking at.
FAQ
Are vendors lying about AI ROI? Rarely, and that is what makes the problem durable. Commissioned studies, self-attribution, self-report surveys and survivorship each produce inflated numbers from true inputs. The system tolerates honesty at every step and still outputs a figure your business will not see.
Is a Forrester TEI study worthless? No. It maps where value could come from, which cost categories move, and what the implementation asked of the customers interviewed. I use TEIs as a checklist of things to measure myself and never as the measurement.
What is the single best question to ask a vendor about their ROI claim? “What did the customers who churned measure?” No vendor has a good answer, and watching them handle the question tells you how seriously they take their own evidence.
How do I justify an AI tool without trusting vendor numbers? Baseline first, pilot small, holdout something, count all costs, and report the range honestly. The full sequence is in proving AI ROI without attribution, and it works precisely because none of its inputs come from a deck.
This framework belongs to the measurement topic. The cost side lives in the hidden costs audit, the baseline discipline in the first-90-days benchmarks, and the wider practice starts at the complete guide to AI marketing.