Where machine learning genuinely improves conversion optimisation, where it is repackaged testing, and how to avoid fooling yourself with bad statistics.
AI conversion rate optimization covers two very different things. One is a genuine statistical improvement over fixed-split A/B testing. The other is conventional testing with a language model writing the variants.
Both get sold under the same label. The first changes results measurably; the second mostly changes how quickly you can produce copy that still needs testing. Knowing which you are buying determines whether the spend returns anything.
The part that genuinely works: adaptive allocation
Standard A/B testing splits traffic 50/50 and waits for significance. That is statistically clean and wasteful — half your traffic keeps going to the losing variant long after the data suggests which one it is.
Multi-armed bandit algorithms shift traffic toward better-performing variants continuously while preserving enough exploration to stay honest. Thompson sampling is the common implementation: sample from each variant's posterior distribution, allocate to whichever sampled highest, update as data arrives.
The practical gain is reduced regret — less traffic spent on losers during the test. On a page converting at 3% with a variant that turns out 15% better, a bandit typically captures meaningfully more conversions during the test period than a fixed split, because allocation shifts as evidence accumulates.
Where bandits fit well:
Many variants. Testing eight headlines with a fixed split means each gets 12.5% of traffic and the test runs forever. Bandits concentrate traffic efficiently.
Short-lived opportunities. Seasonal campaigns where a clean two-week test is not available.
Continuous optimisation where the objective is cumulative conversions rather than a clean causal estimate.
Where they fit badly:
When you need a defensible effect size. Bandits optimise outcomes, not inference. If you need to state "this change produced +12%" with a confidence interval, run a fixed split.
Delayed conversions. If conversion happens days after the visit, the feedback loop is too slow for the algorithm to allocate sensibly.
Low traffic. Below roughly 1,000 conversions per variant, the algorithm is reacting to noise.
Contextual bandits extend this by conditioning on visitor attributes — traffic source, device, geography, returning status. Instead of "which headline is best," the question becomes "which headline is best for this visitor type." This is real personalisation and it works, given sufficient traffic per context. That last condition is the constraint: slicing by six attributes across four variants creates a lot of thin cells, and thin cells produce confident nonsense.
The part that is oversold: generated variants
Language models produce headline and copy variants quickly and cheaply. That is genuinely useful — variant production was a bottleneck and now is not.
What it does not do is know which will convert. Generated copy is a hypothesis, exactly like human-written copy, and requires the same testing to validate. Tools that generate and deploy without measurement are not optimising anything; they are changing your site and hoping.
The failure pattern is volume without discipline: generate forty variants, test them all simultaneously, declare a winner. With forty comparisons at conventional significance thresholds you should expect roughly two false positives from pure chance. Teams then ship the noise and record a win that does not replicate.
If you generate variants at scale, you must correct for multiple comparisons — Bonferroni, Benjamini-Hochberg, or a sequential testing method built for it. Most tools in this category do not, and their reported win rates should be read accordingly.
Predictive scoring, and its honest limits
Models that score visitors on purchase likelihood have real applications: prioritising live-chat outreach, calibrating discount depth, deciding retargeting spend.
Two failure modes recur.
Self-fulfilling feedback loops. Score a visitor low, show them nothing, they do not convert, the model records confirmation. The model is now training on its own decisions. Preventing this requires holding out a random control that receives the default experience regardless of score — permanently, not just at launch.
Drift. A model trained on last quarter's traffic degrades as traffic mix, seasonality, and competitor behaviour change. Without scheduled retraining and monitoring on a holdout, degradation is silent. The model keeps producing confident scores that are progressively less connected to reality.
Session replay analysis: the quiet win
The most reliably valuable AI application in CRO gets the least attention: clustering session recordings and behavioural data to surface friction.
Manual replay review does not scale — nobody watches 10,000 sessions. Clustering rage clicks, form abandonment points, scroll dead-ends, and navigation loops turns that into a ranked list of "here is where people struggle, ordered by how many."
This is unglamorous and it works, because it produces *hypotheses grounded in observed behaviour* rather than variants generated from a prompt. The subsequent test is conventional. The AI contribution is finding the problem, which is the genuinely hard part.
Running this without deceiving yourself
Fix measurement first. Most CRO programmes fail on instrumentation, not algorithms. If conversion tracking is inconsistent, cross-device attribution is broken, or bot traffic is unfiltered, no method helps. Verify tracking against a known-good source before optimising anything.
Define one primary metric. Not "conversions and engagement and time on page." One metric, chosen in advance, that the test is judged on. Everything else is secondary and cannot be promoted to primary after the fact.
Set duration before starting. Compute required sample size from baseline rate and minimum detectable effect. Run full weeks to avoid day-of-week composition effects. Do not stop early because significance appeared — with continuous monitoring, that guarantees false positives at a rate far above your nominal threshold.
Hold out a control. A permanent slice receiving the unoptimised experience. Without it you cannot distinguish genuine improvement from seasonality, traffic-mix change, or model drift.
Expect most tests to fail. Mature programmes see roughly 10–30% of tests produce a real, replicating win. A tool reporting 80% wins is measuring something other than causal impact, and you should ask what.
What to ask a vendor
Four questions that separate substance from packaging:
What algorithm allocates traffic, and can we see the parameters? "Proprietary AI" is not an answer. Thompson sampling, UCB, epsilon-greedy — these are known methods with known properties.
How do you correct for multiple comparisons? If they do not understand the question, their reported win rates are inflated by construction.
Do you maintain a permanent holdout? Without one, attribution of improvement is assertion rather than measurement.
What happens to models on traffic-pattern change? Retraining cadence and drift monitoring should have specific answers.
Where to start
Sequence matters more than tooling:
Verify measurement. Instrumentation correctness before anything else.
Find friction with behavioural clustering. Let observed struggle generate hypotheses.
Test the biggest hypothesis conventionally. Fixed split, predetermined duration, one primary metric.
Introduce bandits when you have many variants and enough traffic to support them.
Add contextual personalisation only when each context has real volume.
Most organisations get the majority of their available gain from steps one through three. Steps four and five matter at scale, and adopting them before the foundation is right produces confident measurements of nothing.
Frequently asked questions
Does AI actually improve conversion rates? Adaptive traffic allocation genuinely reduces waste during testing, and behavioural clustering genuinely finds friction faster than manual review. AI-generated copy is a productivity gain for variant production, not a source of validated lift — it still requires testing.
Bandits or A/B tests? A/B tests when you need a defensible effect size for a decision. Bandits when you have many variants, limited time, and care about cumulative conversions rather than clean inference. They answer different questions.
How much traffic is needed? Roughly 1,000 conversions per variant for reliable results. Below that, both bandits and fixed splits mostly measure noise — bandits just do it with more confidence, which is worse.
Why do vendor win rates look so high? Usually uncorrected multiple comparisons, early stopping on observed significance, or no holdout control. Mature programmes see 10–30% of tests produce replicating wins. Substantially higher figures indicate a methodology problem.
What is the most common mistake? Optimising before measurement is trustworthy. Broken attribution, unfiltered bot traffic, and inconsistent event tracking invalidate every downstream conclusion regardless of method sophistication.
Further reading
Related
Written by
Moueen Togarvi
Founder & CEO at Voquarn Code, focused on product engineering, search growth, and practical AI systems.