A/B Testing
Aussi : Split Testing, A/B Test, Online Controlled Experiment
A/B testing is a controlled experiment that compares two versions of something (A and B) by splitting traffic randomly to learn which performs better on a chosen metric.
What It Is
A/B testing (also called split testing) is a controlled online experiment in which two or more variants are shown to randomly assigned groups of users at the same time. One group sees the control (version A, usually the current experience), and the other sees the variant (version B, the proposed change). By comparing a predefined metric between groups, you can attribute differences in behavior to the change itself rather than to chance or external factors.
The random assignment is the key. Because users are split randomly, the two groups are statistically similar in every other respect, so any meaningful difference in outcomes can be linked to the variant.
Why it matters
- Causal evidence: Unlike before/after comparisons, A/B testing isolates cause and effect, reducing the risk of acting on correlation or seasonal noise.
- Lower risk: Changes are validated on a fraction of traffic before a full rollout.
- Shared language: It gives marketing, product, and data teams an objective basis for decisions instead of opinion or hierarchy.
- Compounding gains: A steady cadence of small validated wins often outperforms occasional large redesigns.
How it is used in practice
1. Form a hypothesis: State the expected effect, for example "A shorter checkout form will increase completion rate."
2. Choose a primary metric: Pick one decision metric (conversion rate, revenue per visitor, click-through) plus guardrail metrics.
3. Compute sample size: Use the baseline rate, the minimum detectable effect, and a significance level to decide how long to run.
4. Randomize and run: Split traffic, run until the planned sample is reached, and avoid stopping early just because results look good.
5. Analyze: Check statistical significance and the confidence interval, then decide to ship, iterate, or discard.
Concrete Example
An e-commerce CMO tests a green "Buy now" button (B) against the current blue button (A). Traffic is split 50/50 across 40,000 visitors. Version A converts at 4.0 percent, version B at 4.6 percent. With a p-value below 0.05, the team concludes the lift is unlikely to be random and rolls out the green button to all users.
Common Pitfalls
- Peeking at results and stopping early inflates false positives.
- Too many metrics raise the chance of a spurious "winner."
- Underpowered tests that run too short cannot detect real effects.