Experimentation is the scientific method applied to business. It is the only reliable way to establish causality, to know not just that two things are correlated, but that one causes the other.
Organizations that run rigorous experiments make better decisions and compound their learning over time. Those that don't run experiments make decisions based on intuition, correlation, and hope.
In 2011, JCPenney's CEO Ron Johnson eliminated promotional pricing, convinced (by intuition and some data correlation) that customers preferred "fair and square" everyday low prices. Revenue collapsed 25% in one year. The company never recovered.
The data was available. The failure was in causal inference: he confused customer behavior under promotional pricing with customer preference. Had JCPenney run a controlled experiment (test "everyday low prices" in 100 stores, maintain promotional pricing in 100 comparable stores), the result would have been clear before a company-wide rollout.
This is the cost of skipping experimentation.
A/B Testing Best Practices
Knowledge check
1. Why is experimentation considered the only reliable way to establish causality rather than mere correlation?
2. The JCPenney example primarily illustrates which conceptual failure?
3. Why does the lesson insist on defining a hypothesis before looking at results?
4. Select ALL practices that are essential to sound experiment design according to the lesson.
Select all the correct answers.
5. Select ALL statements that correctly describe the multiple comparison problem and how to handle it.
Select all the correct answers.
Hypothesis first: Define what you're testing and what you expect before looking at results. "We believe changing the CTACTAA button, link, or message that prompts users to take a specific action such as sign up, buy, download, or learn more.View full definition → from 'Sign Up' to 'Start Free Trial' will increase conversion by 15% for first-time visitors." A hypothesis forces clarity about the mechanism, not just the outcome.
Randomization: Test and control groups must be randomly assigned. Self-selection (letting users choose which experience they see) invalidates the experiment, the groups will be systematically different.
Statistical power: Before running the experiment, calculate the minimum sample size needed to detect the effect you care about. Running a test on 100 users to detect a 1% conversion improvement is statistically impossible, you'll never have enough power to be confident.
Multiple comparison problem: Run 20 tests, and one will appear statistically significant by chance. Use Bonferroni correction or FDR (False Discovery Rate) control when running many simultaneous tests.
Decision thresholds: Decide before the experiment what statistical confidence (typically 95%) and practical significance (minimum detectable effect) are required before shipping. Don't change these after seeing results, that's p-hacking.
Running occasional A/B tests manually is fine. Scaling to hundreds of simultaneous experiments requires a platform:
Feature flagging: Infrastructure to show different experiences to different users without code deployments. Tools: LaunchDarkly, Unleash (open-source), Split.io.
Assignment logging: Track which experience each user saw, with timestamp. Required for analysis.
Statistical analysis pipeline: Automated calculation of p-values, confidence intervals, and effect sizes. Reduces human error in analysis.
Decision workflow: Structured process for reviewing results and making ship/don't-ship decisions.
Airbnb, Netflix, and Uber built internal experimentation platforms before this was a product category. The open-source Wasabi (Intuit) and commercial options (Optimizely, VWO) now make this accessible to organizations without platform engineering resources.
Traditional A/B tests have a fixed duration. During the test, you're sending traffic to a potentially inferior variant, incurring the cost of the experiment.
Multi-armed bandit algorithms address this: they continuously adjust traffic allocation to send more traffic to the better-performing variant, while still exploring alternatives. As evidence accumulates, the winning variant gets progressively more traffic.
Use cases: content recommendation, ad creative optimization, pricing experiments. Not suitable for: experiments where user experience consistency matters (switching between variants mid-session is confusing).
1. Pourquoi la randomisation est-elle essentielle dans un A/B testA/B testA/B testing is a controlled experiment that compares two versions of something (A and B) by splitting traffic randomly to learn which performs better on a chosen metric.View full definition → ?
A) Elle accélère l'exécution du test
B) Elle garantit que les groupes test et contrôle sont similaires, éliminant les biais de sélection qui invalideraient les résultats
C) Elle réduit le coût du test
D) Elle augmente automatiquement la taille de l'échantillon
Réponse: B
2. Qu'est-ce que le "multiple comparison problem" dans l'expérimentation ?
A) Difficulté à comparer deux produits différents
B) Le fait que lancer 20 tests simultanément garantit statistiquement qu'environ 1 apparaîtra comme significatif par hasard, nécessitant une correction statistique
C) Problème technique lié à plusieurs bases de données
D) Impossibilité de tester plus de 2 variantes à la fois
Réponse: B
3. Quelle est la principale différence entre un A/B testA/B testA/B testing is a controlled experiment that compares two versions of something (A and B) by splitting traffic randomly to learn which performs better on a chosen metric.View full definition → classique et un algorithme multi-armed bandit ?
A) Le bandit multi-bras est plus précis statistiquement
B) Le bandit multi-bras ajuste continuellement la répartition du trafic vers la meilleure variante pendant le test, réduisant le coût de l'expérimentation
C) Le A/B testA/B testA/B testing is a controlled experiment that compares two versions of something (A and B) by splitting traffic randomly to learn which performs better on a chosen metric.View full definition → classique est automatiquement plus rapide
D) Le bandit multi-bras nécessite plus de données
Réponse: B
These actions are compiled in the role's Playbook.