How an A/B test works
A classic test has two versions: the control (A), which is the current experience, and the variant (B), which carries the change. Visitors are assigned at random through a cookie or a server-side binding — once a user lands in a group, they always see the same version.
Once the sample is large enough, the results are compared on a key metric. If the difference is statistically significant (typically p < 0.05, or a probability to be best of 95% or higher), the winner ships as the new default.
CR(A) = 2.1% → CR(B) = 2.5% Lift = +19%
RPV(A) = $6.40 → RPV(B) = $7.50 Lift = +17%
Tip: use RPV (revenue per visitor) as your primary metric — it captures conversion rate and order value at the same time.
Frequentist vs Bayesian: two approaches
| Parameter | Frequentist | Bayesian |
|---|---|---|
| Primary metric | p-value | Probability to be best |
| Stopping early | Not allowed | Allowed |
| Works with MAB | ✗ | ✓ |
| Speed of decision | Slower | Faster when the gap is large |
For e-commerce the Bayesian approach is the better fit: it lets you react sooner and it is natively compatible with automatic traffic allocation (MAB). Gravity Field uses Bayesian statistics by default.
Autopilot: Multi-Armed Bandit (MAB)
Multi-armed bandit is an extension of A/B testing. Instead of a fixed 50/50 split, the algorithm reallocates traffic dynamically toward the winning version while the test is still running. That cuts the conversions lost to the losing variant by 30–50% compared with a classic split.
A/B tests for recommendations and personalization
In e-commerce, A/B tests are most often run not on UI elements but on recommendation strategies: comparing algorithms, widget configurations and the placement of recommendation blocks. The key metric is attributed revenue, measured against an attribution window of 7–14 days.
Common mistakes
- Peek-and-stop: calling the test the moment the numbers look good is the fastest route to a false positive. Across 20 interim checks, the odds of seeing p < 0.05 by chance exceed 60%.
- Several variables at once: change the headline, the button colour and the photo together and you can no longer attribute the result to any of them.
- Too small a sample: at a 1% CR and 10K visitors a day, detecting a 15% MDE takes roughly 17–21 days.
- Ignoring seasonality: a test run in the week before a major sale draws an unrepresentative audience.