What a holdout is for
A standard A/B test measures the effect of one specific change over a short period. A real personalization programme is dozens of consecutive experiments over a year. Summing their results naively does not work: changes interact, some effects decay, the novelty of the first weeks fades.
A holdout group solves that. It is a slice of the audience frozen in the original state for the whole period. Comparing its metrics against everyone else six months later gives an honest answer to the question of what the personalization programme actually delivered.
How a holdout works in practice
Traffic
├── 90% — regular users
│ receive every personalization change
│ participate in A/B tests
└── 10% — holdout group
receive no personalization
participate in no tests
see the baseline site for the whole period
After six months:
Holdout RPV: 4.20
Main base RPV: 4.87
Aggregate effect: +16%
Technical requirements
Sticky assignment: a user who lands in the holdout has to stay there the whole time. Switching between groups destroys the integrity of the measurement.
Isolation from every experiment: if the holdout is accidentally swept into a few A/B tests, the data is spoiled. The system has to check holdout membership before enrolling anyone in any test.
Sufficient size: with a 5% holdout and a 2% baseline conversion rate, detecting a 5% cumulative effect can take six to twelve months depending on traffic.
Holdout versus a continuous bandit
A multi-armed bandit continuously reallocates traffic toward the better variation, which is incompatible with a holdout in the classic sense. That is a genuine trade-off: the bandit monetises the winner faster, and gives up the ability to measure the long-run accumulated effect honestly.
Tip: if an organisation is running a large personalization programme, a holdout group is worth the potential revenue cost on 5–10% of traffic. It is what lets you prove the real return of the whole programme to a CFO a year later, rather than presenting a pile of disconnected tests.