Definition

A/B test.

Also called: Split test, Experiment

A controlled experiment that randomly splits users between variants to measure the causal effect of a change.

A/B tests are the standard way to learn whether a change caused an outcome. In games, monetization tests are the hardest: revenue is heavily skewed, so detecting a modest ARPU change needs large samples, and many important effects (retention, payer behaviour) take time to appear.

Decide the primary metric, the guardrail metrics, the minimum detectable effect and the stopping rule before the test starts.

Calculate it

Whale-Skewed A/B Test CalculatorDetermine true sample size and runtime for A/B tests on skewed monetisation metrics like ARPU and IAP conversion.
Monetization Test Guardrail BuilderWrite the one-page brief a monetization test needs before it starts: target metric, protected metric, acceptable damage and rollback point.

Field notes

When to A/B test, and when not toMost teams under-experiment on pricing and over-experiment on copy. Here's the heuristic I use.
Monetization Optimisation Should State What It Refuses to Damage“Increase ad ARPDAU by 10%” is a valid target. The brief is incomplete until it also says what the team is not willing to damage getting there.

Seeing this number move and not sure what it means for your game?

Bring it to a first call →

Calculator by The Game Scientist ↗

Calculate it

Field notes

Related terms