A/B testing and safe rollout

A school wants to know whether a new maths textbook helps. It gives the new book to half its classes, picked by drawing names from a hat, keeps the old book in the other half, and compares the end-of-term scores. If the new-book classes score higher, the book gets the credit, because the hat made the two groups alike in every other way.

The trial has traps the head teacher learns one at a time. Students in different classes share notes, so the old-book group is not untouched. One class had a flu outbreak and lost a third of its pupils, so the groups are no longer the sizes the hat made them. A teacher who checks the scores every Friday and announces victory the first week the new book is ahead will find victory in a trial where nothing changed. Each trap has a name in the sections below.

A school trial has twenty classes and one reading at the end of term, while an online experiment has a million people and a dashboard that updates every hour. That gap is where the analogy breaks, because it makes the Friday temptation permanent and the arithmetic of how many people are needed the first thing to settle.

An experiment starts with a hypothesis that links the change to an outcome in one sentence. For notifications it might be that sending fewer, better-ranked ones raises the number of people who open the app each day without raising the number who switch notifications off. The outcome is the primary metric, the thing that must not get worse is a guardrail metric, and both are written down before the test starts, along with who is eligible and how long it will run.

The unit of assignment is the thing the hat draws, and for most products it is the person, so one person sees one version every time they come back. The group kept on the current version is called the control and the group given the new one is called the treatment. Assigning per request instead can show one person both versions, and the first can change how they react to the second.

In a marketplace or a social network the versions leak between groups, because a seller in the new group sells to a buyer in the old one. The design then assigns whole cities or whole groups of friends together, which is the school’s shared-notes problem.

Before reading any result, check that the groups are the sizes the assignment should have produced. If a million people were split in half and the groups are 500,000 and 490,000, the gap of 10,000 is about twenty times larger than chance would produce, since random variation in such a split is a few hundred people either way. Something is dropping people from one group. Engineers call this a sample ratio mismatch, and its causes are assignment bugs, lost events and filters that depend on the version, such as a new version that crashes on old phones and loses those users from the logs.

An A/A test, where both groups get the same version, is the cheapest way to find such bugs before the real test, because any difference it reports is a bug by definition. Assignment is also logged on its own, at the moment of the draw, and never reconstructed from later events the version can influence. Analysing only the people who opened a notification lets the new ranker choose who is counted, and the hat’s work is undone.

A change from 10.0 percent to 10.2 percent of people converting, meaning completing the action the product wants, is a rise of 0.2 percentage points and a relative rise of 2 percent. Both descriptions are true of the same effect, so report the one the business will act on and give the denominator. Report an uncertainty interval with it, which is the range the true effect could plausibly be given the noise, and compare the low end of that range with the smallest lift that would pay for shipping the change.

The number of people needed depends on how small the lift is. A common rule of thumb is that each group needs about 16 times the baseline rate times one minus the baseline rate, divided by the square of the lift. For a 10 percent baseline and a 0.2 point lift that is 16 × 0.09 / 0.002², or about 360,000 people per group. If 50,000 eligible people arrive a day, the test needs about 15 days, which also covers two weekends, and that duration is fixed before the test starts.

Checking the dashboard every day and stopping the first time the result looks significant, meaning too large to be chance, is the Friday mistake, and it can turn a 5 percent false alarm rate into one several times higher. Either read the result once at the planned end, or use a sequential method designed to be checked repeatedly, which needs more people for the same certainty.

Three kinds of release look alike and answer different questions. A shadow release sends a copy of every request to the new version and throws its answers away, which tests speed and crashes with no risk to anyone. A canary sends a small share of live traffic, one percent say, to the new version to catch problems at small scale. The randomized experiment measures the effect on people, and it is the only one of the three that can say the change worked.

The release plan sets limits for errors, response time and quality drops at each stage, with a tested way back to the previous version, which is the kept model bundle from the training topic. A winning primary metric is weighed against the guardrails and against effects that take weeks to show, such as people who turn notifications off a month later. A test that was extended to cover those is a different test from one that ended on day 15.

Reading the result before checking the split is the confusion this whole topic exists to prevent. A result from groups that have lost people unevenly is a result about who was lost, and no amount of statistics on it is worth reading.

Stopping early because the number looks good is the Friday mistake again. The dashboard will show a significant result at some point in most tests where nothing changed, and only reading at the planned end date protects against that.

A canary gets confused with an experiment. A canary with no crashes has shown that the version runs, and nothing about whether people are better off.

Quoting an offline metric invites the follow-up about experiments, which takes three or four minutes near the end of the evaluation phase. Name the primary metric and one guardrail, say the unit of assignment and why, give the sample size arithmetic for the lift you expect, and name shadow and canary as the safety steps that come before it.

If the interviewer asks what you do when offline wins and online loses, that is the error analysis topic and the feedback loops topic together, and saying so in one sentence buys the time to answer it properly.