A/B testing and safe rollout

An experiment begins with a hypothesis connecting the proposed change to an expected outcome. For notification ranking, that might be whether selecting fewer notifications improves useful engagement without increasing opt-outs. Define the main outcome and guardrail metrics, which track effects that must remain acceptable, along with who is eligible and how long the result will be observed.

User-level assignment can keep one person’s experience consistent across visits. Request-level assignment may be appropriate for some isolated interventions, but can create carryover when earlier recommendations affect later behavior. Marketplace or social-network interactions can also let treatment affect control. Choose the unit based on the interference risk rather than habit.

Comparing the observed group sizes with the intended allocation helps check whether assignment and measurement worked as expected. A sample ratio mismatch means the sizes differ more than random variation would explain, which can arise from assignment bugs, missing events, or treatment-dependent filtering. An A/A test, where both groups receive the same version, can help check the pipeline before testing a change, although each subsequent experiment still needs validation.

Log assignment independently from downstream events that the treatment can influence. If you only analyze people who clicked a notification, the new ranker has helped select who enters the analysis. That can destroy the comparability created by randomization.

An illustrative conversion change from 10.0% to 10.2% is a 0.2 percentage-point increase and a 2% relative increase. Those are different descriptions of the same effect. Report the denominator and an uncertainty interval, then compare the effect with the smallest change worth the cost of deployment.

Estimate the required sample size using baseline variance and a meaningful effect size. Run long enough to cover relevant behavior cycles and label delays. Repeatedly checking a conventional fixed-horizon significance test and stopping at a favorable moment inflates false positives; use a predetermined analysis or a valid sequential method.

Shadow evaluation sends copies of requests to the candidate without using its answers for the user, which helps check response times and output differences. A canary release sends a small share of live traffic to a new version to limit exposure while checking its operation. A randomized experiment measures the effect on users, so each step needs a clear purpose and its own conditions for stopping.

The release plan needs limits for errors, response time, and serious quality regressions, together with a tested way to return to the previous version. A positive main metric still needs to be weighed against guardrail failures and effects that emerge over time. To make such a release possible, we next examine how a serving system prepares and returns predictions under live traffic.