When a team ships a new ranking model to half of its users and keeps the old one for the other half, it ends up with two conversion counts. Someone then has to decide whether the difference between them is real or noise. The experiments topic of the ML System Design course says to read the size of the effect together with its uncertainty, and the standard tool for a click or conversion metric is the two-proportion z-test. This problem asks you to compute it.

You are given four integers and a significance level. The control group had control_n users of whom control_conversions converted, and the treatment group had treatment_n users of whom treatment_conversions converted. Compute the conversion rate of each group as conversions divided by users. The lift is the treatment rate minus the control rate.

For the test statistic, pool both groups into one rate, which is total conversions divided by total users. The pooled standard error is the square root of pooled rate times one minus pooled rate times the sum of one over control_n and one over treatment_n. The statistic z is the lift divided by the pooled standard error, and the two-sided p-value is the probability that a standard normal variable is at least as far from zero as z. If the pooled standard error is zero, which happens when nobody or everybody converted in both groups, set z to 0.0 and the p-value to 1.0.

For the confidence interval, use the unpooled standard error, which is the square root of control rate times one minus control rate over control_n, plus treatment rate times one minus treatment rate over treatment_n. Multiply it by the standard normal quantile at one minus alpha over two, which the statistics module gives you as NormalDist().inv_cdf, and add and subtract the result from the lift to get ci_high and ci_low. The result is significant when the p-value is below alpha.

Return a dictionary with the keys lift, z, p_value, ci_low, ci_high and significant. The first five are floats and the last is a bool. Lift, z and p_value are checked to within one millionth and the interval bounds to within one ten-thousandth.

Worked example. Control had 2400 conversions out of 20000 users, a rate of 0.12, and treatment had 2600 out of 20000, a rate of 0.13, so the lift is 0.01. The pooled rate is 5000 divided by 40000, which is 0.125, and the pooled standard error is about 0.003307, so z is about 3.024 and the two-sided p-value is about 0.0025. The unpooled standard error is about 0.003307 as well, and with alpha equal to 0.05 the quantile is about 1.96, so the interval runs from about 0.0035 to about 0.0165. The p-value is below 0.05, so the result is significant.