Statistics and A/B Testing Interview Questions for Data Scientists

What is conditional probability?

Conditional probability is the probability of an event A given that event B has happened, written P(A | B) and equal to P(A and B) divided by P(B). It narrows the sample space to the outcomes where B is true. The Conditional Probability lesson works through it with visual examples.

When are two events independent, and is that the same as mutually exclusive?

Two events are independent when P(A and B) equals P(A) times P(B), which means knowing one tells you nothing about the other. Mutually exclusive events cannot happen together, so if both have positive probability they are strongly dependent, because seeing one tells you the other did not happen. The joint and marginal probability lesson covers how joint tables show this.

A test for a condition is 99% sensitive with a 5% false positive rate, and 1% of people have the condition. If someone tests positive, how likely are they to have it?

The probability is about 16.7%, which is much lower than most people guess. In a group of 10,000 people, 100 have the condition and 99 of them test positive, while 9,900 do not have it and 5% of them, or 495, also test positive. Bayes theorem gives 99 divided by 99 plus 495, which is 99 / 594 = 0.167. Read the Bayes Theorem lesson, then solve Bayes Theorem and Bayesian Updating in code.

When would you use a binomial distribution and when a Poisson distribution?

Use a binomial distribution for the number of successes in a fixed number n of independent trials that share a success probability p, such as conversions out of n visitors, with mean np and variance np(1 - p). Use a Poisson distribution for the number of events in a fixed interval at a constant average rate, such as support tickets per hour, where the mean and the variance both equal the rate. A Poisson distribution approximates a binomial one when n is large and p is small. The Distributions lesson compares the common distributions side by side.

What should you know about the normal distribution?

You should know that about 68%, 95% and 99.7% of a normal distribution lies within one, two and three standard deviations of the mean, and that 1.96 standard deviations on each side covers 95%. Sums of independent normal variables are also normal, and the central limit theorem makes sample means approximately normal even when the data is not. The Random Variables lesson covers expectation and variance, which most distribution questions build on.

Would you report the mean or the median for revenue per user?

For a right-skewed metric such as revenue per user, the median usually describes a typical user better, because a few very large spenders pull the mean above it. The mean is still the right number when the business cares about totals, since total revenue equals the mean times the number of users. The descriptive statistics lesson covers mean, median, skewness and spread, and the Percentiles problem has you compute quantiles directly.

What does the central limit theorem say?

The central limit theorem says that the mean of n independent draws from a distribution with finite variance is approximately normal for large n, with mean equal to the population mean and standard deviation σ / √n. It holds whatever the shape of the original distribution, which is why tests on means and proportions work for skewed business metrics. See the Central Limit Theorem lesson and the Sampling Distribution of the Mean problem.

What is the difference between standard deviation and standard error?

Standard deviation measures how spread out individual observations are, while standard error measures how much an estimate such as the sample mean would vary from sample to sample. For a mean the standard error is σ / √n, so collecting four times as much data halves it. The sampling distribution lesson shows the difference interactively, and Standard Error Comparison has you compute it.

How is the law of large numbers different from the central limit theorem?

The law of large numbers says the sample mean gets closer to the population mean as the sample grows, while the central limit theorem describes the shape of the remaining error around that mean. The first tells you an estimate converges, and the second tells you how to put an interval around it. The Law of Large Numbers lesson compares the two directly.

What is bootstrapping and when would you use it?

Bootstrapping estimates the uncertainty of a statistic by resampling the data with replacement many times, computing the statistic on each resample and using the spread of those values. It is useful when there is no simple formula for the standard error, such as for a median or a ratio metric. The resampling lesson explains the method, and Bootstrap Confidence Interval has you implement it.

What is stratified sampling?

Stratified sampling splits the population into groups such as regions or device types and samples from each group separately, usually in proportion to its size. It guarantees every group is represented and reduces variance when the groups differ from each other. The population and sample lesson covers sampling methods, and Stratified Sampling is the matching problem.

How do you interpret a 95% confidence interval?

A 95% confidence interval comes from a procedure that captures the true value in 95% of repeated samples. Any single interval either contains the true value or does not, so saying there is a 95% probability that the true value lies in this particular interval is the frequentist mistake interviewers listen for. The Confidence Intervals lesson explains the coverage idea.

A sample of 100 users has a mean session length of 50 minutes and a standard deviation of 10 minutes. What is the 95% confidence interval?

The 95% confidence interval runs from about 48.04 to 51.96 minutes. The standard error is 10 / √100 = 1, and the interval is 50 plus or minus 1.96 times 1. With 99 degrees of freedom the exact t multiplier is about 1.98, which barely changes the answer. The CLT Confidence Interval problem has you build this in code.

What makes a confidence interval narrower?

A confidence interval gets narrower with a larger sample, a less variable metric or a lower confidence level. Its width scales with 1 / √n, so halving the width needs four times as many observations. That relationship is also why sample size grows quickly as the effect you want to detect gets smaller.

How is a confidence interval different from a Bayesian credible interval?

A credible interval is a range that contains the parameter with a stated probability given the data and a prior, which is the statement people often wrongly make about a confidence interval. A confidence interval makes a statement about the procedure over repeated samples and uses no prior. The Bayesian vs frequentist lesson compares their coverage with an interactive demo.

How do you run a hypothesis test?

You state a null hypothesis and an alternative, choose a significance level such as 0.05 before looking at the data, compute a test statistic, find its p-value and reject the null hypothesis if the p-value is below the significance level. Then you report the effect size and a confidence interval alongside the decision. The Hypothesis Testing lesson walks through each step.

What is a p-value?

A p-value is the probability of seeing a result at least as extreme as the one observed if the null hypothesis were true. It is not the probability that the null hypothesis is true, and a small p-value does not mean the effect is large or important. The P-Values lesson shows the tail area behind the number, and P-Value Analysis has you compute and interpret one.

When do you use a z-test and when a t-test?

A z-test fits when the population standard deviation is known or when you test proportions on large samples. A t-test fits when the standard deviation is estimated from the sample, and its heavier tails account for that extra uncertainty, with n - 1 degrees of freedom for a one-sample test. The two give nearly the same answer for large samples. Read the t-Tests lesson, then solve Z-Test (One Sample) and One-Sample t-Test.

When would you use a chi-square test?

Use a chi-square test for counts in categories, either to check whether observed counts match expected proportions or to test whether two categorical variables are independent in a contingency table. It compares observed and expected counts with the sum of (observed - expected)² / expected. The Chi-Square Test problem has you implement it, and the same test detects the sample ratio mismatch described below.

How would you compare the means of three or more groups?

Compare three or more group means with a one-way ANOVA F-test, which checks whether the variation between groups is large relative to the variation within groups. Running every pairwise t-test instead raises the chance of a false positive with each extra comparison. The ANOVA lesson covers the F statistic and what to do after a significant result.

Should a test be one-sided or two-sided?

A test should be two-sided by default, because a change can make a metric worse as well as better. A one-sided test is defensible only when the direction was fixed before the experiment and a result in the other direction would lead to the same decision as no effect. Switching to one-sided after seeing the data halves the p-value without any new evidence.

What is the difference between a type I error and a type II error?

A type I error is rejecting a null hypothesis that is true, which is a false positive, and its probability is the significance level α. A type II error is failing to reject a null hypothesis that is false, which is a false negative, and its probability is β. The Type I vs Type II Errors lesson has an interactive demo of the trade-off between them.

What is statistical power and what increases it?

Statistical power is the probability of detecting an effect that really exists, equal to 1 - β. It goes up with a larger sample, a larger true effect, a less variable metric and a higher significance level. A common choice is 80% power at a 0.05 significance level, which is a convention for balancing the cost of a longer test against the risk of missing a real effect.

What goes wrong when an experiment is underpowered?

An underpowered experiment misses real effects most of the time, and the results that do reach significance tend to overstate the true effect, because only the lucky high estimates cross the threshold. Planning the sample size before launch is the fix. The power calculator in the Type I vs Type II Errors lesson shows how quickly power drops with a smaller sample.

What does a correlation coefficient measure?

The Pearson correlation coefficient measures the strength and direction of a linear relationship on a scale from -1 to 1. A correlation of zero does not mean the variables are independent, since y = x² on values symmetric around zero has zero correlation and a perfect relationship. The Correlation lesson covers this, and Pearson Correlation has you compute it.

How do you test whether a correlation is statistically significant?

Test a correlation r from n pairs with the statistic t = r√(n - 2) / √(1 - r²), which follows a t distribution with n - 2 degrees of freedom when the true correlation is zero. With large samples even a tiny correlation becomes significant, so also report its size. The Correlation Significance problem implements this test.

Why does correlation not imply causation, and how do you get at causation?

Correlation does not imply causation because a third variable can drive both, the causal direction can run the other way, or the pattern can come from how the data was selected. A randomized experiment breaks those links by assigning the treatment at random, which is why product teams run A/B tests. The A/B testing lesson explains how randomization makes a difference causal.

What are the assumptions of linear regression?

Linear regression assumes a linear relationship between the features and the target, independent errors, constant error variance and, for the usual p-values and intervals, normally distributed errors. Strongly correlated features do not bias predictions, but they make individual coefficients unstable and hard to interpret. A coefficient is the expected change in the target for a one-unit increase in that feature with the other features held fixed. The Linear Regression Closed Form problem has you fit the model with the normal equation.

How do you choose the primary metric for an A/B test?

Choose a primary metric that reflects the goal of the change, moves within the length of the test and has low enough variance to detect a realistic effect. Add guardrail metrics such as latency, cancellations or revenue that must not get worse. A long-term metric such as retention after 90 days is often the real goal, so teams pick a short-term metric that predicts it. The A/B testing lesson covers metric choice with a worked case study.

What unit should you randomize on?

Randomize on the unit the change affects, which is usually the user, so the same person always sees the same version. Randomizing by session or page view lets one person see both versions, which contaminates the comparison. The unit of analysis should match the unit of randomization, because analyzing page views when you randomized users makes the standard error too small.

How do you calculate the sample size for an A/B test on a conversion rate?

The sample size per group for a two-sided test on two proportions is n = (z_α/2 + z_β)² × (p₁(1 - p₁) + p₂(1 - p₂)) / δ², where δ is the smallest difference worth detecting. You need the baseline rate, that minimum detectable effect, the significance level and the power. The worked example below runs the numbers, and A/B Test Conversion Rates has you analyze the result in code.

What is an A/A test and why would you run one?

An A/A test splits users into two groups that see the same experience, to check that the assignment, logging and analysis pipeline works. At a 0.05 significance level, about 5% of A/A comparisons should come out significant by chance, and a much higher rate points to a bug such as a wrong variance formula.

What is peeking and why is it a problem?

Peeking means checking the result of a running test repeatedly and stopping as soon as it looks significant. Each look is another chance for noise to cross the threshold, so a team that peeks daily and stops at the first p-value under 0.05 has a false positive rate well above 5%. Fix the sample size in advance, or use a sequential testing method built for repeated looks.

What are novelty and primacy effects?

A novelty effect is a short-term lift that happens because users try something new and fades as the change becomes familiar. A primacy effect is the reverse, where existing users resist a change at first and adapt later. Check for both by plotting the effect by day and by comparing new users with returning ones.

What is a sample ratio mismatch and how do you detect it?

A sample ratio mismatch happens when the groups end up with a different split than the experiment was designed for, which usually means a bug in assignment or logging. Detect it with a chi-square test on the group counts. If a 50/50 test has 25,500 users in one group and 24,500 in the other out of 50,000, each group is 500 away from the expected 25,000, so the statistic is 500² / 25,000 + 500² / 25,000 = 10 + 10 = 20 with one degree of freedom, and the p-value is about 0.000008. Investigate a mismatch like that before reading any metric, and practise the test with Chi-Square Test.

What is the multiple testing problem in A/B tests?

The multiple testing problem is that checking many metrics or segments raises the chance that at least one looks significant by luck. With 20 independent metrics each tested at 0.05, the chance of at least one false positive is 1 - 0.95²⁰, which is about 64%. A Bonferroni correction tests each metric at 0.05 / 20 = 0.0025, and the Benjamini-Hochberg procedure controls the false discovery rate with less loss of power. Naming one primary metric before launch avoids most of the problem.

How do network effects break an A/B test?

Network effects break an A/B test when users in one group affect the outcomes of users in the other, for example in a marketplace where treated buyers take inventory from control buyers, or in a social product where a feature changes what friends see. The groups are then no longer independent, and the measured difference is biased. Common fixes are randomizing by cluster, such as a city or a friend group, running geographic experiments, or alternating the treatment over time in a switchback test. The 60-minute A/B Test Analysis assessment covers sample size and spillover bias.

How much statistics do you need for a data scientist interview?

You need enough statistics to compute and explain probability, distributions, standard errors, confidence intervals, hypothesis tests and power, and to apply them to an A/B test. Most of the questions ask you to pick the right method for a situation, run it on small numbers and say what the result means for a decision.

Should you use a t-test or a z-test for an A/B test?

For an A/B test on a conversion rate, the usual choice is a two-proportion z-test, because the variance of a proportion follows from the rate itself and samples are large. For a continuous metric such as revenue per user, a two-sample t-test is the standard choice, and the Welch version is safer when the two groups have different variances. With thousands of users per group the t and z results are almost identical.

How long should an A/B test run?

An A/B test should run until it reaches the sample size you planned before launch, and for at least one full week so that every day of the weekly cycle is covered. Running in whole weeks also stops a weekend or weekday effect from biasing the result. Stopping as soon as the p-value drops below 0.05 inflates the false positive rate.

How do you explain a p-value to a non-technical stakeholder?

Explain a p-value as how surprising the result would be if the change had no real effect. A p-value of 0.01 means that a difference this large or larger would appear about 1 time in 100 by chance alone if the change did nothing. Then move straight to the effect size and its confidence interval, because those are what the decision depends on.

Is there a timed practice assessment for statistics and A/B testing?

Yes, TensorTonic has two timed assessments for these rounds in its Data Scientist track. Applied Probability and Statistics runs for 45 minutes, from Bayes posteriors to a two-proportion z statistic, and A/B Test Analysis spends 60 minutes on sample size and spillover bias.