Machine Learning Math Cheat Sheet by Module

Variance is the average squared distance from the mean, and the standard deviation is its square root, in the same units as the data.

The IQR is the spread of the middle half of the data, and points more than 1.5 IQR beyond the quartiles are flagged as outliers.

Dividing by n minus 1 makes the sample variance an unbiased estimate of the population variance.

The spread of the sample mean shrinks with the square root of the sample size.

A sample proportion varies around the true rate p by about this amount.

The mean of many independent draws with finite variance is approximately normal whatever the original distribution, and the second argument here is a variance.

The z-score counts how many standard errors the sample mean sits from the population mean.

Use z when the population standard deviation is known and t with n minus 1 degrees of freedom when it is estimated, where z* is 1.645, 1.96 and 2.576 for 90, 95 and 99 percent.

This is the normal-approximation interval for a rate such as a click-through rate.

The test statistic measures how far the data sit from the null value, in standard-error units.

Chi-square compares observed and expected counts, with k minus 1 degrees of freedom for goodness of fit and (r minus 1)(c minus 1) for a test of independence.

The p-value is the probability of a result at least this extreme if the null hypothesis is true, where Φ is the standard normal CDF.

Testing m hypotheses at α/m each keeps the chance of any false positive at or below α.

α is the false-positive rate, β is the false-negative rate and power is the chance of detecting a real effect.

The t statistic replaces σ with s, so its reference distribution has heavier tails for small samples.

The pooled rate gives the standard error under the null hypothesis that both variants convert equally.

Halving the lift you want to detect roughly quadruples the users needed in each group.

F compares the variation between group means with the variation within groups, for k groups and N observations.

r measures linear association on a scale from minus 1 to 1 and says nothing about causation.

Spearman is Pearson computed on ranks, so it captures any monotonic relationship, and dᵢ is the difference between the two ranks of item i.

Resample the data with replacement many times, recompute the statistic each time and keep the middle 1 minus α of the results.

The model is trained k times, scored each time on the one fold it did not see, and the k scores are averaged.

MLE picks the parameters that make the observed data most probable, and the log turns the product of probabilities into a sum.

The Gaussian MLE of the variance divides by n, so it is slightly biased low.

Maximising a Gaussian likelihood is the same as minimising squared error, which is why MSE is the default regression loss.

Bayesian inference updates a prior belief about the parameters with the likelihood of the observed data.

MAP adds a log-prior to the log-likelihood, and a zero-mean Gaussian prior on the weights gives L2 regularisation.

Any joint distribution factors into a chain of conditionals, which is exactly how a language model scores a sentence.

Two events are independent when the joint probability factors, which for P(B) above zero means knowing that B happened does not change the probability of A.

Naive Bayes assumes the features are independent given the class, so the likelihood factors into one term per feature.

The CDF accumulates probability, and for a continuous variable the PDF f is its derivative.

The binomial counts successes in n independent trials that each succeed with probability p, and n equal to 1 is the Bernoulli case.

The Poisson models counts of rare events in a fixed window at an average rate λ.

About 68, 95 and 99.7 percent of the mass lies within 1, 2 and 3 standard deviations of the mean.

The exponential models the waiting time between Poisson events and has no memory of how long you have already waited.

Expectation is the probability-weighted average, with an integral in place of the sum for continuous variables, and it is linear even when the variables are dependent.

Shifting a variable leaves its variance unchanged, and scaling it multiplies the variance by the square of the factor.

The covariance term vanishes only when X and Y are uncorrelated, for example when they are independent.

Summing or integrating a joint distribution over one variable leaves the distribution of the other.

Covariance is positive when the variables tend to move together and zero when they are uncorrelated.

With every column of the data centred, the diagonal holds the variances and the off-diagonal entries hold the covariances.

Transforming the data linearly transforms its covariance on both sides.

This distance measures how far a point is from the mean in units that account for the scale and correlation of the features.

Its contours of equal density are ellipses whose shape and tilt are set by the covariance matrix.

Bayes turns the probability of the evidence given a hypothesis into the probability of the hypothesis given the evidence.

The evidence term in Bayes’ theorem is the sum over every hypothesis that could have produced the evidence.

The sample average converges to the true mean, which is why the average loss over a minibatch estimates the full training loss.

A non-negative variable rarely exceeds a large multiple of its mean.

For any distribution with finite variance, at most 1/k² of the probability lies k or more standard deviations from the mean.

Averaging random samples estimates an expectation, and four times as many samples halve the error.

The next state depends only on the current one, so the distribution over states evolves by repeated multiplication with the transition matrix.

The stationary distribution, written here as a row vector, is unchanged by one more step of the chain.

The L1 norm sums absolute values, the L2 norm is Euclidean length and the L-infinity norm is the largest absolute entry.

Cosine similarity compares direction only and ignores length, which is why it is the usual score for comparing embeddings.

Each output entry is a row of A dotted with a column of B, at a cost of m times n times p multiply-adds.

Transposing a product reverses the order, and matrix multiplication does not commute.

Attention is two matrix products around a row-wise softmax, scaled by the square root of the key dimension.

A square matrix is invertible exactly when its determinant is not zero.

The determinant is the factor by which the linear map scales volume, and inverting a product reverses its order.

Orthogonal matrices rotate or reflect vectors without changing their lengths or the angles between them.

Subtracting every earlier direction from each new vector and normalising the result builds an orthonormal basis.

This is the closest point to a on the line through b.

Least squares picks the weights whose residual is orthogonal to every column of X.

P maps any vector onto the column space of X, and applying it twice changes nothing.

Adding λI makes the system invertible for any λ above zero and shrinks the weights toward zero.

LU solves a square system with two triangular solves, and QR solves least squares stably without forming the product of A transpose and A.

An eigenvector keeps its direction under A and is only scaled by its eigenvalue λ.

The eigenvalues of a square matrix add up to its trace and multiply to its determinant.

A real symmetric matrix always has real eigenvalues and an orthonormal set of eigenvectors.

For a symmetric matrix, positive definiteness holds exactly when every eigenvalue is positive.

Every symmetric positive definite matrix factors into a lower-triangular matrix times its own transpose.

Any matrix acts as a rotation or reflection, a scaling along the axes and another rotation or reflection, and the singular values are the square roots of the eigenvalues of A transpose A.

Keeping the top k singular triplets gives the closest rank-k matrix, which is the idea behind the low-rank updates in LoRA.

The pseudoinverse gives the minimum-norm least-squares solution even when A is not invertible.

The principal components are the top eigenvectors of the covariance matrix, and projecting the centred data onto the first k of them reduces the dimension.

Each component’s share of the total variance tells you how many components to keep.

For a symmetric A the gradient of the quadratic form simplifies to 2Ax.

Setting this gradient to zero gives the normal equations.

Backpropagation through a matrix multiply is two more matrix multiplies with transposes.

Comparing this numerical estimate with an analytic gradient is the standard way to test backpropagation code.

The gradient stacks every partial derivative into one vector that points in the direction of steepest increase.

The rate of change along a unit direction u is largest when u points the same way as the gradient.

Derivatives of composed functions multiply along a path and add across paths.

The sigmoid derivative peaks at 0.25, which is one reason deep stacks of sigmoids suffer from vanishing gradients.

The Jacobian is the best linear approximation of a vector-valued function near a point.

The Hessian holds the second derivatives and describes the curvature of the loss.

Near a point, a smooth function looks like a quadratic, which is the basis of Newton’s method.

The area under a curve is approximated by n trapezoids of equal width h, and the same sum with uneven widths is how ROC AUC is computed from a list of points.

AUC is the area under the ROC curve, and it equals the chance that a random positive scores above a random negative, with ties counted as half.

Gradient descent is a sequence of small discrete steps along this field of negative gradients.

The type of a critical point is set by the signs of the Hessian’s eigenvalues, and the test is inconclusive when some of them are zero.

The error at each layer is the next layer’s error sent back through the transposed weights and scaled by the local slope of the activation.

Each weight gradient is the outer product of the layer’s error signal and the layer’s input.

Each step moves the parameters a small distance, set by the learning rate, against the gradient.

Clipping rescales the whole gradient when its norm exceeds c, so one bad batch cannot blow up training.

A twice-differentiable function is convex exactly when its Hessian is positive semidefinite everywhere.

With a step no larger than one over the smoothness constant L, gradient descent never increases the loss.

The minibatch gradient is an unbiased but noisy estimate of the full gradient, where a batch of 1 is plain SGD and a batch of the whole dataset is batch gradient descent.

The learning rate falls smoothly from its peak to its floor over T steps.

Warmup ramps the learning rate up over a fixed number of warmup steps so that early updates do not destabilise training.

Momentum accumulates a running velocity, so in a steady direction the effective step approaches η/(1 minus β), which is ten times η at β = 0.9.

Nesterov evaluates the gradient at the look-ahead point, which damps overshooting.

Each parameter’s step shrinks with its total squared gradient so far, which can stall long runs.

RMSProp replaces the running sum in AdaGrad with a moving average, so the step size stops shrinking forever.

Adam keeps a moving average of the gradient and a moving average of its element-wise square.

Bias correction undoes the pull toward zero in the first steps, and the usual defaults are β₁ = 0.9, β₂ = 0.999 and ε = 1e-8.

AdamW applies weight decay directly to the weights instead of adding it to the gradient that Adam rescales.

SAM looks for weights whose whole neighbourhood has a low loss, which favours flat minima.

With this penalty every SGD step shrinks each weight by a constant factor, which is why it is also called weight decay.

The L1 penalty pushes small weights to exactly zero, and its proximal step is soft thresholding with τ equal to the learning rate times λ.

During training each unit is dropped with probability p and the survivors are scaled up, so nothing changes at inference.