What Maths Do You Need for Machine Learning, and in What Order?

What maths do you need for machine learning?

Machine learning needs four areas of maths, best learned in this order. Linear algebra covers vectors, matrices and matrix multiplication. Calculus covers derivatives, gradients and the chain rule. Probability and statistics cover distributions, expectation and likelihood. Optimization covers gradient descent and its variants. Learn each area until you can implement it in code, then go deeper only where your goals require it.

Information theory adds the language of losses such as cross-entropy. For most practical ML work, an undergraduate introduction to each area is enough. The ML Math lessons explain the theory one topic at a time, and this guide links each topic to a lesson and to a problem where you turn it into tested code.

In what order should you learn maths for machine learning?

Learn linear algebra first, then calculus, then probability and statistics, then optimization, then information theory. Each area depends on the ones before it, because gradients are vectors, losses come from likelihoods, and optimization combines gradients with losses. Learning in dependency order means every new idea builds on something you already understand.

You do not need to finish each area before starting the next. A practical approach is to learn each one to a working level in this order, start building models, then return to deepen whichever area your work demands.

How much maths do you need for machine learning?

How much maths you need depends on your goal. Using ML libraries needs mainly intuition and basic statistics. Working as an ML engineer needs working fluency in all four core areas. Implementing methods from scratch needs precise calculus and linear algebra, and research needs deeper theory, including probability, optimization and often proofs.

ML interviews usually test the ML engineer row, which means explaining and applying the maths behind common methods, such as deriving a gradient, choosing a loss or interpreting a confidence interval. If you are deciding whether you need a degree for this, our guide on whether you need a math degree for ML breaks it down by role.

What linear algebra do you need for machine learning?

For machine learning you need vectors and dot products, matrices and matrix multiplication, transposes, norms, and the idea of a matrix as a transformation. Later, add eigenvalues, eigenvectors and the singular value decomposition, which underpin PCA. Above all, you need to reason fluently about shapes, because most ML code is matrix operations.

• A dot product measures how aligned two vectors are, and it is the basic operation behind similarity, linear models and attention.
• A norm is the length of a vector, and the L2 norm appears in regularization, normalization and distances.
• A matrix maps one vector space to another, and multiplying matrices composes those maps. A neural network layer is a matrix multiplication plus a bias.
• Knowing that an (n, d) matrix times a (d, k) matrix gives an (n, k) matrix prevents most implementation bugs.
• Eigenvectors are the directions a matrix only stretches, and they describe covariance, curvature and stability.
• The singular value decomposition factorizes any matrix into rotations and scalings. PCA, low-rank approximation and methods such as LoRA rely on this idea.

Where does linear algebra appear in ML?

A dataset of n examples with d features is an (n, d) matrix, and a layer with weights W of shape (d, k) maps it to (n, k). In this example, X holds 32 examples with 10 features each, W is a layer from 10 features to 4, and H comes out with shape (32, 4). The snippets in this guide run in one session in order, so later ones reuse np and rng.

import numpy as np

rng = np.random.default_rng(0)
X = rng.normal(size=(32, 10))
W = rng.normal(size=(10, 4))
b = np.zeros(4)
H = X @ W + b

PCA is linear algebra end to end. The principal directions are the eigenvectors of the covariance matrix, and the SVD of the centered data gives them directly as the rows of Vt. The check below prints True.

Xc = X - X.mean(axis=0)
U, S, Vt = np.linalg.svd(Xc, full_matrices=False)
evals, evecs = np.linalg.eigh(np.cov(Xc, rowvar=False))
print(np.allclose(np.sort(S**2 / (len(X) - 1))[::-1], evals[::-1]))

The squared singular values, divided by n minus 1, equal the covariance eigenvalues. Knowing why this holds is a common interview follow-up.

The CUDA plan turns core linear algebra into kernels, which forces you to think about every index. Work through Dot Product, Matrix Transpose, Outer Product, Matrix-Vector Multiplication, Matrix Multiplication, L2 Normalization and Cosine Similarity. For PCA and other unsupervised methods in NumPy, see Cracking ML.

What calculus do you need for machine learning?

For machine learning you need derivatives, partial derivatives, gradients and the chain rule, which together explain how models learn. Jacobians and Hessians help with vector-valued functions and curvature. Integration matters less, mainly for expectations, probability densities and metrics such as AUC. You rarely need advanced techniques beyond these.

• A derivative is the rate of change of a function, and a derivative of zero marks a possible minimum.
• For a function of many variables, the gradient is the vector of partial derivatives. It points in the direction of steepest increase, so training steps in the opposite direction.
• The chain rule says the derivative of a composition is the product of derivatives, and backpropagation is the chain rule applied layer by layer.
• The Jacobian collects the derivatives of a vector-valued function, and the Hessian collects second derivatives and describes curvature, which explains why some loss surfaces are hard to optimize. The Jacobian and Hessian lesson covers both.
• Integrals give expectations over continuous distributions and the area under a curve, and the integration and AUC lesson connects them to a metric you will use constantly.

Where does calculus appear in ML?

The sigmoid function, used in logistic regression, has a neat derivative, $\sigma'(z) = \sigma(z)(1 - \sigma(z))$. You can confirm a derivative like this numerically with a central difference, which is the same check used to test backpropagation code. The largest gap between the two prints as about 6e-11.

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

z = np.linspace(-4, 4, 9)
eps = 1e-6
numeric = (sigmoid(z + eps) - sigmoid(z - eps)) / (2 * eps)
analytic = sigmoid(z) * (1 - sigmoid(z))
print(np.abs(numeric - analytic).max())

Start with Implement Sigmoid in NumPy and Implement Gradient Descent for a 1D Quadratic, where you derive and apply a gradient yourself. Then derive the full gradient of a model in the Logistic Regression Training Loop.

What probability and statistics do you need for machine learning?

For machine learning you need probability to model uncertainty, which means random variables, common distributions, expectation, variance, conditional probability and Bayes' rule. You need statistics to learn from data and judge results, which means maximum likelihood estimation, bias and variance, sampling, confidence intervals and hypothesis tests. Together they explain where losses come from and whether results are real.

Which probability topics matter?

• The common distributions are Bernoulli for binary outcomes, categorical for classes and Gaussian for continuous noise.
• Expectation is the average value and variance is the spread, and training minimizes an expected loss.
• A classifier models $p(y \mid x)$, and Bayes' rule underlies naive Bayes and Bayesian methods.
• Independence is the assumption behind many models and most sampling methods.

Which statistics topics matter?

• Maximum likelihood estimation chooses the parameters that make the observed data most probable, and most training objectives are MLE in disguise. The maximum likelihood lesson works through the derivations.
• Bias is an estimator's systematic error and variance is its sensitivity to the sample, and the same idea explains underfitting and overfitting.
• A validation score is an estimate from a sample, so it has uncertainty, and the bootstrap is a simple way to estimate it.
• Hypothesis tests decide whether a difference is real and trade false positives against false negatives. The Type I and Type II errors lesson explains it with ML examples.

Where do probability and statistics appear in ML?

Mean squared error is not arbitrary. If you assume Gaussian noise with fixed variance, the negative log-likelihood is proportional to the sum of squared errors, so minimizing MSE is maximum likelihood. For a constant prediction, the best value is the sample mean. Here nll is the Gaussian negative log-likelihood up to constants, and the script prints 2.954 next to a sample mean of 2.9537.

y = rng.normal(loc=3.0, scale=2.0, size=1000)
mus = np.linspace(0, 6, 6001)
nll = [0.5 * np.sum((y - m) ** 2) for m in mus]
print(mus[np.argmin(nll)], y.mean())

The same reasoning with Bernoulli outputs gives binary cross-entropy, which is why classification uses a different loss from regression.

The probability study plan turns these topics into problems. Gaussian Naive Bayes combines distributions, MLE and Bayes' rule in one model, and Bootstrap Mean and Confidence Interval puts uncertainty estimates into code. For the evaluation side, the metrics and evaluation problems show statistics applied to model assessment.

What optimization do you need for machine learning?

For machine learning you need to understand loss functions, gradient descent and its variants, learning rates, mini-batches, momentum and adaptive methods such as Adam, plus regularization. Knowing what convexity guarantees, and why deep learning lacks it, explains much of how training behaves. Constrained optimization matters for some methods, such as support vector machines.

• The loss function is the quantity training minimizes, and its choice encodes your assumptions about the data and the task. Mean Squared Error is the place to start, and our guide on which loss function to use covers the rest.
• Gradient descent repeatedly steps against the gradient, and stochastic gradient descent uses a mini-batch to estimate the gradient cheaply.
• The learning rate is the step size. Too small a rate is slow, and too large a rate diverges.
• Momentum smooths noisy gradients, and Adam adapts the step size for each parameter using running averages of the gradient and its square.
• A convex loss has a single global minimum, so gradient descent finds it. Linear and logistic regression are convex, while deep networks are not, so their training can stall near saddle points or settle in different minima.
• L2 and L1 regularization penalties discourage large or non-sparse weights, trading a little training fit for better generalization.

Where does optimization appear in ML?

The learning rate's effect has a precise explanation. For $f(x) = x^2$, the gradient is $2x$, so each step multiplies x by $(1 - 2\eta)$. Gradient descent converges only when that factor has magnitude below one, which means $\eta < 1$. The three runs below print 0.0115, 0.0115 and 38.3. The first converges smoothly, the second converges while overshooting at every step, and the third diverges.

def gradient_descent(lr, steps=20, x=1.0):
    for _ in range(steps):
        x = x - lr * 2 * x
    return x

print(gradient_descent(0.1))
print(gradient_descent(0.9))
print(gradient_descent(1.1))

In general, the stable learning rate depends on the curvature of the loss, which is what the Hessian measures. That link between calculus and optimization is why learning rates need tuning for each problem. Implement the core update rules yourself in the gradient descent problem and Implement Adam Optimizer Step, and see how they behave on real networks in Cracking Deep Learning.

What information theory do you need for machine learning?

For machine learning you need entropy, cross-entropy and KL divergence. Entropy measures uncertainty in a distribution, cross-entropy measures how well one distribution predicts another, and KL divergence is the gap between them. They define the standard classification loss, language model perplexity, decision tree splits and the training objectives of models such as VAEs.

The three quantities are linked by a simple identity, which says cross-entropy equals entropy plus KL divergence. Because the entropy of the true labels is fixed, minimizing cross-entropy is the same as minimizing KL divergence. In the check below, p is the true distribution and q is the model's prediction, and it prints True.

p = np.array([0.7, 0.2, 0.1])
q = np.array([0.5, 0.3, 0.2])
H_p = -(p * np.log(p)).sum()
CE = -(p * np.log(q)).sum()
KL = (p * np.log(p / q)).sum()
print(np.isclose(CE, H_p + KL))

The cross-entropy lesson and the KL divergence lesson explain the theory. Practice it in Cross-Entropy Loss and Log Loss per Sample, where clipping probabilities prevents taking the log of zero.

What maths do deep learning and LLMs need?

Deep learning and LLMs use the same four core areas, applied to tensors at scale. The key additions are the softmax function, the variance arguments behind initialization and attention scaling, normalization statistics, numerical stability tricks such as log-sum-exp, and trigonometric functions for positional encodings. No new branch of maths is required to start.

Why does attention divide by the square root of the dimension?

Attention scores are dot products of query and key vectors. If each has $d_k$ independent components with variance one, their dot product has variance $d_k$. Large scores push softmax into a region with tiny gradients, so Attention Is All You Need divides by $\sqrt{d_k}$ to bring the variance back to about one. The loop below prints the raw and scaled variances, which come out as 16.3 and 1.02 for d = 16, 63.1 and 0.99 for d = 64, and 259.4 and 1.01 for d = 256.

for d in [16, 64, 256]:
    q = rng.normal(size=(10000, d))
    k = rng.normal(size=(10000, d))
    s = (q * k).sum(axis=1)
    print(d, round(s.var(), 1), round((s / np.sqrt(d)).var(), 2))

This combines linear algebra, through dot products, with probability, through the variance of a sum, and it is a common interview question.

What other maths shows up in deep learning and LLMs?

• Softmax turns scores into a probability distribution, and computing it stably requires subtracting the maximum, as in the Softmax kernel problem.
• Layer and batch normalization rescale activations using a mean and variance, as in the Mean and Variance problem.
• Weight initialization scales are chosen so that activation variance stays stable across layers, which is another variance argument.
• The original transformer uses sines and cosines of different frequencies for positional encodings, which you can build in Implement Positional Encoding.
• Perplexity is the exponential of the average cross-entropy, and it is used to evaluate language models.

The Build LLM from Scratch plan brings these together, and Scaled Dot-Product Attention puts the scaled softmax into a GPU kernel.

What maths can you skip or learn later?

Most ML practitioners can skip or postpone measure theory, real analysis, abstract algebra, advanced proof techniques and most of numerical analysis. Learn them later only if your work requires them. Research in areas such as diffusion models, learning theory or geometric deep learning may need stochastic calculus, functional analysis or group theory.

How should you learn maths for machine learning?

Learn maths for machine learning by pairing every concept with code. Study the idea, work a tiny example by hand, implement it in NumPy, then use it inside an ML algorithm and test the result. Implementation exposes gaps that reading hides, and connecting each concept to a model makes it stick. Use this five-step loop for every topic.

1. Learn the concept from a lesson, textbook chapter or lecture.
2. Work a small example by hand, such as a 2 by 2 matrix product or the entropy of three outcomes.
3. Implement it in NumPy without library shortcuts.
4. Use it inside an ML method, such as the gradient inside logistic regression.
5. Verify it with a numerical check or a reference implementation.

What does a 12-week roadmap for ML maths look like?

This suggested roadmap follows the dependency order at about 6 to 8 hours a week, combining theory with implementation. Adjust the pace to your background, since learners with recent maths study can move faster while others may need longer on the early weeks.

Which resources teach maths for machine learning?

Good resources for ML maths include free textbooks such as Mathematics for Machine Learning, university courses with free lecture materials, and visual video series for intuition. Combine one main resource per area with hands-on implementation, because watching or reading alone rarely builds the fluency that ML work and interviews require.

Keep the ML math formula sheet open as a reference while you work through any of them.

What are the common mistakes when learning maths for machine learning?

• Trying to master everything before building anything stalls progress, so learn each area to a working level and deepen it as you go.
• Learning in the wrong order makes things harder, because multivariable calculus without linear algebra makes gradients confusing.
• Reading without implementing leaves gaps, since a concept you cannot code is one you do not yet understand well enough for ML work.
• Many learners focus on calculus and skip the statistics that explains losses and evaluation.
• Memorizing formulas without meaning is less useful than knowing, for example, that the gradient points uphill.
• Exact maths and floating-point code differ, so overflow, underflow and the log of zero are real problems to handle.
• Avoiding maths entirely works only until something goes wrong, because libraries hide the maths until you need it.

Practice this

Then browse the problem library for more, or work through the ML Math lessons in order.

Sources

• Deisenroth, Faisal and Ong, Mathematics for Machine Learning, Cambridge University Press (2020).
• Goodfellow, Bengio and Courville, Deep Learning, MIT Press (2016), chapters 2 to 4.
• Blitzstein and Hwang, Introduction to Probability, 2nd edition, CRC Press (2019).
• Strang, Introduction to Linear Algebra, Wellesley-Cambridge Press, and MIT 18.06 course materials.
• Boyd and Vandenberghe, Convex Optimization, Cambridge University Press (2004).
• MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge University Press (2003).
• Vaswani et al., “Attention Is All You Need”, arXiv:1706.03762 (2017), Section 3.2.1 on scaled dot-product attention.

The code outputs above were produced by running the snippets as shown.

Related guides

• Do you need a math degree to work in machine learning?
• Machine learning math cheat sheet
• How to read an ML paper and implement it
• How much Python do you need before learning ML?