How to Read a Machine Learning Paper and Implement It From Scratch

How do you read an ML paper and implement it?

Read the paper in three passes. Skim it for the claim, study the method for its equations and shapes, then check the appendix for hyperparameters. Before coding, write a table of every symbol and its shape. Build the smallest version first, verify each component against a reference or a closed-form check, overfit a tiny dataset, then reproduce one key finding.

The Research section breaks landmark papers into coding stages, each checked by tests, so you can practice the workflow in this guide with feedback at every step.

What does implementing an ML paper actually involve?

Implementing an ML paper means turning its method into working code that you have verified, then using that code to reproduce at least one of the paper's findings. It takes three skills, which are reading for the precise method, translating equations into tensor operations, and testing each piece. Writing the code is the easy part, and verification is what makes it trustworthy. The table below summarizes the full workflow, and the rest of the guide explains each step.

Which ML paper should you implement first?

Implement a paper whose core idea fits in a few hundred lines, trains on a CPU or one GPU, and can be checked against a library or released weights. Optimizers, normalization layers and attention are good first choices. Avoid papers whose results depend on huge compute, private data or unstable training. Ask these five questions when choosing.

• Is the core small? You implement the method, not the paper's entire experimental setup.
• Can you check it? A library function, released weights or a closed-form result gives you a reference.
• Will it train on your hardware? At small scale, the qualitative result should still appear.
• Is the idea still in use? Implementing a widely used method pays off in interviews and later projects.
• Do you know its prerequisites? A transformer is much easier after you have implemented attention.

Good first papers include Adam, batch normalization, ResNet, Attention Is All You Need and LoRA, because each has a small core and a clear reference to check against. The Adam paper, for example, reduces to a few update equations, which you can practice in Implement Adam Optimizer Step. For classical methods, the NumPy implementations in Cracking ML make good warm-ups, and after Attention Is All You Need, Build LLM from Scratch continues to a full language model. Inference papers on topics such as KV caching and quantization pair well with the Inference Engineering plan. Our list of 15 ML papers to implement from scratch gives a full order.

How should you read an ML paper?

Read an ML paper in three passes, each with a different goal. The first pass finds the main claim in about 15 minutes. The second extracts the method, meaning the equations, the architecture and the algorithm. The third collects every implementation detail, mostly from the appendix, tables and footnotes. Do not try to understand everything in one read. This approach adapts the three-pass method that Keshav described in 2007 for implementation work.

Pass 1: Find the claim

Read the title, abstract, introduction, section headings, figures and conclusion, and skip the math. Answer three questions in one or two sentences each. What problem does the paper solve, what is the one new idea, and what result shows the idea works? If you cannot answer them, read the introduction again before going further.

Pass 2: Extract the method

Read the method section slowly, with a notebook. Copy every equation and label every symbol. Find the architecture figure and the algorithm box, if there is one, and match them to the equations.

Mark anything undefined or ambiguous, such as a symbol used before it is introduced or a normalization with no stated axis. These marks become your list of questions for the third pass.

Pass 3: Collect implementation details

Now read for engineering. Most of the decisions you need are in places readers skip.

• The appendix and supplementary material give hyperparameters, architecture sizes and training schedules.
• The tables list model dimensions, batch sizes and learning rates.
• The footnotes hold small but crucial choices, such as scaling factors.
• The experimental setup describes data preprocessing, splits and the evaluation protocol.
• The released code is the most reliable answer to anything ambiguous, and you should read it after you have tried yourself.
• Later versions and follow-up papers contain corrections, clarifications and better defaults.

How do you turn equations into code?

Turn equations into code by writing a symbols-and-shapes table first, then translating each operation once its input and output shapes are fixed. Sums become reductions over a named axis, products become matrix multiplications or elementwise operations, and every softmax or normalization needs an explicit axis. Most implementation bugs are shape bugs.

Write a symbols-and-shapes table

Take the scaled dot-product attention equation from Attention Is All You Need.

Before writing code, fix every shape.

The table answers the question the equation leaves open. Softmax runs over the key axis, so each query's weights sum to one.

Translate one operation at a time

With the shapes fixed, the code follows directly. X has shape (T, d_model), Wq and Wk have shape (d_model, d_k), Wv has shape (d_model, d_v), and the function returns (T, d_v). The scores matrix is (T, T), where row i scores query i against every key. The boolean future mask blocks attention to later positions, and subtracting the row maximum before exponentiating gives a stable softmax over the last axis, which is the key axis.

import numpy as np

def causal_self_attention(X, Wq, Wk, Wv):
    Q, K, V = X @ Wq, X @ Wk, X @ Wv
    scores = Q @ K.T / np.sqrt(K.shape[-1])
    T = X.shape[0]
    future = np.triu(np.ones((T, T), dtype=bool), k=1)
    scores = np.where(future, -np.inf, scores)
    scores -= scores.max(axis=-1, keepdims=True)
    w = np.exp(scores)
    w /= w.sum(axis=-1, keepdims=True)
    return w @ V

What are the common translation rules?

The Softmax kernel problem drills the same max-subtraction trick that makes log-sum-exp stable. If a derivation involves vector calculus, the Jacobian and Hessian lesson covers the matrix calculus behind backward passes. To go one level lower, the same attention equation becomes a GPU kernel in the CUDA Kernels plan, where the softmax axis and numerical stability matter just as much, and the Triton Kernels plan shows the same idea at block level. If the paper derives its loss from a likelihood, the maximum likelihood lesson and the cross-entropy lesson explain the step from probability to code.

What details do ML papers leave out?

ML papers often leave out the details that decide whether an implementation works, such as initialization, epsilon values, where normalization goes, learning-rate warm-up, data preprocessing, loss reduction and the differences between training and evaluation modes. Look for them in the appendix, the released code and follow-up papers, and record every default you have to choose yourself. Well-known examples show how much these details matter.

• The original 1997 LSTM paper has no forget gate. Gers, Schmidhuber and Cummins added it in 2000, and almost every modern LSTM includes it.
• Attention Is All You Need applies layer normalization after each residual addition, while many later models place it before each sub-layer, which is easier to train. The axis matters too, since layer norm normalizes over features for each token, which you can see clearly when you write it as a Layer Normalization kernel.
• The dropout paper uses a retention probability and scales weights at test time, whereas most modern code divides by the retention probability during training instead.
• Batch normalization uses batch statistics in training and running estimates at inference, so forgetting to switch modes changes every output.
• Adam's bias correction depends on a step count that starts at 1 and counts updates rather than epochs.

Before coding, find each of these details and note where it came from.

How do you implement an ML paper step by step?

Implement an ML paper bottom-up. Set up a minimal project with fixed seeds, build the smallest components first, verify each one, assemble the model, prove it can overfit a tiny batch, then train at small scale and compare with the paper's trend. Only scale up once every earlier step passes.

Step 1: Set up a minimal project

Fix random seeds, put hyperparameters in one config file, and prepare a tiny version of the dataset. Small inputs make every later test fast.

Step 2: Build the smallest components first

Implement leaf operations before composite ones, so activation, normalization and attention come before the block, and the block comes before the model. The same applies to a classical paper, where the loss and gradient come before the training loop, as in Logistic Regression from Scratch. Write down the shape of every tensor as you go.

Step 3: Verify each component as you go

Test each component before building on it, using the methods in the next section. A bug found in a single layer takes minutes to fix, while the same bug found in a full training run can take days.

Step 4: Assemble and overfit a tiny batch

Combine the components and train on 10 to 100 examples. A correct model with enough capacity should drive training loss close to zero. If it cannot, something is wrong in the model, the loss or the optimizer.

Step 5: Train small and compare the trend

Train at a scale your hardware allows. Compare your run with the paper's qualitative finding, such as one method training faster than another, rather than with its exact numbers.

Step 6: Reproduce one result and run ablations

Pick one figure or table to reproduce. Then remove or change one component at a time to test which parts of the method matter. Ablations are what turn a reimplementation into understanding.

Step 7: Document every difference

List every place where your setup differs from the paper, including data, scale, hyperparameters and any defaults you chose. This makes your results interpretable and your write-up honest.

How do you verify a paper implementation is correct?

Verify a paper implementation with independent checks for each component. Compare gradients with finite differences, compare outputs with a library or released weights, test closed-form cases and properties that must always hold, and confirm that the full model can overfit a tiny batch. One check per component is the minimum.

Property tests are especially useful because they need no reference. For causal attention, changing the last token must not change any earlier output. This test reuses the function above, changes only the last token of the input, and prints True False.

rng = np.random.default_rng(0)
T, d = 8, 16
X = rng.normal(size=(T, d))
Wq, Wk, Wv = (rng.normal(size=(d, d)) / np.sqrt(d) for _ in range(3))

X2 = X.copy()
X2[-1] += 1.0
a = causal_self_attention(X, Wq, Wk, Wv)
b = causal_self_attention(X2, Wq, Wk, Wv)
print(np.allclose(a[:-1], b[:-1]), np.allclose(a[-1], b[-1]))

The first check passes because earlier positions cannot see the change. The second shows that the last position does change, so the test is not passing by accident. Remove the mask and the first check fails.

Every stage of a paper in the Research section is checked by tests in the same way. Its Attention Is All You Need track moves from tokenization and positional encoding to scaled dot-product attention, multi-head attention and the encoder block.

What should you do if your results do not match the paper?

If your results do not match the paper, debug in order of likelihood, starting with the data pipeline, then shapes and broadcasting, training versus evaluation mode, loss reduction, the learning-rate schedule, initialization and numerical precision. After that, consider scale and randomness. A small gap can be acceptable if the trend matches, as long as you report it honestly. Work through this checklist from top to bottom.

1. Inspect a batch of data, including inputs, labels, preprocessing and splits, because data bugs are the most common cause.
2. Assert shapes at every step. A (B,) tensor minus a (B, 1) tensor silently produces a (B, B) tensor.
3. Check that dropout and normalization switch correctly between training and evaluation.
4. Confirm the loss reduction, the weighting and numerical stability. Log loss with clipping and cross-entropy loss show why stability matters.
5. Check the learning rate, warm-up, weight decay and step counting in the optimizer and schedule.
6. Use the paper's initialization scheme if it states one.
7. Try full precision to rule out changes caused by mixed precision.
8. Run several seeds, since a gap smaller than the variation between seeds is not a real gap.
9. Compare against the trend rather than the headline number, because smaller models, less data or shorter training can change results.

Reproduction is genuinely hard. Raff attempted to independently reproduce 255 ML papers in 2019 and succeeded for 162 of them, and found that factors such as readability and how fully hyperparameters were specified were linked to success. A partial reproduction that is clearly documented is still valuable.

How long does it take to implement an ML paper?

Implementing an ML paper takes anything from an evening for a single equation, such as an optimizer update, to several weeks for a full architecture with training. Most of the time goes on finding missing details and debugging rather than on writing the core code, so start small to build the skill.

These are planning assumptions for a learner with working Python and PyTorch, not measured figures.

What does a one-week plan for your first paper look like?

This suggested plan suits a medium-sized paper at about an hour or two a day. Larger papers will take longer.

How do you show a paper implementation in your portfolio?

Show a paper implementation as a short reproduction report that states the paper's claim, your setup, your results beside the paper's, every difference in setup and at least one ablation. Lead the README with the result and a figure. Reproducing a finding and testing it is more impressive than copying code that runs. A good report covers these parts in order.

• State the paper's main finding in one sentence.
• Name the figure, table or trend you set out to reproduce.
• Show your numbers or plot next to the paper's.
• List the differences in data, scale, hyperparameters and any defaults you chose.
• Report what happened when you removed or changed each component.
• Explain how you confirmed each component was correct.

Research engineer interviews often ask candidates to explain or implement part of a paper they have worked on. TensorTonic's timed mock assessments include a Research Engineer role for rehearsing that conversation, and our guide on how to build an ML portfolio covers presentation in more depth.

What are the common mistakes when implementing ML papers?

• Reading once and starting to code means you will guess at details the paper actually states in its appendix.
• Copying the official code first skips the learning, so try it yourself and then use the code to resolve ambiguities and check your work.
• Using the library module that is the paper, such as nn.Transformer, hides every decision you are trying to learn.
• Building everything before testing anything makes bugs hard to locate, so verify each component as you write it.
• Chasing the headline number wastes time, because exact results depend on compute and tuning you may not have, and reproducing the finding is enough.
• Changing several things at once means you cannot tell which change helped.
• Ignoring later corrections can leave a known bug in your version, so check for updated versions, errata and follow-up work.

Practice this

Then work through full papers in the Research section, or browse the problem library.

Sources

• Keshav, “How to Read a Paper”, ACM SIGCOMM Computer Communication Review 37(3) (2007).
• Raff, “A Step Toward Quantifying Independently Reproducible Machine Learning Research”, NeurIPS (2019), arXiv:1909.06674.
• Pineau et al., “Improving Reproducibility in Machine Learning Research”, arXiv:2003.12206 (2020).
• Vaswani et al., “Attention Is All You Need”, arXiv:1706.03762 (2017).
• Kingma and Ba, “Adam: A Method for Stochastic Optimization”, arXiv:1412.6980 (2014).
• Hochreiter and Schmidhuber, “Long Short-Term Memory”, Neural Computation 9(8) (1997).
• Gers, Schmidhuber and Cummins, “Learning to Forget: Continual Prediction with LSTM”, Neural Computation 12(10) (2000).
• Srivastava et al., “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”, JMLR 15 (2014).
• Ioffe and Szegedy, “Batch Normalization”, arXiv:1502.03167 (2015).

The code outputs above were produced by running the snippets as shown.

Related guides

• 15 ML papers worth implementing from scratch, and in what order
• Why bother learning machine learning from scratch?
• Inference engineering: what it is and how to learn it
• What maths do you need for machine learning?