How to Learn PyTorch, from Tensors to a Training Loop

Name: TensorTonic
Author: TensorTonic

A learning order for PyTorch, from tensors and autograd to nn.Module, the training loop and DataLoader, with a coding problem to practise each topic and the mistakes beginners make most.

By TensorTonic's founderLast updated November 11, 202612 min read

How do you learn PyTorch?

You learn PyTorch by working through its core pieces in a fixed order and writing code for each one, starting with tensors and ending with a training loop that loads data, runs on a device and saves a checkpoint. TensorTonic has a PyTorch study plan with 30 problems in six sections, and each problem is checked against tests, so you know a piece works before you build on it. This guide gives the order, a problem to practise at each step and the mistakes that slow most beginners down.

PyTorch is a Python library for building and training neural networks. Its central object is the tensor, an array that can live on a CPU or a GPU and that records the operations applied to it so that gradients can be computed automatically. Layers, optimizers and data loaders are all built on top of those two ideas, which is why the learning order below starts with them.

What do you need before learning PyTorch?

You need working Python and basic NumPy before learning PyTorch, along with a rough idea of what a neural network layer and a loss function are. PyTorch tensors follow the same rules as NumPy arrays for shapes, broadcasting and indexing, so time spent on NumPy carries over directly.

For Python, you should be able to write functions, loops and classes without looking things up, and this checklist of the Python you need for ML lets you test yourself.
For NumPy, you should understand array shapes, broadcasting and indexing, which the NumPy study plan covers in 25 problems across five sections.
For math, you should be comfortable with matrix multiplication and know how the chain rule combines derivatives, and the guide to the maths you need for machine learning covers the rest.

In what order should you learn PyTorch?

You should learn PyTorch in the order data moves through a training run, which means tensors and shapes first, then broadcasting and indexing, autograd, modules and layers, losses and optimizers, the training loop, data loading, devices, saving and loading, and finally debugging. Each step uses the ones before it, so a gap early on turns into confusing errors later. The table pairs every step with a lesson to read where one exists and problems to practise.

The order to learn PyTorch in, with a lesson to read and problems to practise for each step
Step	What you learn	Read	Practice
1. Tensors and shapes	Creating tensors, dtypes, reshape, view, transpose, squeeze and unsqueeze		Tensor Factory, Tensor Shape Manipulation
2. Broadcasting and indexing	How shapes line up in element-wise operations, slicing, fancy indexing and boolean masks	What is matrix multiplication?	Tensor Operations, Fancy Indexing, Boolean Masking
3. Autograd	requires_grad, backward, leaf tensors, accumulated gradients and torch.no_grad	What is a computational graph?	Basic Autograd, Run Reverse-Mode Autodiff
4. nn.Module and layers	Subclassing nn.Module, nn.Parameter, the forward method, initialization, batch norm and residual connections	What is batch norm?	Custom Linear Layer, Weight Initialization, Batch Normalization, Residual Block
5. Losses and optimizers	Cross-entropy on logits, a stable softmax, SGD with momentum, Adam and learning rate schedulers	Cross-Entropy Loss: Formula and BCE vs MSE	Loss Functions, Softmax from Scratch, Custom SGD with Momentum, Optimizer Scheduler
6. The training loop	zero_grad, forward, loss, backward and step, then validation, early stopping and gradient accumulation	Gradient Accumulation in LLM Training	Mini Training Loop, Gradient Accumulation, Early Stopping
7. Dataset and DataLoader	Writing __len__ and __getitem__, batching, shuffling, padding and sampling		Custom Dataset Class, Balanced DataLoader, Pad Sequences
8. GPUs and devices	Moving the model and each batch to one device, and estimating how much memory training needs	Mixed Precision Training: FP16 vs BF16	Mixed-Precision Training Memory Accountant
9. Saving and loading	state_dict, load_state_dict, and checkpoints that hold the optimizer state so training can resume
10. Debugging	Overfitting one batch, checking gradients, clipping exploding gradients and reading loss curves	Why is my training loss not decreasing?	Gradient-Check an Autodiff Graph, Gradient Clipping (Global Norm)

The indexing problems in step 2 are written in NumPy, and the same slicing, fancy indexing and boolean masks work on PyTorch tensors with the same syntax.

What should you learn first in PyTorch?

You should learn tensors and their shapes first, because nearly every PyTorch error a beginner sees is a shape error. Practise creating tensors, changing their shape with reshape and view, swapping dimensions with transpose and permute, and adding or removing dimensions of size one. Then learn broadcasting, the rule that lets a tensor of one shape combine with a tensor of another, since it is useful and also the source of bugs that raise no error.

How does autograd work in PyTorch?

Autograd works by recording every operation on a tensor that has requires_grad set, building a graph of those operations, and walking that graph backwards when you call backward on the loss. The gradient of each leaf tensor ends up in its grad attribute, and PyTorch adds new gradients to whatever is already there instead of replacing them. That accumulation is why a training loop clears gradients at every step, and the Build Micrograd from Scratch study plan has 25 problems that build a small autograd engine so you can see how the backward pass is assembled.

What does a PyTorch training loop look like?

A PyTorch training loop repeats the same five calls for every batch, which are clearing old gradients, running the forward pass, computing the loss, running the backward pass and updating the weights. A complete loop also switches between training and evaluation modes and measures the loss on data the model does not train on.

Put the model in training mode with model.train().
Take each batch from the DataLoader and move its inputs and targets to the same device as the model.
Clear the gradients from the previous step with optimizer.zero_grad().
Run the forward pass and compute the loss from the outputs and the targets.
Call loss.backward() to compute the gradients, then optimizer.step() to update the weights.
At the end of each epoch, switch to model.eval(), run the validation set inside torch.no_grad(), and record the validation loss.

A Dataset class returns one example at a time through __getitem__ and reports its size through __len__, and a DataLoader wraps it to produce shuffled batches. Once the loop works, saving the model's state_dict, a dictionary of its parameter and buffer tensors, is the usual way to keep the trained weights, and a checkpoint for resuming training also stores the optimizer's state_dict.

Why write things from scratch before using the built-ins?

Writing a layer, a loss or an optimizer from scratch first shows you what the built-in version does, which makes its arguments, its expected shapes and its failure modes much easier to understand. For example, nn.CrossEntropyLoss expects raw logits and applies log-softmax itself, which is obvious once you have written a stable softmax and a cross-entropy loss by hand and easy to get wrong if you have not.

The PyTorch study plan follows this approach. Its early sections include a custom linear layer and a softmax written by hand, and its last two sections build convolutions, recurrent cells and attention from tensor operations. Once you have written a component yourself, switch to the built-in version for real work, since it is faster, better tested and handles edge cases that a first implementation skips.

How do you practise PyTorch each day?

You practise PyTorch each day by solving one or two small problems that each cover a single idea, then using that idea in a slightly larger piece of code. A short session works well when it has a clear target, such as making one set of tests pass or getting one tricky shape right.

Take the next problem in the study plan and write a solution before looking anything up.
Print the shape of every tensor you create until you can predict each shape before you run the code.
After a problem passes, rewrite its key line with a different operation, such as view instead of reshape or einsum instead of matmul, and check that the result matches.
Every week or so, join the pieces you have practised into a small model and train it on a toy dataset until the loss falls.
Keep a list of the errors you hit and what caused each one, because the same few errors tend to come back.

What should you build once the basics click?

Once the basics click, you should build a complete model that uses every piece together, and the Build a Tiny GPT from Scratch project is a good first one. In that PyTorch notebook you tokenize Shakespeare, set up next-token prediction, write causal self-attention and a pre-norm block, train a character-level GPT and generate text, and then remove the causal mask to see how the model cheats.

For breadth across architectures, Cracking Deep Learning has 35 problems in six sections, running from perceptrons and backpropagation through convolutional and recurrent networks to Transformers, generative models and advanced architectures. If language models interest you most, the guide on how to build an LLM from scratch gives a stage-by-stage build order.

Which resources teach PyTorch?

TensorTonic teaches PyTorch through tested coding problems, with a 30-problem study plan that runs from tensors to attention and a notebook project that trains a tiny GPT. Other well-known resources also teach PyTorch, and they pair well with problems you solve yourself.

The PyTorch project publishes official tutorials and documentation, which start with tensors and a first training loop and go on to more advanced topics.
Eli Stevens, Luca Antiga and Thomas Viehmann wrote the book Deep Learning with PyTorch, published by Manning.

Whichever one you follow, type the code yourself and run it before you move on. Reading a working example is much easier than producing one, and only producing one shows you what you have understood.

What mistakes do beginners make when learning PyTorch?

The mistakes beginners make most often in PyTorch are shape bugs that broadcasting hides, forgotten zero_grad calls, the wrong training or evaluation mode, tensors on different devices, tensors kept attached to the graph, and mismatched dtypes. Each one is easy to fix once you know to look for it.

Subtracting a tensor of shape (N,) from one of shape (N, 1) broadcasts to an (N, N) result instead of raising an error, so check the shape after any operation you are unsure of.
Leaving out optimizer.zero_grad() lets gradients from earlier batches add up, because PyTorch accumulates gradients by default, and the updates become larger and noisier than intended.
Evaluating without model.eval() leaves dropout active and makes batch norm use the statistics of the current batch, and forgetting to call model.train() afterwards trains the next epoch in evaluation mode.
Moving the model to the GPU while the inputs stay on the CPU raises an error that names two devices, and the fix is to send every batch to the model's device inside the loop.
Appending the loss tensor to a list for logging keeps its computation graph alive and makes memory grow every step, so store loss.item() instead and use detach when you need a tensor's values without its history.
Passing class labels as floats to nn.CrossEntropyLoss raises a dtype error because it expects integer class indices, and tensors made from NumPy arrays default to float64 while model weights are float32.
Applying softmax to the outputs before nn.CrossEntropyLoss applies it twice, since the loss already includes log-softmax, which makes training slower and less stable.

Practice this

Basic Autograd shows how PyTorch records operations and fills in the gradient of each leaf tensor when you call backward.
Custom Linear Layer is the smallest useful nn.Module, with its own parameters and a forward method.
Mini Training Loop puts the forward pass, the loss, the backward pass and the optimizer step together in the order every training run uses.

Then work through the full PyTorch study plan, or train a complete model in the Build a Tiny GPT from Scratch project.

Should you learn PyTorch or TensorFlow first?

Most people starting deep learning today are better served by learning PyTorch first, because it is the most widely used deep learning framework and most recent research code is written in it. TensorFlow is still used in some production systems. Tensors, automatic differentiation, layers and training loops work the same way in both, so a second framework is much quicker to pick up once you know the first.

How long does it take to learn PyTorch?

How long it takes to learn PyTorch depends mostly on how comfortable you already are with Python and NumPy. Someone who knows both reaches a working training loop much sooner than someone who is still learning array shapes. Writing models confidently and debugging them takes longer than learning the API, and that skill comes from building and training several small models yourself.

Do you need a GPU to learn PyTorch?

You do not need a GPU to learn PyTorch, because the same code runs on a CPU and small models train quickly there. A GPU becomes useful when you train larger models on larger datasets. Moving working code to a GPU later only needs the model and each batch sent to the new device.

Is PyTorch asked in ML interviews?

PyTorch comes up in many ML engineer and research engineer interviews, usually as a coding question. Common tasks are writing a training loop, a custom nn.Module, a loss function, attention or a Dataset class, and interviewers often follow up with autograd questions such as why gradients must be zeroed before each backward pass.

Should you learn NumPy before PyTorch?

Learning some NumPy before PyTorch helps, because PyTorch tensors follow the same rules for shapes, broadcasting and indexing. NumPy lets you practise those rules without autograd or devices getting in the way. You do not need to master it first, and array creation, indexing and broadcasting are enough to start.