How to Learn Computer Vision, from Pixels to Vision Transformers

How do you learn computer vision?

You learn computer vision by writing its core operations yourself in the order models depend on them, starting with images as arrays and ending with Vision Transformers, CLIP and diffusion models. Each stage reuses the one before it, so preprocessing feeds convolutions, convolutions build CNNs, and CNN features power detection and segmentation. TensorTonic has a coding problem with tests for each topic in this roadmap, and the Cracking CV study plan collects 35 computer vision problems in five sections.

This guide is a computer vision roadmap in eleven stages. Each stage has a short explanation and a table that pairs every topic with a page to read and a problem to solve, so you can learn computer vision one tested function at a time.

What does computer vision cover?

Computer vision covers the methods that let a model understand images and video, from classifying a whole image to locating, outlining and describing what it contains. The main tasks are image classification, object detection, which returns a box and a label for each object, and segmentation, which labels every pixel. Modern computer vision also includes models that connect images with text, such as CLIP, and models that generate new images, such as diffusion models.

Almost all of these tasks share the same foundations. A detector and a segmentation model both start from a CNN or Transformer backbone trained like a classifier, and a text-to-image model uses image encoders and attention layers you will have written earlier in the roadmap.

What do you need to know before learning computer vision?

You need Python, NumPy, the basics of PyTorch and an understanding of how a neural network is trained before you start learning computer vision. NumPy matters more than people expect, because most vision code is careful indexing and reshaping of four-dimensional arrays. The math is linear algebra for arrays and matrix products, calculus for gradients, and enough probability to read a softmax output.

• For arrays, you should be comfortable with shapes, broadcasting and reshaping, which the NumPy study plan covers.
• For PyTorch, you should be able to write a small network and a training loop, and the guide to learning PyTorch and the PyTorch study plan cover both.
• For neural networks, you should understand backpropagation, activations and losses at the level of the Cracking Deep Learning study plan.
• For math, the maths for machine learning guide lists what to learn and in what order.

In what order should you learn computer vision?

You should learn computer vision in the order data flows through a vision system, which means image preprocessing first, then convolutions and CNNs, then training and transfer learning, then the dense prediction tasks, and finally Transformers, contrastive models and generative models. The Cracking CV plan covers the image basics, convolution and pooling, detection, losses and augmentation and Vision Transformer material, and the research tracks and projects linked below cover the architectures, CLIP and diffusion in more depth.

Stage 1: How is an image stored, and how do you prepare it for a model?

An image is stored as an array of height by width by channels, usually with 8-bit integer values from 0 to 255, and you prepare it for a model by converting it to floats, putting the channels first and normalizing each channel. Image libraries such as PIL and OpenCV load images in height, width and channel order, and OpenCV uses BGR channel order, while PyTorch convolutions expect a batch laid out as batch, channels, height and width. Resizing and cropping bring every image to the size the model expects, and bilinear interpolation is the usual way to resize without blocky artefacts.

Normalization subtracts a per-channel mean and divides by a per-channel standard deviation, so the inputs sit near zero with a similar scale. When you later reuse a pretrained model, you must apply the same statistics it was trained with, which is why it pays to compute them yourself once.

Stage 2: How do classic filters find edges?

Classic filters find edges by sliding a small hand-designed kernel over the image and computing a weighted sum at each position, which gives a large response wherever brightness changes quickly. A Gaussian blur uses weights from a 2D normal distribution to average each pixel with its neighbours and remove noise. The Sobel filter estimates the horizontal and vertical derivatives, and the square root of the sum of their squares is large at edges.

These filters are worth writing because a convolutional layer performs the same sliding weighted sum with kernels that are learned instead of designed. Blurring before taking derivatives is standard, since derivatives amplify pixel noise, and the Gaussian kernel is separable, so a 2D blur can be computed as one horizontal and one vertical 1D pass.

Stage 3: How do convolutions and pooling work?

A convolution layer slides a set of learned K by K filters over the input, and each filter has one slice per input channel whose results are summed into a single output map. A layer with C_out filters has a weight tensor of shape C_out by C_in by K by K, and with padding P and stride S the output height is (H plus 2P minus K) divided by S, rounded down, plus 1. Pooling then shrinks each feature map by taking the maximum or the mean over small windows, with no learned weights.

The variants each solve a specific problem. Dilated convolutions see a wider area without extra weights, depthwise separable convolutions cut the cost for mobile networks, and transposed convolutions upsample feature maps for segmentation. Global average pooling reduces each channel to one number at the end of a classifier, which lets the same network accept different input sizes.

Stage 4: Which CNN architectures should you study, and why does ResNet matter?

The CNN architectures worth studying are VGG, Inception and ResNet, and ResNet matters most because its residual connection made very deep networks trainable. A residual block computes a small stack of convolutions F of x and adds the input back, so the block outputs F of x plus x and only has to learn a correction. The identity path gives gradients a direct route to early layers, which let He and colleagues train networks with 152 layers on ImageNet in 2015.

VGG showed that stacks of 3 by 3 convolutions cover a large receptive field with fewer weights than one large kernel. Inception ran several kernel sizes in parallel and used 1 by 1 convolutions to keep the cost down, and ResNet bottleneck blocks use the same 1 by 1 trick around a 3 by 3 layer. Squeeze-and-excitation reweights the channels of a block from their pooled values, and it appears in later efficient networks.

Stage 5: How do you train an image classifier that generalises?

You train an image classifier that generalises by combining a cross-entropy loss and a standard optimizer with data augmentation that shows the model realistic variations of each image. Random resized crops, horizontal flips and colour jitter teach the model that these changes do not alter the class, while batch normalization and dropout keep training stable and reduce overfitting. MixUp and CutMix go further by blending two images and their labels, which acts as a strong regularizer.

An augmentation is only safe when it keeps the label true, so flipping a photo of a cat is fine while flipping text or rotating a 6 into a 9 is not. Report top-1 and top-5 accuracy on a held-out set, and look at the confusion between classes, because a single accuracy number hides which classes the model mixes up.

Stage 6: How does transfer learning work for vision models?

Transfer learning works by taking a backbone pretrained on a large dataset such as ImageNet, replacing its final classification layer with one sized for your classes, and training on your smaller dataset. You usually train only the new head first with the backbone frozen, then unfreeze the later layers and continue with a smaller learning rate. Early layers detect generic edges and textures that transfer well, while later layers are more specific to the original task.

Most real vision projects start this way, because a fine-tuned pretrained model beats a model trained from scratch whenever data is limited. Reuse the preprocessing of the backbone exactly, and consider keeping batch normalization in evaluation mode when your batches are small.

Stage 7: How does object detection work?

Object detection works by running a backbone over the image and predicting, at many positions and scales, a class score and a box for each candidate object. Many detectors place anchor boxes of preset sizes on the feature map, label each anchor by its IoU with the true boxes, and regress small offsets from the anchor to the object. At inference, non-maximum suppression removes duplicate boxes that overlap a higher-scoring box of the same class, and the detector is scored with mean average precision.

Write IoU first, since anchor matching, NMS, the IoU-based losses and mAP all depend on it. Feature pyramids let a detector find small objects on fine maps and large ones on coarse maps, focal loss handles the huge number of easy background anchors in one-stage detectors, and DETR replaces anchors and NMS with Hungarian matching between a fixed set of predictions and the true objects.

Stage 8: How does image segmentation work?

Image segmentation works by predicting a class for every pixel, which needs a network whose output has the same height and width as its input. U-Net does this with an encoder that downsamples and adds channels, a decoder that upsamples back to full resolution, and skip connections that concatenate encoder features into the decoder to restore fine detail. Semantic segmentation labels pixels by class, instance segmentation separates individual objects, and panoptic segmentation does both.

Segmentation reuses earlier stages directly, with transposed convolutions for upsampling and IoU computed over masks instead of boxes. Dice loss helps when the foreground covers a small part of the image, and mean IoU across classes is the standard metric.

Stage 9: How does a Vision Transformer read an image?

A Vision Transformer reads an image by cutting it into non-overlapping patches, flattening each patch and projecting it to a vector, so the image becomes a sequence of tokens like words in a sentence. A 224 by 224 image with 16 by 16 patches becomes 196 tokens, and this projection is the same as a convolution whose kernel size and stride equal the patch size. A learned class token and position embeddings are added, Transformer encoder blocks let every patch attend to every other patch, and the final class token feeds the classifier.

A ViT gives up the locality and translation equivariance that convolutions build in, so it needs more data or stronger augmentation to match a CNN. Windowed attention, as in the Swin Transformer, limits attention to local windows to keep the cost manageable at high resolution.

Stage 10: How does CLIP learn from images and captions?

CLIP learns from images and captions by training an image encoder and a text encoder so that each image and its own caption end up with similar embeddings. For a batch of N pairs it computes an N by N matrix of cosine similarities scaled by a learned temperature, and the loss is a cross-entropy over the rows plus one over the columns, with the matching pairs on the diagonal. Every other caption in the batch acts as a negative example, and the original CLIP was trained this way on 400 million image and text pairs.

Once trained, CLIP classifies images it was never labelled for by embedding one text prompt per class and picking the closest one. SigLIP replaces the softmax over the batch with an independent sigmoid for every pair, which makes very large batches cheaper to train.

Stage 11: Where do generative models and diffusion fit in?

Generative models fit at the end of the roadmap because they reuse almost everything before them, from convolutions and U-Nets to attention and text embeddings. A VAE learns a smooth latent space with a reconstruction loss and a KL penalty, and a GAN trains a generator against a discriminator. A diffusion model adds Gaussian noise to images and trains a network, usually a U-Net or a Transformer, to predict that noise, then generates by removing noise step by step from pure randomness.

Diffusion is a large topic of its own, and the guide to learning diffusion models breaks it into nine stages. The Build Diffusion Model from Scratch study plan covers that material in 26 problems across seven sections.

Can you learn computer vision without a GPU?

Yes, you can learn computer vision without a GPU, because every operation in this roadmap can be written and tested on small arrays, and small models on small images train on a laptop CPU. Fashion-MNIST images are 28 by 28 pixels, and a tiny CNN or ViT trains on them quickly enough to run many experiments. A GPU only becomes necessary when you train large models on full-size images or fine-tune big pretrained backbones on large datasets.

The coding problems linked in this guide run in the browser and are checked against tests, so there is nothing to install. Our guide on practising ML without a GPU covers where to find compute when you want to train something larger.

What should you build to learn computer vision?

You should build a few small end-to-end projects that each answer a question, such as why a CNN beats a ViT on little data or how a contrastive model takes shortcuts. A project like this forces you to combine preprocessing, a model, a training loop and an evaluation, and it gives you something concrete to explain in an interview.

The Computer Vision projects sheet lists more builds, and the guide to building an ML portfolio explains how to present them. If you are preparing for interviews, the computer vision interview questions guide has model answers for the questions asked most often on each of these stages.

What mistakes do people make when learning computer vision?

• Skipping preprocessing and treating it as library glue leaves you unable to find bugs such as a BGR image fed to an RGB model or normalization statistics that differ between training and inference.
• Jumping to Vision Transformers before convolutions and ResNet makes it hard to understand why ViTs need more data, since that answer depends on what convolutions assume about images.
• Training from scratch on a small dataset when a pretrained backbone is available wastes time and usually gives a worse model.
• Applying geometric augmentation to an image without applying it to the boxes or masks gives labels that no longer match the pixels.
• Mixing up box formats, such as corners against a centre with width and height, produces detection code that runs and returns wrong numbers.
• Checking only the training loss hides overfitting, so track accuracy, mAP or mean IoU on a held-out set from the first run.
• Starting with a large model on a large dataset makes every experiment slow, while a small model on Fashion-MNIST or CIFAR-10 shows the same bugs in minutes.

Practice this

Then work through the full Cracking CV study plan, or train a ViT against a CNN in the Build a Tiny Vision Transformer project.

Related guides

• Computer vision interview questions with answers
• How to learn diffusion models from scratch, one stage at a time
• How to learn PyTorch, from tensors to a training loop
• Deep learning interview questions with answers
• 15 ML papers worth implementing from scratch, and in what order
• How to practise ML with no GPU, no dataset and no team