Deep Learning Interview Questions and How to Answer Them

Name: TensorTonic
Author: TensorTonic

The deep learning interview questions that come up most, grouped by topic from backpropagation to RNNs, each with a short model answer and a coding problem to practise it.

By TensorTonic's founderLast updated November 11, 202615 min read

What deep learning interview questions should you expect?

Deep learning interview questions test whether you can explain how a neural network learns and why training fails, across backpropagation, activations, initialization, normalization, regularization, optimizers, losses, CNNs and RNNs. Most are spoken questions that the interviewer follows with a why or a when, and some ask you to write a layer or an optimizer step from scratch. TensorTonic has a tested coding problem for every question in this guide, and its Cracking Deep Learning study plan collects 35 deep learning problems in six sections.

This guide groups the questions by topic and gives a short model answer for each, followed by a lesson to read and a problem to solve. The general ML questions, such as the bias-variance tradeoff and data leakage, are covered in how to prepare for an ML engineer interview, and the twenty functions most often asked as code are in ML coding interview questions.

What backpropagation and gradient questions do interviewers ask?

Interviewers ask how backpropagation computes gradients, what the gradient of a common loss is, and how you would check that a gradient is correct. These questions come first because every later topic, from initialization to vanishing gradients, depends on how gradients flow.

How does backpropagation work?

Backpropagation computes the gradient of the loss with respect to every weight by applying the chain rule backwards through the network, one layer at a time. The forward pass stores the inputs and activations of every layer, and the backward pass multiplies the incoming gradient by the local derivative of each layer, so the cost of the backward pass is close to that of the forward pass.

The backpropagation lesson works through it by hand, and the Backpropagation (Single Hidden Layer) problem has you write it for a small network.

What is the gradient of softmax followed by cross-entropy?

The gradient of cross-entropy with respect to the logits that feed a softmax is the predicted probabilities minus the one-hot target, p minus y. This short result is why frameworks fuse softmax and cross-entropy into one operation, which is both faster and more stable than computing them separately.

Derive it yourself in the Softmax + Cross-Entropy Gradient problem.

How do you check that a gradient is correct?

You check a gradient by comparing it with a numerical estimate from centred finite differences, (f(x + h) minus f(x minus h)) divided by 2h, for a small h. A relative error close to zero means the analytic gradient is right, and a large one points to a bug in the backward pass that would otherwise never raise an error.

Practise it on Gradient-Check a Scalar Neuron and then on Gradient-Check an Autodiff Graph.

Why do gradients accumulate in PyTorch?

Gradients accumulate in PyTorch because each call to backward adds to the .grad field of a tensor instead of overwriting it. This is correct when one value feeds several paths in the graph, and it also makes gradient accumulation over several batches possible, which is why you call zero_grad before each new backward pass.

The Accumulate Shared-Path Gradients problem shows why the sum is the right answer.

What activation function questions come up?

Activation function questions ask why a network needs nonlinearity at all and why ReLU and its variants replaced sigmoid and tanh in hidden layers. A strong answer connects each activation to the gradient it produces.

Why does a neural network need a nonlinear activation?

A neural network needs a nonlinear activation because a stack of linear layers without one is equivalent to a single linear layer. The nonlinearity between layers is what lets a deep network represent functions that a linear model cannot.

The Activation Functions problem has you implement the common ones side by side.

Why did ReLU replace sigmoid and tanh in hidden layers?

ReLU replaced sigmoid and tanh in hidden layers because its gradient is exactly 1 for every positive input, while the sigmoid gradient is at most 0.25 and both sigmoid and tanh saturate towards zero gradient for large inputs. ReLU is also cheap to compute. Its weakness is the dying ReLU, a unit whose input is negative for every example and so never receives a gradient, which Leaky ReLU avoids with a small slope for negative inputs.

The AlexNet ReLU lesson covers why ReLU made deeper networks practical, and the chain rule lesson shows how small derivatives shrink a gradient layer by layer.

What is GELU, and why do transformers use it?

GELU multiplies its input by the standard normal cumulative distribution at that input, x times Φ(x), which gives a smooth curve close to ReLU for large positive and negative values. Models such as BERT and GPT-2 use it in their feed-forward layers, where its smoothness near zero tends to train well.

Implement it in the GELU Activation problem.

What weight initialization questions should you prepare for?

Weight initialization questions ask why the starting weights matter and how Xavier and He initialization choose their scale. The underlying idea is that the variance of the activations and gradients should stay roughly the same from layer to layer.

Why can you not initialize all the weights to zero?

You cannot initialize all the weights to zero because every unit in a layer would then compute the same output and receive the same gradient, so the units would stay identical throughout training. Random initialization breaks this symmetry, while biases can safely start at zero.

The Weight Initialization problem has you set the starting weights of a PyTorch model yourself.

What is the difference between Xavier and He initialization?

Xavier initialization sets the weight variance to 2 divided by the sum of fan-in and fan-out, which keeps the variance stable for activations such as tanh that are roughly linear near zero. He initialization sets the variance to 2 divided by fan-in, which makes up for ReLU setting about half of its inputs to zero. Use He initialization with ReLU-family activations and Xavier with tanh or sigmoid.

Write both in the Xavier Initialization and He Initialization problems.

What normalization questions do interviewers ask?

Normalization questions focus on how batch normalization works, why it behaves differently at training and inference time, and when layer normalization is the better choice. The train and eval difference is one of the most common follow-ups in the whole interview.

How does batch normalization work?

Batch normalization normalizes each feature using the mean and variance of the current mini-batch, then applies a learned scale and shift so the layer can still represent any output range. During training it also keeps running averages of the mean and variance, and at inference it uses those running averages instead of batch statistics, so a single example gives a deterministic output.

The batch normalization lesson explains the effect on training, and the Batch Normalization (Forward) problem has you write the forward pass.

What is the difference between batch normalization and layer normalization?

Batch normalization computes statistics over the batch for each feature, while layer normalization computes them over the features of each example. Layer normalization therefore does not depend on batch size and behaves the same at training and inference, which is why transformers and recurrent networks use it, and batch normalization remains common in CNNs trained with reasonably large batches.

Compare the two by solving Layer Normalization after the batch normalization problem.

What regularization and dropout questions come up?

Regularization questions ask how dropout works, how weight decay differs from an L2 penalty, and what else reduces overfitting. Interviewers usually expect you to name more than one method and say when each one helps.

How does dropout work, and why is it scaled?

Dropout sets each activation to zero with probability p during training and scales the surviving activations by 1 divided by (1 minus p), so the expected value of each activation stays the same. At inference dropout is switched off and the layer passes its input through unchanged. Because units cannot rely on any particular other unit being present, the network learns more redundant features.

Implement it in the Dropout (Training Mode) problem, and read the AlexNet dropout lesson for where it first made a large difference.

What is the difference between L2 regularization and weight decay?

L2 regularization adds the squared size of the weights to the loss, and with plain SGD this gives exactly the same update as shrinking the weights by a fixed fraction each step, which is called weight decay. With Adam the two differ, because the L2 gradient is divided by the adaptive scaling term, so weights with large past gradients are regularized less. AdamW fixes this by applying weight decay directly to the weights, separately from the gradient step.

The regularization lesson covers the penalty view, and the AdamW (Decoupled Weight Decay) problem has you write the decoupled update.

What other methods reduce overfitting in a neural network?

Other common methods are early stopping on a validation metric, data augmentation, label smoothing and using a smaller model or more data. Early stopping keeps the weights from the epoch with the best validation score, and label smoothing replaces one-hot targets with slightly softened ones so the model is less overconfident.

Practise two of them in the Early Stopping and Label Smoothing Loss problems.

What optimizer and learning rate questions do interviewers ask?

Optimizer questions ask what momentum and Adam add to plain gradient descent, why Adam needs bias correction, and how the learning rate schedule affects training. Our guide on which optimizer to use covers the practical choice in more depth.

What is the difference between SGD with momentum and Adam?

SGD with momentum keeps an exponential moving average of past gradients and steps in that direction, which smooths out noise and speeds up progress along consistent directions. Adam keeps the same kind of average and also an average of squared gradients, and it divides each step by the square root of the second average, so every parameter gets its own effective learning rate.

The momentum lesson and the adaptive learning rates lesson explain both, and the Classical Momentum and Adam Optimizer Step problems have you write each update.

Why does Adam use bias correction?

Adam uses bias correction because its two moving averages start at zero and are therefore biased towards zero in the first steps. Dividing the first average by 1 minus beta1 to the power t, and the second by 1 minus beta2 to the power t, removes that bias, and the correction fades as the step count t grows.

The guide on which optimizer to use compares Adam with the other common choices.

What happens when the learning rate is too high or too low, and why use warmup?

A learning rate that is too high makes the loss oscillate or diverge, and one that is too low makes training slow or leaves it stuck. Warmup starts with a small learning rate and raises it over the first steps, which avoids large unstable updates while the optimizer statistics are still poor estimates, and a decay schedule such as cosine annealing then lowers it so training can settle.

Write the schedules in the Warmup + Linear Decay LR Schedule and Cosine Annealing LR Scheduler problems.

How does batch size affect training?

A larger batch gives a less noisy estimate of the gradient and uses the hardware more efficiently, but it also means fewer updates per epoch. The learning rate usually needs to be retuned when the batch size changes, and very small batches add noise that can act as a mild regularizer.

The Batch vs Stochastic vs Mini-Batch GD problem compares the three on the same task, and the SGD variants lesson explains the trade-off.

What loss function questions should you expect?

Loss function questions ask which loss fits a task, why losses take logits instead of probabilities, and how to handle class imbalance or outliers. Our guide on which loss function to use has a full decision table by task.

Why should a classification loss take logits instead of probabilities?

A classification loss should take logits because computing the log of a softmax directly with the log-sum-exp trick is numerically stable, while taking the log of a probability that has already rounded to zero gives infinity. This is why CrossEntropyLoss in PyTorch expects raw logits and BCEWithLogitsLoss is preferred over applying a sigmoid and then BCELoss.

Practise the stable version in Numerically Stable Softmax and then Cross-Entropy Loss.

How does focal loss help with class imbalance?

Focal loss multiplies the cross-entropy of each example by (1 minus p_t) to the power gamma, where p_t is the predicted probability of the true class. Easy examples that the model already gets right then contribute very little, so training concentrates on hard and rare examples, which helps when one class dominates the data.

Implement it in the Focal Loss problem.

When would you use Huber loss instead of mean squared error?

You would use Huber loss when the regression targets contain outliers, because it is quadratic for small errors and linear for errors larger than a threshold delta. Large errors then have a bounded gradient and cannot dominate training the way they do with mean squared error.

Write it in the Implement Huber Loss problem, and see the guide on which loss function to use for other tasks.

What CNN questions do interviewers ask?

CNN questions ask why convolutions suit images, how to compute output shapes and parameter counts, and why residual connections made very deep networks trainable. The shape and parameter arithmetic is a frequent quick check, so practise it until it is automatic.

Why use a convolutional layer instead of a fully connected one for images?

A convolutional layer suits images because it connects each output only to a small local patch and reuses the same filter weights at every position. This weight sharing needs far fewer parameters than a fully connected layer, and it makes the layer translation equivariant, so a pattern is detected wherever it appears in the image.

The 2D Convolution from Scratch problem has you write the sliding-window operation yourself.

How do you compute the output size and parameter count of a convolution?

The output height of a convolution is the floor of (H + 2P minus K) divided by S, plus 1, for input height H, padding P, kernel size K and stride S, and the width follows the same formula. The parameter count is K times K times the input channels times the output channels, plus one bias per output channel.

Check your arithmetic against code in the 2D Convolution (Image Filtering) and Simple CNN Layer (NumPy) problems.

What problem do residual connections solve?

Residual connections solve the degradation problem, where adding more layers to a plain network made even its training error worse. A residual block outputs F(x) plus x, so the layers only need to learn a change to their input, and the identity path gives the gradient a direct route back through the network.

The ResNet skip connection lesson explains the idea, and the ResNet Residual Block problem has you build one.

What RNN and LSTM questions come up?

RNN questions ask why vanilla recurrent networks struggle with long sequences and how LSTM and GRU cells fix that. They are a natural place for interviewers to test whether you understand gradient flow through time.

Why do vanilla RNNs struggle with long sequences?

Vanilla RNNs struggle with long sequences because backpropagation through time multiplies the gradient by the recurrent weight matrix and the activation derivative once per time step. Over many steps that product tends to shrink towards zero or grow without bound, so the network either cannot learn long-range dependencies or becomes unstable.

The RNN vanishing gradients lesson walks through the product, and the RNN Step Forward (Tanh Cell) and RNN Step Backward (Vanilla RNN) problems have you write both passes.

How does an LSTM fix the vanishing gradient problem?

An LSTM adds a cell state that is updated by addition, where the forget gate scales the old cell state and the input gate adds new information. When the forget gate stays close to 1, the gradient can flow back through many steps of the cell state without being repeatedly squashed, and the output gate controls how much of the cell state becomes the hidden state.

The LSTM cell state lesson explains the additive path, and the LSTM Cell problem has you write all four gate computations.

What is the difference between a GRU and an LSTM?

A GRU uses two gates, an update gate and a reset gate, and keeps a single hidden state, while an LSTM uses three gates and a separate cell state. The GRU has fewer parameters per unit and often performs similarly, so the choice usually comes down to trying both on the task.

Build one in the Build a Mini GRU Cell (Forward Pass) problem, with the GRU cell lesson alongside.

What training problem questions do interviewers ask?

Training problem questions describe a run that misbehaves and ask what causes it and what you would do. Good answers name the likely causes in order and the check that rules each one out.

What causes vanishing and exploding gradients, and how do you fix them?

Vanishing and exploding gradients come from multiplying many layer derivatives together during backpropagation, which shrinks the gradient towards zero or grows it without bound in deep or recurrent networks. The usual fixes are ReLU-family activations, careful initialization, normalization layers and residual connections, and exploding gradients are also controlled by clipping the global gradient norm before each update.

The Gradient Clipping (Global Norm) problem has you write the clipping step, and Chain Rule Through a 3-Layer Network shows how the product of derivatives builds up.

How do you recognise and fix overfitting?

You recognise overfitting when the training loss keeps falling while the validation loss stops improving or starts to rise. The fixes are more data or data augmentation, regularization such as dropout or weight decay, early stopping and a smaller model, and you should confirm each change on the validation set.

The regularization lesson covers the main methods, and Dropout (Training Mode) is a good one to implement first.

What do you check first when the training loss is not going down?

When the training loss is not going down, first check that the loss at the start is close to the natural log of the number of classes for a classifier, then try to overfit a single small batch. If the model cannot drive the loss on one batch close to zero, the bug is in the model, the loss or the training loop, and the learning rate is the next thing to test.

The guide on a training loss that is not decreasing lists the causes in the order to check them, and the Train a Tiny Micrograd MLP problem has you write a loop that trains.

What about transformer and attention questions?

Transformer and attention questions come up in many deep learning interviews, and they usually start with scaled dot-product attention, multi-head attention and layer normalization. The guide to building an LLM from scratch covers each of these components with a problem to solve, and the Transformers section of the Cracking Deep Learning plan includes encoder and decoder blocks.

Which deep learning interview questions ask you to write code?

The deep learning questions most often asked as code are the ones whose answer is a short, self-contained function, such as a backward pass, a normalization layer, dropout, an optimizer step, a convolution or a recurrent cell. In these rounds the interviewer checks your shapes, your handling of the training and inference modes, and your numerical stability as well as the final output.

Deep learning interview questions that are often asked as code, what you write, and a problem to practise each
Question	What you write	Practice
Backpropagation for a small network	The forward pass, the cached activations and a gradient for every weight	Backpropagation (Single Hidden Layer)
Softmax with cross-entropy	A stable softmax and the mean loss over a batch	Cross-Entropy Loss
Batch normalization	Batch statistics, the learned scale and shift, and running averages	Batch Normalization (Forward)
Dropout	A random mask, inverted scaling and the switch between training and inference	Dropout (Training Mode)
An Adam step	Both moment estimates, bias correction and the parameter update	Adam Optimizer Step
A 2D convolution	The sliding window with padding and stride, and the output shape	2D Convolution from Scratch
An LSTM cell	The four gate computations, the cell state and the hidden state	LSTM Cell
A residual block	Two layers with a skip connection added to their output	Residual Block
Gradient clipping	The global norm over all gradients and the rescaling step	Gradient Clipping (Global Norm)

To rehearse under time pressure, the Neural Net Internals Sprint gives you 60 minutes for two neural network problems, and the Activations, Losses, and Optimizers sprint gives you 30 minutes for three implementations.

How should you practise deep learning interview questions?

Practise deep learning interview questions by answering each one out loud first and then writing the code for it, because writing the code exposes the gaps that a spoken answer hides. Work through the topics in the order of this guide, since later topics such as vanishing gradients and residual connections depend on the earlier ones.

TensorTonic study plans for deep learning interview preparation, with their size and coverage
Study plan	Size	What it covers
Cracking Deep Learning	35 problems in six sections	Foundations, CNNs, RNNs, Transformers, generative models and advanced architectures
PyTorch	30 problems in six sections	Tensors and autograd, layers, training loops, data loading, and convolutions, recurrent cells and attention from scratch
Build Micrograd from Scratch	25 problems in seven sections	A small autograd engine, from numerical derivatives to training an MLP on it
Calculus for ML	30 problems in six sections	Derivatives, Jacobians, Hessians and the gradients used in training
Optimization	25 problems in six sections	Gradient descent, momentum, Adam, learning rate schedules, regularization and second-order methods

Start with Cracking Deep Learning if you have limited time, since it covers the architectures and training techniques in one plan. Add Build Micrograd from Scratch if backpropagation still feels uncertain, and use the chain rule and loss landscape lessons when a derivation or an optimizer question needs more background.

What mistakes do candidates make on deep learning questions?

Reciting a definition without the reason behind it leaves the interviewer to ask the follow-up, so say why the technique works and when it fails in the same answer.
Forgetting that batch normalization and dropout behave differently in training and inference is one of the most common errors, and it is also a common real bug in evaluation code.
Describing Adam with weight decay as the same thing as AdamW misses the point of AdamW, which applies the decay directly to the weights.
Losing track of tensor shapes in a derivation or a convolution question makes the arithmetic fall apart, so write the shape of each tensor as you go.
Applying a sigmoid or softmax before a loss that expects logits applies it twice and harms both stability and learning.
Naming a single fix for overfitting or vanishing gradients sounds thin, and interviewers expect two or three options with the trade-off between them.

The guide to ML interview mistakes covers the mistakes that cost candidates across the other rounds of the loop.

Practice this

Backpropagation (Single Hidden Layer) covers the chain rule, cached activations and weight gradients that most other questions build on.
Batch Normalization (Forward) makes you handle batch statistics and the learned scale and shift, which prepares you for the train and eval follow-up.
LSTM Cell has you write every gate and the cell state, which is the clearest way to explain how an LSTM keeps gradients alive.

Then continue with the Cracking Deep Learning study plan, or take the Neural Net Internals Sprint to test yourself against the clock.

Do you need to know PyTorch for a deep learning interview?

You need PyTorch for most practical deep learning interviews, because interviewers ask how autograd records operations, why gradients accumulate, and what model.train() and model.eval() change. Conceptual questions can be answered without any framework, but coding questions are usually set in PyTorch or NumPy. Knowing both lets you follow whichever one the prompt uses.

Should you memorise the backpropagation equations?

You should be able to derive the backpropagation equations for a two-layer network from the chain rule instead of reciting them. Interviewers often change the activation or the loss and ask for the new gradient, which a memorised formula does not cover. Tracking the shape of every tensor as you go is the fastest way to catch a mistake in the derivation.

Are CNN and RNN questions still asked now that transformers are common?

CNN and RNN questions are still asked, because they test convolution arithmetic, weight sharing and gradient flow through time, which are core ideas in deep learning. Computer vision roles usually go deeper into CNNs, and RNN questions often lead into why attention replaced recurrence for long sequences.

How are deep learning interview questions different from ML coding questions?

Deep learning interview questions are mostly spoken explanations of why a technique works and when it fails, while ML coding questions ask you to implement a model or layer from scratch. The two overlap on layers such as batch normalization, dropout and attention, which can be asked either way. Practising the code for a concept also makes your spoken answer more precise.

Are deep learning interview questions the same for ML engineers and research scientists?

The core deep learning questions are the same for both roles, since both need backpropagation, optimizers, normalization and the common architectures. Research scientist interviews usually go further into derivations and recent papers. ML engineer interviews usually spend more time on training stability, debugging and how a model runs in production.