Training Loss Not Decreasing: Symptoms and Causes

Name: TensorTonic
Author: TensorTonic

A diagnostic table for broken training runs that starts from the shape of the loss curve and gives the likely causes and the first check to run for each one.

By TensorTonic's founderLast updated November 1, 202614 min read

Why is my training loss not decreasing?

Training loss usually stops decreasing because the learning rate is wrong, the gradient never reaches the weights, the inputs and labels do not line up, or the loss is computed incorrectly. Check the first loss against its expected value, then try to overfit one small batch. If that batch will not fit, the bug is in your code.

Training loss symptoms, their likely causes, what to check first and where to learn more
Symptom	Likely causes	What to check	Learn and practise
The loss does not move
Loss is flat from the very first step.	The gradient never reaches the weights. Parameters are frozen, the optimizer was built over a different parameter list, the loss was computed from a detached tensor or a Python float, or the learning rate is zero because a warmup or schedule starts at zero.	Print the gradient norm of every parameter after one backward pass, since a None or exactly zero gradient marks the break. Print the learning rate the optimizer used at step 1 and confirm the weights change between two steps.	Backpropagation Computational graphs Apply One Full SGD Training Step
Loss sits at ln(C) for a C-class classifier, or at about 0.693 for a binary one.	The model predicts the uniform distribution and has learned nothing. Labels may be shuffled apart from the inputs, the inputs may all be identical or zero after preprocessing, or the learning rate may be far too low.	Compare the loss with ln(C), which is 2.303 for 10 classes, 4.605 for 100 and 6.908 for 1,000. A loss pinned there means chance-level prediction, so look at a few inputs beside their labels and check that the logits differ across examples.	Cross-entropy Implement Cross-Entropy Loss
Loss drops once, then sits at the entropy of the label distribution.	The model has learned the class frequencies and ignores the inputs. That happens when inputs and labels are misaligned or when a preprocessing bug removes the signal.	Compute the entropy of the training labels, which is 0.325 nats for a 90 to 10 binary split. If the loss sits at that value, plot inputs next to labels and check the variance of an input batch.	Shannon entropy Compute Entropy for a Node
Regression loss sits at the variance of the targets.	The model predicts a constant equal to the target mean, because the mean squared error of always predicting the mean equals the variance of the targets. Unscaled targets, inputs with no usable signal or a tiny learning rate keep it there.	Compute the variance of the training targets and compare it with the loss. Standardise the inputs and the targets, and fit a linear baseline to confirm the inputs carry signal.	Basic statistics Implement z-Score Standardization
Loss decreases very slowly in a smooth, almost straight line.	The learning rate is too low, inputs are on very different scales, or a deep network with sigmoid or tanh activations and no normalisation passes tiny gradients to its early layers.	Raise the learning rate by 3x to 10x and see whether the slope steepens. Compare the gradient norms of the first and last layers, since a gap of several orders of magnitude points to vanishing gradients.	SGD variants Chain rule Implement ReLU Activation
The loss moves the wrong way
Loss rises or explodes within the first few hundred steps.	The learning rate is too high, a transformer is trained with Adam and no warmup, the learning rate was scaled up with a larger batch and no warmup was added, or the weights were initialised too large.	Cut the learning rate by 10x and add a linear warmup. Log the global gradient norm, because a norm that grows step after step confirms divergence.	SGD variants Warmup + Linear Decay LR Schedule
Loss becomes NaN or inf.	A hand-written loss takes log(0) or divides by zero, softmax is computed without subtracting the maximum, gradients explode, an fp16 activation or gradient overflows past 65,504, or the data already contains a NaN.	Assert that inputs and labels are finite, use the library loss that takes logits, and turn on anomaly detection to find the first operation that produced a NaN. Train in bf16, or keep dynamic loss scaling on in fp16 so that overflowing steps are skipped, and clip the gradient norm.	Mixed precision Gradient Clipping (Global Norm) Binary Log Loss per Sample
Loss bounces up and down without trending down.	The learning rate is too high for the batch size, the batch is so small that the gradient estimate is very noisy, or momentum is too high.	Plot a moving average of the loss before judging it. Halve the learning rate or double the batch size and compare, and if the averaged curve trends down the run is only noisy.	Momentum and Nesterov Batch Shuffling and Mini-Batch Generator
Loss falls, then spikes sharply and sometimes recovers, which is common in large transformers.	The peak learning rate is too high for later training, warmup is too short, a corrupted batch slipped in, or attention logits grow until the softmax saturates.	Plot the gradient norm and the learning rate beside the loss and see whether the spikes line up. Lower the peak learning rate, lengthen warmup, clip the global gradient norm and inspect the batches at the spike steps.	GPT pretraining Generate a Warmup-Stable-Decay Schedule
The loss moves but the shape is suspicious
Loss drops to almost exactly zero within a few steps.	Every label is the same class, the target or a feature computed from it is among the inputs, or the loss compares a tensor with itself.	Count the labels in each class first. Then check how well each feature predicts the target on its own, since one feature with near-perfect accuracy is usually a leak, and recompute the loss for one example by hand to confirm that the prediction and the target are different tensors.	Data splits and leakage
Loss is negative when it should not be.	Negative log-likelihood was given probabilities instead of log-probabilities, binary cross-entropy got targets outside 0 and 1, or a custom loss has a sign error. Cross-entropy and mean squared error are never negative for valid inputs.	Check whether the loss expects logits, probabilities or log-probabilities, check the range of the targets, and recompute the loss for one example by hand. A Gaussian or other continuous negative log-likelihood can be negative legitimately, because a density can exceed 1.	Which loss function to use Implement Cross-Entropy Loss
Loss stops at a high floor, even on a tiny subset of the data.	A model that cannot fit one batch has a bug. Common ones are a softmax before a loss that already applies log-softmax, a sigmoid before a logits-based binary loss, one-hot labels where class indices are expected, padding tokens counted in the loss, or too little capacity.	Try to overfit a single batch of 2 to 32 examples with regularisation off. A healthy model and loss drive it near zero within a few hundred steps, and if it cannot, the bug is in the model or the loss.	Which loss function to use Build an Assistant-Only SFT Loss Mask One-Hot Encoding (Multi-class)
Loss drops in sharp steps at fixed epochs.	A step learning-rate schedule produces this, and it is expected behaviour.	Overlay the learning-rate schedule on the loss plot. If the drops do not line up with it, check whether the data loader changes the data partway through training.	Learning Rate Scheduler (Linear Decay) Cosine Annealing LR Scheduler
Loss repeats a sawtooth pattern with the same period as an epoch.	The data is not shuffled, so each epoch walks through classes or sources in the same order and the model chases each block in turn.	Confirm that shuffling is on in the training loader and is redone each epoch, then look at the label distribution of consecutive batches.	Batch Shuffling and Mini-Batch Generator
Loss plateaus for a long time, then suddenly drops.	The optimizer was crossing a saddle point or a flat region, which is normal in deep networks, or a warmup or schedule phase ended.	Keep training if the gradient norm is not zero and the loss is still creeping down. Check whether the drop lines up with a schedule change.	Loss landscapes Local minima and saddle points
Training loss and validation loss disagree
Training loss falls while validation loss rises.	The model is overfitting, which is common with small datasets, large models or many epochs.	Find the epoch with the lowest validation loss and stop there. Add weight decay, dropout or augmentation, get more data or shrink the model.	Regularization Implement Dropout (Training Mode)
Training loss and validation loss are both high and both flat.	The model is underfitting because it is too small, trained too briefly, regularised too strongly or given features with little signal.	Compare with a simple baseline, then increase capacity, train longer and reduce weight decay and dropout. If a larger model also fails to beat the baseline, the features are the problem.	Building a baseline
Validation loss is lower than training loss.	This is usually harmless. Dropout and augmentation are on during training and off during evaluation, and training loss is averaged over an epoch while the model was still improving. A large, persistent gap can mean the splits overlap.	Compute the training loss in evaluation mode on a fixed subset, and check for duplicate rows across the split.	Regularization
Validation loss is far worse than training loss from the first epoch.	The splits are preprocessed differently, the model is left in training mode during evaluation, BatchNorm runs on very small batches, or the splits come from different distributions.	Run one batch through the training and validation preprocessing and compare the tensors. Confirm evaluation mode is set before validation.	Batch normalization Batch Normalization (Forward)
Loss decreases but accuracy or the target metric does not improve.	The classes are imbalanced and the model predicts the majority class, the decision threshold is wrong, the metric has a bug, or the loss does not match the metric.	Look at the confusion matrix and the distribution of predicted classes, and compare with a majority-class baseline.	Which evaluation metric to use Compute Confusion Matrix with Normalization
Validation loss rises while validation accuracy still rises.	The model is growing overconfident on the examples it gets wrong. Cross-entropy punishes confident mistakes heavily, so the loss can worsen while accuracy improves.	Treat it as an early sign of overfitting. Pick the checkpoint by the metric you care about and consider label smoothing or stronger regularisation.	Label Smoothing Loss

Each row starts from what you see on the loss chart and lists the likely causes and the cheapest check to run first. On a phone, scroll the table sideways to see every column.

What does a broken loss curve look like?

A broken loss curve usually has a recognisable shape and sits at a recognisable value, so the chart narrows the cause before you read any code. The monitor below draws six of the shapes from the table, with the reference values that identify them as dashed green lines.

Loss monitor

Six traces and what each one means

training step

What you see

The loss starts at ln 10 for a ten-class model and never leaves it, apart from a tiny wobble.

Likely causes

No gradient reaches the weights because a layer is frozen, the tensor was detached or the optimizer holds a different parameter list.
The optimizer step is missing from the loop.
The learning rate is zero because a warmup or schedule starts at zero and never rises.

Check first

Run one backward pass and print the gradient norm of every parameter. A None or exactly zero gradient shows where the chain breaks.

Switch to the log scale to see which flat lines are really flat. Trace E stays flat on both scales, while trace F keeps falling.

Two of the six are easy to mistake for each other. A run that has stopped learning stays flat on a log axis, while a run that is only converging slowly keeps sloping down once the axis is logarithmic, so always look at both before you change anything.

What should the loss be at the first step?

For a classifier with C classes and small random weights, the first cross-entropy loss should be close to ln(C), because the model starts by predicting every class with roughly equal probability.

\mathcal{L}_0 \approx \ln C, \qquad \ln 2 \approx 0.693, \quad \ln 10 \approx 2.303, \quad \ln 1000 \approx 6.908

A language model with a 50,257-token vocabulary should therefore start near 10.82. A first loss far above ln(C) means the initial logits are too large, which usually comes from unscaled inputs or an initialisation that is too wide. The usual fix is He initialisation for ReLU layers and Xavier initialisation for tanh or sigmoid layers.

\text{He: } \operatorname{Var}(W) = \frac{2}{n_{\text{in}}} \qquad \text{Xavier: } \operatorname{Var}(W) = \frac{2}{n_{\text{in}} + n_{\text{out}}}

For regression with mean squared error and outputs that start near zero, the first loss is close to the mean of the squared targets, which equals their variance once the targets are standardised. The more useful reference is the loss of always predicting the mean of the targets, which equals their variance, so a loss stuck at that value means the model has learned a constant.

\frac{1}{n}\sum_{i=1}^{n} (y_i - \bar{y})^2 = \operatorname{Var}(y)

On an imbalanced binary task, the loss often drops from ln 2 to the entropy of the label split in the first few steps and then looks stuck. Setting the output bias to the log-odds of the positive rate p makes the model start at that value, so the early drop disappears and any further progress is real learning. The lessons on cross-entropy and Shannon entropy derive both reference values.

b_0 = \ln \frac{p}{1 - p}

How do you check that a model can learn at all?

Take one batch of 2 to 32 examples, turn off dropout, weight decay and augmentation, and train on that batch alone for a few hundred steps. A working model and loss should push the loss close to zero, because a network with enough parameters can memorise a handful of examples.

This single test splits every broken run into two groups. If the batch fits, the code can learn, and the problem lies in the data, the learning rate or the regularisation. If it does not fit, the bug is in the model, the loss or the training loop, and changing the dataset or the optimizer will not help. The loop below runs the test and also prints which parameters received no gradient on the first step.

xb, yb = next(iter(train_loader))
for step in range(300):
    opt.zero_grad()
    loss = loss_fn(model(xb), yb)
    loss.backward()
    if step == 0:
        missing = [n for n, p in model.named_parameters() if p.requires_grad and p.grad is None]
        norms = {n: p.grad.norm().item() for n, p in model.named_parameters() if p.grad is not None}
        print("no gradient:", missing)
        print("smallest:", min(norms.items(), key=lambda kv: kv[1]))
    opt.step()
    if step % 50 == 0:
        print(step, loss.item())

Look first at the list of parameters with no gradient, since a frozen layer or a detached tensor shows up there immediately. The loss pairings that most often stop a batch from fitting, such as a softmax placed before a loss that already applies log-softmax, are covered in the guide on which loss function to use.

How do you tell whether the learning rate is the problem?

Train a few hundred steps at three or four learning rates spaced a factor of ten apart and compare the curves. As a rule of thumb, a rate that is too high makes the loss bounce, spike or rise, while a rate that is too low gives a smooth line that barely slopes. The SGD variants lesson walks through a full learning-rate range test, and the backpropagation lesson shows the three regimes side by side.

The reason a large rate diverges is easiest to see on a quadratic loss. Gradient descent only converges when the step is smaller than two divided by the largest curvature, which is the largest eigenvalue of the Hessian.

0 < \eta < \frac{2}{\lambda_{\max}}

Real networks are not quadratic, and their curvature is often highest early in training, which is one reason warmup helps. A short warmup lets the model reach a smoother region before the full learning rate applies, and with Adam it also gives the second-moment estimates time to settle. The Jacobian and Hessian lesson explains where the curvature comes from.

Which gradient checks find a broken backward pass?

The fastest check is the gradient norm of every parameter after one backward pass, since a None gradient means the parameter is outside the graph and an exactly zero gradient means something upstream blocks it. Comparing the norms of the first and last layers also shows vanishing or exploding gradients, which the chain rule lesson explains as products of many small or large factors.

For a hand-written backward pass, compare the analytic gradient with a centred finite difference and look at the relative error between them.

\frac{\partial f}{\partial \theta_i} \approx \frac{f(\theta + h e_i) - f(\theta - h e_i)}{2h}, \qquad \text{rel. error} = \frac{\lvert g_a - g_n \rvert}{\max(\lvert g_a \rvert, \lvert g_n \rvert)}

As a rule of thumb, in float64 with h around 1e-5, a relative error near 1e-7 or below means the gradient is correct, and an error above about 1e-3 usually means a bug unless the input sits near a kink such as the zero point of a ReLU. The Gradient-Check a Scalar Neuron problem has you build this check yourself.

Once gradients are correct, the ratio of the update size to the weight size for each layer shows whether the learning rate fits. A rough heuristic for SGD-style training is a ratio near 1e-3 per step, where much smaller suggests the rate is too low and much larger suggests it is too high.

\frac{\lVert \Delta W \rVert}{\lVert W \rVert} \sim 10^{-3}

Why does my loss become NaN?

A loss becomes NaN when some value overflows or an undefined operation such as log(0) or 0 divided by 0 appears, and the cause is usually a learning rate that is too high, an unstable hand-written loss, low-precision overflow or bad data. Assert that inputs and labels are finite before training starts, because a single NaN in the data spreads to every weight within one step.

Hand-written losses should take logits and use a log-softmax, or clamp probabilities away from zero with a small epsilon. A softmax must subtract the largest logit before exponentiating, which leaves the result unchanged and keeps every exponent at or below zero.

\operatorname{softmax}(z)_i = \frac{e^{z_i - \max_j z_j}}{\sum_k e^{z_k - \max_j z_j}}

In mixed precision, fp16 can only represent values up to 65,504 and rounds small gradients to zero, so it needs dynamic loss scaling, which scales the loss up to keep small gradients representable and skips any step whose gradients overflow. bf16 keeps the exponent range of fp32 and rarely needs it. The mixed precision lesson covers both. Clipping the global gradient norm to a threshold c stops a single bad batch from throwing the weights far away.

g \leftarrow g \cdot \min\left(1, \frac{c}{\lVert g \rVert_2}\right)

What data bugs stop a model from learning?

The data bugs that stop learning most often are inputs and labels that no longer line up, preprocessing that erases the signal, an unshuffled loader and padding that counts toward the loss. Print a batch of inputs next to their labels before anything else, because a misaligned pair is invisible in the code and obvious on screen.

A quick way to confirm the inputs matter is to train for a short while with the labels shuffled. For the first few hundred steps that loss should stall near the label-prior entropy, since a network memorises random labels only slowly, and if your real run stalls at the same value, the model is ignoring the inputs. A real run whose training loss falls to almost zero within a few steps points the other way, usually toward labels that are all the same class or a feature that leaks the answer, which the lesson on data splits and leakage covers.

In sequence models, check that padding and prompt tokens are masked out of the loss, and that the loss is divided by the number of real tokens. When you accumulate gradients over k micro-batches, each micro-batch loss must be divided by k, or the summed gradient is k times too large, which with SGD multiplies the effective learning rate by k, as the gradient accumulation lesson explains.

Why is validation loss going up while training loss goes down?

Validation loss rising while training loss falls means the model is overfitting, and the training run itself is working. Stop at the epoch with the lowest validation loss, then add weight decay, dropout or augmentation, as the regularization lesson describes.

A rising validation loss with rising validation accuracy is a milder version of the same thing, where the model grows overconfident on the examples it gets wrong. If the loss falls while the metric you care about stays flat, the loss and the metric disagree, which the guide on which evaluation metric to use helps you untangle.

In what order should you debug a training run?

Debug a training run from the cheapest check to the most expensive one, so that each step rules out a whole family of causes before you touch the next.

Check that the first loss matches ln(C) for classification or the mean of the squared targets for regression.
Print a batch of inputs beside their labels and confirm that they match.
Overfit one small batch with dropout, weight decay and augmentation turned off.
Train a few hundred steps at learning rates a factor of ten apart and keep the best.
Log gradient norms per layer and the update-to-weight ratio for the run you keep.
Turn regularisation and augmentation back on one at a time and watch the validation loss.
Change the architecture or the optimizer only after every check above passes, using the guide on which optimizer to use.

Building the pieces by hand is the quickest way to recognise these failures on sight. The ML Math lessons cover the theory behind each check, and the guide on ML coding interview questions shows how often the same components come up in interviews.

What else do people ask about training loss that will not decrease?

Is it normal for training loss to plateau and then drop again?

Yes, a plateau followed by a sudden drop is common in deep networks and often means the optimizer was crossing a flat region or a saddle point. It is only a warning sign when the gradient norm is near zero for the whole plateau or when the loss is pinned at the chance value of ln(C).

Should I lower or raise the learning rate when the loss is not decreasing?

Lower it if the loss rises, spikes or bounces, and raise it if the loss falls in a slow, smooth line. If you cannot tell, train a few hundred steps at three learning rates a factor of ten apart and keep the one whose loss falls fastest without spiking.

Can the optimizer be the reason my loss is not decreasing?

It rarely is on its own. A wrong learning rate, a missing warmup or a bug in the model, the loss or the data explains almost every flat curve, so change the optimizer only after the starting loss is right and the model can overfit one batch.

How many epochs should I wait before deciding the loss is not decreasing?

Counting epochs is the wrong test, because the checks that matter take only a few hundred steps. They test whether the first loss matches its expected value, whether the model can overfit one small batch and whether the loss moves on a log scale.

Why does my fine-tuning loss not decrease when pretraining worked?

The usual causes are a learning rate copied from pretraining, a loss mask that hides every target token or counts the prompt and padding, a chat template that differs from the one the model was trained on, or layers frozen by mistake. A pretrained model should start far below ln of its vocabulary size, so check the first loss and the number of tokens that count toward it in each batch.

Does a larger model fix a loss that will not go down?

A larger model helps only when the current one is underfitting, which means it can overfit one batch but stalls on the full data. If the model cannot overfit one batch, a larger model will not help, because the cause is almost always a bug in the code.

Practice this

Each of these problems has you implement one of the fixes from the table, which makes the failure it prevents much easier to spot in your own runs. Then browse the problem library.

Gradient Clipping (Global Norm) has you rescale gradients so one bad batch cannot blow up the run.
Gradient-Check a Scalar Neuron has you compare an analytic gradient with a finite difference.
Implement Numerically Stable Softmax has you write the max-subtraction trick that prevents overflow to NaN.
Warmup + Linear Decay LR Schedule has you build the schedule that keeps early training from diverging.