Which Loss Function Should You Use? A Decision Table by Task

Name: TensorTonic
Author: TensorTonic

By TensorTonic's founderLast updated November 1, 202610 min read

Which loss function should I use?

Pick the loss that matches your target and how its errors behave. Use mean squared error for ordinary regression, Huber or mean absolute error when outliers are common, binary cross-entropy for yes or no labels and multi-label tasks, softmax cross-entropy for one class out of many, and a contrastive or ranking loss when only similarity or order matters.

In the formulas,

y

is the target,

\hat{y}

is the prediction,

r = y - \hat{y}

the error,

z

a logit,

p = \sigma(z)

a probability and

K

the number of classes. Each formula is the loss for one example, averaged over the batch.

Your task	Use this loss	Formula	Output and PyTorch	Watch out for	Learn and practise
Regression
Predict a number when the noise is roughly normal and outliers are rare	Use this lossMean squared error (MSE)	Formula $L = (y - \hat{y})^2$	Output and PyTorchOne linear output with no activation.`nn.MSELoss`	Watch out forIt predicts the conditional mean and is the negative log-likelihood under Gaussian noise, so a few large errors dominate the total. Standardise the target before training.	Learn and practiseMaximum likelihood estimationMean Squared Error (MSE)Calculate Squared-Error Loss Gradients
Predict a number when outliers or heavy tails should not dominate	Use this lossMean absolute error (MAE, L1 loss)	Formula $L = \|y - \hat{y}\|$	Output and PyTorchOne linear output with no activation.`nn.L1Loss`	Watch out forIt predicts the conditional median. Its gradient keeps the same size near the optimum, so lower the learning rate late in training or switch to Huber.	Learn and practiseBasic statistics
Predict a number with some outliers while keeping smooth gradients near zero	Use this lossHuber loss (Smooth L1 is a scaled form)	Formula $L = \begin{cases} \tfrac{1}{2} r^2 & \|r\| \le \delta \\ \delta\left(\|r\| - \tfrac{1}{2}\delta\right) & \|r\| > \delta \end{cases}$	Output and PyTorchOne linear output with no activation.`nn.HuberLoss(delta) or nn.SmoothL1Loss(beta)`	Watch out forIt is quadratic for errors below δ and linear above it. Set δ near the error size you would call an outlier, in the units of the scaled target, and start from δ = 1.	Learn and practiseBasic statisticsImplement Huber Loss
Forecast a percentile or an interval, or charge over and under prediction differently	Use this lossQuantile (pinball) loss	Formula $L_\tau = \max\big(\tau r,\ (\tau - 1)\, r\big)$	Output and PyTorchOne linear output per quantile.`No built-in loss, so write it in one line`	Watch out forThe minimiser is the τ-quantile, and τ = 0.5 gives half the MAE. Train τ = 0.1 and τ = 0.9 heads for an 80% interval, and sort the outputs if they cross.	Learn and practiseBasic statisticsPercentiles / Quantiles
Predict a count, such as clicks, events or arrivals	Use this lossPoisson negative log-likelihood	Formula $L = e^{z} - y\,z$ $z = \log \text{rate}$	Output and PyTorchOne linear output read as the log of the rate.`nn.PoissonNLLLoss(log_input=True)`	Watch out forUse it when the variance grows with the mean. MSE on raw counts gives the largest counts far too much weight.	Learn and practiseProbability distributionsPoisson PMF and CDF in Python
Predict a number and how uncertain the model is about it	Use this lossGaussian negative log-likelihood	Formula $L = \tfrac{1}{2}\left(\log \sigma^2 + \frac{(y - \mu)^2}{\sigma^2}\right)$	Output and PyTorchTwo heads, a linear mean μ and a variance σ² kept positive with softplus.`nn.GaussianNLLLoss`	Watch out forClamp the variance from below. Without a floor the model can shrink the variance on easy points until the loss diverges.	Learn and practiseMaximum likelihood estimation
Classification
Predict one yes or no label	Use this lossBinary cross-entropy (log loss)	Formula $L = -\big[y \log p + (1 - y)\log(1 - p)\big]$ $p = \sigma(z)$	Output and PyTorchOne logit, with the sigmoid applied inside the loss.`nn.BCEWithLogitsLoss`	Watch out forPass raw logits, because the logits version is numerically stable. The cross-entropy lesson shows why MSE trains poorly on this task.	Learn and practiseCross-entropyBinary Log Loss per Sample in PythonLogistic Regression Training Loop
Pick exactly one class out of K	Use this lossSoftmax (categorical) cross-entropy	Formula $L = -z_y + \log \sum_{k=1}^{K} e^{z_k}$	Output and PyTorchK logits, with the softmax applied inside the loss.`nn.CrossEntropyLoss with integer class ids`	Watch out forNever apply softmax before this loss. When the initial predictions are close to uniform, the first loss is close to ln K, which is 2.30 for ten classes.	Learn and practiseCross-entropyImplement Cross-Entropy Loss in PythonImplement Numerically Stable Softmax
Tag each example with any number of labels at once	Use this lossBinary cross-entropy on each label	Formula $L = \frac{1}{K}\sum_{k=1}^{K} \mathrm{BCE}\big(y_k, \sigma(z_k)\big)$	Output and PyTorchK independent logits, one sigmoid each.`nn.BCEWithLogitsLoss with a float 0 or 1 target of shape [batch, K]`	Watch out forSoftmax is wrong here because it makes the labels compete. Tune a decision threshold for each label after training.	Learn and practiseCross-entropyBinary Log Loss per Sample in Python
Classify when one class is moderately rare	Use this lossClass-weighted cross-entropy	Formula $L = -w_y \log p_y$	Output and PyTorchThe same output layer as the binary or multi-class row.`nn.CrossEntropyLoss(weight=...) or nn.BCEWithLogitsLoss(pos_weight=...)`	Watch out forStart with pos_weight equal to negatives divided by positives, or weights inversely proportional to class frequency, then tune. With the mean reduction, PyTorch divides by the sum of the weights in the batch. Weighting shifts the predicted probabilities, so recalibrate if you need real probabilities.	Learn and practiseCross-entropyImplement Cross-Entropy Loss in PythonExpected Calibration Error
Classify with severe imbalance or a flood of easy negatives, as in dense object detection	Use this lossFocal loss	Formula $L = -\alpha_t (1 - p_t)^{\gamma} \log p_t$	Output and PyTorchOne sigmoid per class.`torchvision.ops.sigmoid_focal_loss`	Watch out forThe defaults from the RetinaNet paper are γ = 2 and α = 0.25, and γ = 0 gives back α-weighted cross-entropy. Focal loss pulls the scores away from the true frequencies, so check calibration before reading them as probabilities.	Learn and practiseCross-entropyImplement Focal LossBinary Focal Loss
Stop a classifier growing overconfident, or soften noisy labels	Use this lossCross-entropy with label smoothing	Formula $q = (1 - \varepsilon)\,\mathbf{1}_y + \tfrac{\varepsilon}{K}$ $L = -\sum_{k} q_k \log p_k$	Output and PyTorchK logits, as in the multi-class row.`nn.CrossEntropyLoss(label_smoothing=0.1)`	Watch out forThe usual value is ε = 0.1. The loss can no longer reach zero, because its floor is the entropy of the smoothed target.	Learn and practiseCross-entropyShannon entropyLabel Smoothing Loss
Train a maximum-margin linear classifier, as in an SVM	Use this lossHinge loss	Formula $L = \max\big(0,\ 1 - y\, f(x)\big)$ $y \in \{-1, +1\}$	Output and PyTorchOne raw score with no activation.`nn.MultiMarginLoss for the multi-class form`	Watch out forA point that clears the margin gives zero loss, so easy points stop contributing. The hinge loss does not produce probabilities.	Learn and practiseLagrange multipliers and KKTImplement Hinge Loss (Binary SVM)
Vision
Segment a small object in a large image	Use this lossDice loss, usually added to cross-entropy	Formula $L = 1 - \frac{2\sum p\,g + s}{\sum p + \sum g + s}$	Output and PyTorchA sigmoid or softmax for every pixel.`No built-in loss, so write a short custom one`	Watch out forIt scores overlap, so the many background pixels cannot swamp it. Dice alone can be unstable early in training, which makes Dice plus cross-entropy the common default.	Learn and practiseImplement Dice Loss
Regress bounding boxes for object detection	Use this lossGIoU loss, or Smooth L1 on box offsets	Formula $L = 1 - \mathrm{IoU} + \frac{\|C \setminus (A \cup B)\|}{\|C\|}$	Output and PyTorchBox coordinates or offsets from linear outputs.`torchvision.ops.generalized_box_iou_loss or nn.SmoothL1Loss`	Watch out forPlain 1 minus IoU has zero gradient when the boxes do not overlap, and GIoU fixes that with the smallest enclosing box C. CIoU adds terms for centre distance and aspect ratio.	Learn and practiseIntersection over Union (IoU)
Embeddings, retrieval and ranking
Learn embeddings from pairs labelled same or different	Use this lossContrastive (Siamese) loss	Formula $L = \tfrac{1}{2} y\, d^2 + \tfrac{1}{2}(1 - y)\max(0,\ m - d)^2$	Output and PyTorchOne embedding vector per input.`nn.CosineEmbeddingLoss for the cosine form, or a short custom loss`	Watch out forHere y = 1 marks a similar pair and d is the distance between the two embeddings. Similar pairs are pulled together and different pairs are pushed until they sit at least the margin m apart.	Learn and practiseDot product and normsImplement Contrastive Loss (Siamese)Implement Cosine Embedding Loss in Python
Learn embeddings from anchor, positive and negative triples	Use this lossTriplet loss	Formula $L = \max\big(0,\ d(a, p) - d(a, n) + m\big)$	Output and PyTorchOne embedding vector per input.`nn.TripletMarginLoss`	Watch out forMost random triplets already satisfy the margin and give zero loss, so mine hard or semi-hard negatives.	Learn and practiseDot product and normsImplement Triplet Loss
Train retrieval or self-supervised embeddings with in-batch negatives, including CLIP-style image and text models	Use this lossInfoNCE (softmax contrastive loss)	Formula $L = -\log \frac{e^{s(q, k_+)/\tau}}{\sum_{j} e^{s(q, k_j)/\tau}}$	Output and PyTorchNormalised embeddings and a temperature τ.`nn.CrossEntropyLoss over the similarity matrix, with the diagonal as labels`	Watch out forIt is cross-entropy where the other items in the batch act as the wrong classes, so a larger batch gives more negatives. SigLIP replaces the softmax with an independent sigmoid on each pair.	Learn and practiseMutual informationImplement InfoNCE LossImplement Symmetric CLIP Contrastive LossImplement the SigLIP Pairwise Loss
Learn from pairwise preferences or rankings, such as a reward model or learning to rank	Use this lossPairwise logistic (Bradley-Terry) loss	Formula $L = -\log \sigma\big(s_w - s_l\big)$	Output and PyTorchOne scalar score per item.`F.logsigmoid on the score difference, or nn.MarginRankingLoss for the hinge form`	Watch out forHere s_w is the score of the preferred item and s_l the score of the other one. RankNet and BPR use the same pairwise logistic form.	Learn and practiseRLHFBradley-Terry Reward Model Loss
Language and generative models
Pretrain or fine-tune a language model on next-token prediction	Use this lossToken-level cross-entropy over the vocabulary	Formula $L = -\frac{1}{T}\sum_{t=1}^{T} \log p(x_t \mid x_{<t})$	Output and PyTorchOne logit per vocabulary entry at every position.`nn.CrossEntropyLoss(ignore_index=-100) on labels shifted by one token`	Watch out forPerplexity is the exponential of the mean token loss. In fine-tuning, mask prompt and padding tokens so that only answer tokens count.	Learn and practiseGPT pretrainingPerplexityPerplexity ComputationBuild an Assistant-Only SFT Loss Mask
Match a target distribution, as in knowledge distillation	Use this lossKL divergence, usually added to cross-entropy	Formula $L = \alpha\,\mathrm{CE} + (1 - \alpha)\,T^2\,\mathrm{KL}\big(q^{(T)} \,\\|\, p^{(T)}\big)$	Output and PyTorchStudent logits divided by the temperature T.`nn.KLDivLoss(reduction="batchmean")`	Watch out forHere q is the teacher distribution and p the student distribution, both softened by T. Pass the student log-probabilities as the input and the teacher probabilities as the target. The T² factor keeps gradient sizes steady when T changes, and the direction of the KL matters.	Learn and practiseKL divergenceImplement KL Divergence in Python (NumPy)
Train a generative adversarial network	Use this lossNon-saturating GAN loss, or the Wasserstein critic loss	Formula $L_G = -\log D\big(G(z)\big)$ $L_{\text{critic}} = \mathbb{E}[D(\tilde{x})] - \mathbb{E}[D(x)]$	Output and PyTorchOne discriminator logit, or one unbounded critic score.`nn.BCEWithLogitsLoss for the discriminator, or a custom critic loss`	Watch out forThe original minimax generator loss saturates when the discriminator rejects fakes easily, which is why the non-saturating form is the default. When real and generated data barely overlap, a Wasserstein critic kept close to 1-Lipschitz by a gradient penalty gives more useful gradients.	Learn and practiseJensen-Shannon divergenceImplement Wasserstein Critic Loss (WGAN)
Train a diffusion model	Use this lossMean squared error on the predicted noise	Formula $L = \big\\|\varepsilon - \varepsilon_\theta(x_t, t)\big\\|^2$ $x_t = \sqrt{\bar{\alpha}_t}\,x_0 + \sqrt{1 - \bar{\alpha}_t}\,\varepsilon$	Output and PyTorchA noise estimate with the same shape as the input.`nn.MSELoss`	Watch out forThe simple noise-prediction loss from the DDPM paper is the usual default. It is the variational bound with its per-step weights dropped, and each term is a squared error on Gaussian noise.	Learn and practiseMaximum likelihood estimationMean Squared Error (MSE)
Preference tuning and RL
Fine-tune a language model on preferred and rejected answers without a reward model	Use this lossDPO loss	Formula $L = -\log \sigma\big(\beta\,(h_w - h_l)\big)$ $h = \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}$	Output and PyTorchSequence log-probabilities from the policy and a frozen reference model.`A short custom loss built on F.logsigmoid`	Watch out forHere h_w and h_l are the log-ratios for the preferred and the rejected answer. A β around 0.1 is a common starting point, and it controls how far the policy may drift from the reference model.	Learn and practiseDPOImplement the DPO Loss
Optimise a policy against rewards, as in RLHF or reasoning RL	Use this lossPPO clipped surrogate, or GRPO	Formula $L = -\min\big(\rho_t A_t,\ \mathrm{clip}(\rho_t, 1 - \epsilon, 1 + \epsilon)\, A_t\big)$ $\rho_t = \frac{\pi_\theta}{\pi_{\text{old}}}$	Output and PyTorchToken log-probabilities from the current and previous policy.`A custom loss on the probability ratio`	Watch out forThe usual clip range is ε = 0.2, and a KL penalty to a reference model is usually added. GRPO keeps the same clipping and computes advantages relative to a group of sampled answers.	Learn and practisePPOGRPOCompute a Clipped GRPO Surrogate

How does the type of target decide the loss?

The type of target decides the loss because most standard losses are the negative log-likelihood of a simple noise model for that target. Choosing a loss therefore means choosing what kind of randomness you expect around the prediction. The lesson on maximum likelihood estimation works through the Gaussian case, and the lesson on probability distributions covers the others.

Target	Noise model	Loss	Output layer
A real number	Gaussian	Mean squared error	Linear
A real number with heavy tails	Laplace	Mean absolute error	Linear
A yes or no label	Bernoulli	Binary cross-entropy	One logit and a sigmoid
One class out of K	Categorical	Softmax cross-entropy	K logits and a softmax
A count	Poisson	Poisson negative log-likelihood	Log of the rate

The output layer follows from the same choice. A yes or no label needs one logit, a choice among K classes needs K logits, a count needs the log of a rate and a real number needs an unconstrained linear output. In PyTorch the sigmoid or softmax runs inside the loss, which is why BCEWithLogitsLoss and CrossEntropyLoss both expect raw logits.

Why does the regression loss change what the model predicts?

The regression loss changes the prediction because each loss is minimised by a different summary of the targets. The single number with the lowest squared error is the mean, the one with the lowest absolute error is the median and the one with the lowest pinball loss at level τ is the τ-quantile. A model with features does the same thing for every input, so MSE learns a conditional mean and MAE learns a conditional median.

In the picture below, eleven delivery times are tied to one prediction by ropes, and the loss sets how hard each rope pulls. Switch losses and drag the late order to see which predictions move.

Where each loss puts a single prediction

Late order: 58 min

Squared error predicts the mean, 28.6 minutes. The late order alone pulls it 2.9 minutes above the 25.7 minutes the other ten orders would give, and its rope is the thickest because its pull grows with its distance.

Rope thickness shows how hard each order pulls under the chosen loss, and green ropes pull the prediction upward. The small triangles on the axis mark where the other three losses would land. Squared error is the negative log-likelihood under Gaussian noise and absolute error the negative log-likelihood under Laplace noise, as the lesson on maximum likelihood estimation explains.

With squared error a point pulls in proportion to its error, so one bad label can drag the prediction a long way. Absolute error gives every point the same pull whatever its distance, and Huber caps each pull at δ, which is what robustness to outliers means in practice. Huber also keeps the quadratic shape near zero, so its gradient shrinks as the fit improves, whereas the gradient of MAE stays the same size and can make training bounce around the optimum.

Standardise the target before you choose δ, so that δ = 1 means one standard deviation of the target. Reach for a quantile loss when the business needs a promise, such as a delivery time that nine in ten orders meet or a capacity limit that is rarely exceeded.

Which loss should I use for imbalanced classes?

Start with plain cross-entropy and an imbalance-aware metric, add class weights if the rare class is still ignored, and move to focal loss only when easy negatives swamp the gradient. Plain cross-entropy often trains well on imbalanced data, and the trouble usually comes from the default 0.5 threshold and an accuracy score that hides the rare class. Moving the threshold and reporting precision and recall fixes many cases with no change to the loss.

If the rare class is still ignored, set pos_weight in BCEWithLogitsLoss to the ratio of negatives to positives, or pass class weights inversely proportional to frequency to CrossEntropyLoss. Focal loss suits dense detection, where thousands of easy background boxes would otherwise drown out a few objects. Resampling the data is the alternative on the data side.

Weighting and focal loss both push predicted probabilities away from the true frequencies, so recalibrate if later code reads the scores as probabilities, and measure the result with expected calibration error. The guide to evaluation metrics covers which score to report for an imbalanced problem.

What loss should I use for multi-label classification?

Use one sigmoid per label with binary cross-entropy, which in PyTorch is BCEWithLogitsLoss on a float target with one column per label. Each label becomes its own yes or no question, so a photo can be tagged as both a beach and a sunset. Softmax cross-entropy is wrong here because its probabilities must sum to one, which forces the labels to compete. After training, pick a threshold for each label on validation data, because rare labels usually need a lower threshold than common ones.

Which loss should I use for embeddings, retrieval or ranking?

Use InfoNCE when you can fit large batches with in-batch negatives, contrastive or triplet loss when your labels come as pairs or triples, and a pairwise logistic loss when the goal is an ordering. InfoNCE treats every other item in the batch as a wrong answer, so each step sees many negatives at once, and image and text models in the CLIP style apply it in both directions.

Contrastive loss needs pairs labelled same or different and a margin that sets how far apart different pairs should sit. Triplet loss compares an anchor with one positive and one negative, and it learns only from triplets that break the margin, so mining hard negatives matters. For ranking and reward models, the Bradley-Terry loss applies a logistic loss to the difference between the scores of a preferred and a rejected item. The lesson on dot products and norms covers the similarity measures these losses are built on.

Which losses train and fine-tune language models?

Language models are pretrained and fine-tuned with next-token cross-entropy, and preference tuning then adds a pairwise loss such as DPO or a clipped policy objective such as PPO or GRPO. In supervised fine-tuning the same cross-entropy is masked so that only answer tokens count toward the mean, and distilling into a smaller model adds a KL term toward the teacher's softened distribution.

The GPT pretraining lesson walks through shifted labels and the token loss, and the perplexity lesson connects that loss to perplexity. The RLHF, DPO, PPO and GRPO lessons each cover one stage of preference tuning, and the KL divergence lesson explains why the direction of the KL matters in distillation.

Which loss should I train with when I care about a specific metric?

Train with the smooth loss that the metric rewards, because accuracy, F1, AUC and IoU change only when a prediction crosses a threshold and give no useful gradient. Cross-entropy suits accuracy and calibrated probabilities, MSE suits RMSE, MAE or Huber suits an MAE target, Dice suits IoU and F1 on masks, a pairwise logistic loss suits AUC and ranking, and pinball loss suits a coverage target.

Report the metric on validation data and use it to choose between models, while the loss curve tells you whether training itself is healthy. The guide to machine learning evaluation metrics explains how a loss differs from a metric and which metric fits which problem.

What mistakes break a loss function in practice?

Most loss bugs pair the loss with the wrong kind of input, so the code runs while the model learns slowly or not at all.

Applying softmax before nn.CrossEntropyLoss applies it twice, because the loss already takes the log-softmax of raw logits.
A sigmoid followed by nn.BCELoss works, but it is less numerically stable than BCEWithLogitsLoss on the raw logit.
nn.KLDivLoss expects log-probabilities as its input, and its default reduction divides by the number of elements, so use reduction="batchmean" to get the true KL.
Padding tokens left in the mean dilute a language model's loss, and two losses summed on very different scales let one of them dominate.
Switching the reduction from mean to sum multiplies the size of every gradient by the batch size, which with plain SGD acts like a much higher learning rate.

Checking the loss on the very first batch catches many of these bugs before a long run. The cross-entropy lesson on BCE vs MSE shows why the pairing of output and loss matters, and the guide on why training loss is not decreasing covers that first-batch check and the wider diagnosis.

What else do people ask about choosing a loss function?

Is cross-entropy better than MSE for classification?

Cross-entropy is the better default for classification. With a sigmoid or softmax output, mean squared error gives tiny gradients on predictions that are confidently wrong, while cross-entropy keeps a strong gradient there and matches the likelihood of a class label.

What loss function should I use for regression with outliers?

Use Huber loss when outliers should have limited influence but you still want smooth gradients near zero, and use mean absolute error when you want the model to predict the median. Squared error lets a single large error pull the prediction in proportion to its size.

Can I use accuracy as a loss function?

Accuracy cannot be trained on directly, because it only changes when a prediction crosses the threshold, so its gradient is zero almost everywhere. Train with cross-entropy and report accuracy on validation data.

Should I apply softmax before cross-entropy in PyTorch?

You should not, because nn.CrossEntropyLoss expects raw logits and applies log-softmax itself. An extra softmax squashes the logits into a narrow range and slows learning.

Should I use focal loss or class weights?

Try class weights first, since they are simple and easy to tune. Move to focal loss when the imbalance is extreme and most examples are easy negatives, as in dense object detection.

How do I combine two losses in one model?

Add them as a weighted sum and choose the weights so that each term has a similar size at the start of training. Then tune the weights on validation data, because a term that is ten times larger than the other will decide most of the gradient.

Practice this

Each of these problems has you implement one of the losses from the table and check it against tests. Then browse the problem library or the ML Math lessons.

Implement Huber Loss has you write the regression loss that switches from quadratic to linear at δ.
Implement Focal Loss has you down-weight easy examples so a rare class still drives the gradient.
Label Smoothing Loss has you soften one-hot targets and see why the loss can no longer reach zero.
Implement InfoNCE Loss has you turn a batch of similarities into a cross-entropy over in-batch negatives.