Machine Learning Evaluation Metrics: Which Metric for Which Problem

Name: TensorTonic
Author: TensorTonic

By TensorTonic's founderLast updated November 1, 20269 min read

Which machine learning evaluation metric should you use?

Choose the metric that matches the decision the model supports. Use accuracy only when classes are balanced, PR-AUC with recall at a fixed precision when positives are rare, log loss and calibration error when probabilities drive decisions, MAE or RMSE for regression, NDCG or recall@K for ranking, and perplexity for language models.

Which evaluation metric to use, by situation
Situation	Report this	Watch out for	Learn and practise
Classification
Binary labels, classes roughly balanced, and both errors cost about the same.	Accuracy and F1Also report ROC-AUC to compare models before a threshold is set.	Accuracy stops meaning much as soon as one class dominates the data.	Accuracy, precision, recall, F1 Type I and II errors
Rare positives that you need to find, such as fraud, defects or disease screening.	PR-AUC (average precision), then recall at the precision you can tolerateAlso report the positive rate of the test set and the confusion matrix at the chosen threshold.	Accuracy rewards predicting the majority class. ROC-AUC barely moves as false alarms grow, because the huge negative count dilutes the false positive rate. A random model scores PR-AUC equal to the positive rate.	Precision-recall curve and AP Bayes theorem
False positives are the costly error, such as auto-blocking or auto-removing content.	Precision at a minimum recall, or F0.5	Very high precision with very low recall means the model acts on almost nothing.	Accuracy, precision, recall, F1
False negatives are the costly error, such as screening or safety alerts.	Recall at a minimum precision, or F2Also report the false positive rate, so the alert volume stays visible.	A recall of 1.0 is easy to reach by flagging every case.	Accuracy, precision, recall, F1 Type I and II errors
You need one number for a skewed binary problem that uses all four confusion-matrix cells.	Matthews correlation coefficient (MCC)Always state the threshold it was measured at.	MCC runs from -1 to 1 with 0 for random guessing, and it still changes with the threshold.	Confusion matrix with normalization
You want ranking quality before any threshold is chosen.	ROC-AUCAdd PR-AUC when positives are rare.	ROC-AUC is the chance that a random positive scores above a random negative. It says nothing about any single threshold or about calibration.	Integration and AUC Compute AUC Compute the ROC curve
The probability itself drives a decision, such as pricing, expected cost or a risk score.	Log loss and Brier scoreAlso report expected calibration error (ECE) and a reliability plot.	A model can rank well and still be badly calibrated. Recalibrate on held-out data and never on the test set.	Binary log loss Expected calibration error Isotonic calibration Cross-entropy
Multiclass labels with classes of roughly equal size.	Accuracy, or top-k accuracy when the label space is largeAlso report the recall for each class from the confusion matrix.	One weak class can hide inside a good overall accuracy.	Top-k accuracy Confusion matrix with normalization
Multiclass labels with imbalanced classes, where every class matters.	Macro-F1 or balanced accuracyAlso report precision and recall for each class.	Micro-F1 equals accuracy for single-label multiclass data, and weighted F1 leans toward the large classes.	Micro-F1
Multi-label data, where one example can carry several labels.	Micro-F1 when total label volume matters, macro-F1 when rare labels matter	Exact-match accuracy falls quickly as the number of labels grows.	Micro-F1
Agreement with human raters, or ordinal labels such as star ratings.	Cohen's kappa, or quadratic weighted kappa for ordinal labels	Kappa shifts with the class balance, so compare it only across similar label distributions.	Cohen's kappa
Regression and forecasting
You want the typical error in the target’s own units, and outliers should not dominate.	MAE	MAE is minimised by predicting the median, so it will not pull predictions toward rare large values.	Regression metrics
Large misses are much worse than small ones.	RMSEReport MAE beside it.	RMSE is always at least MAE, and a wide gap between the two means a few large errors.	Mean squared error Regression metrics
You need the share of variance explained, independent of the target’s units.	R²	R² can be negative on held-out data, which means the model does worse than predicting the mean.	R² score
Forecasts across series with very different scales.	MASE or sMAPE, or MAPE when actual values never approach zeroScore it on a time-based split.	MAPE is undefined at zero and caps the error of an under-forecast at 100%, so optimising it biases forecasts low.	Data splits and leakage
Prediction intervals or quantile forecasts.	Pinball loss for each quantileAlso report interval coverage against its nominal level.	A narrow interval looks good until you check how often the truth falls inside it.	Percentiles and quantiles
Ranking, retrieval and recommendation
Retrieval or candidate generation, where relevant items must reach the top K.	Recall@K, or hit rate@K with one held-out item per user	Neither metric cares about the order inside the top K.	Precision and recall at K Hit rate at K Ranking metrics
A ranked list with graded relevance, such as search results or a feed.	NDCG@K	Libraries differ on the gain, using either the grade or two to the power of the grade minus one, so fix one before comparing.	NDCG Ranking metrics
One correct answer, and its position matters.	MRR (mean reciprocal rank)	Only the first relevant result counts, so later relevant results add nothing.	Ranking metrics
Several relevant items with yes or no relevance.	MAP@K (mean average precision)	Ranking MAP@K is a different calculation from detection mAP.	Precision and recall at K
A recommender judged on more than accuracy.	Catalog coverage beside the ranking metrics	A model can score well by recommending the same popular items to everyone.	Catalog coverage
Vision
Object detection.	mAP at an IoU threshold, such as 0.5, or averaged over 0.5 to 0.95	A box counts as correct only above the IoU threshold, so always state which threshold you used.	Intersection over union
Semantic segmentation.	Mean IoU over classes, or Dice for small objects and medical masks	Dice equals 2 IoU / (1 + IoU), so for a single mask the two rank predictions the same way.	Mean IoU Dice loss
Language and generative models
Language model quality on held-out text.	Perplexity	Perplexity is comparable only between models with the same tokenizer and the same evaluation text.	Perplexity GPT pretraining Perplexity computation
Translation or summarisation scored against reference texts.	BLEU for translation, ROUGE for summarisation	Both reward shared words, so a correct paraphrase can score poorly, and BLEU is designed as a corpus-level score.	BLEU score BLEU and ROUGE
Multiple-choice benchmarks for a language model.	Accuracy, picking the option with the highest log-likelihood	Longer options collect lower total likelihood, so many setups normalise by length.	Score multiple-choice continuations
Open-ended answers compared head to head.	Pairwise win rate, aggregated into Elo or Bradley-Terry ratings	Human and model judges tend to favour longer answers, so control for length.	Elo from pairwise votes Judge length debiasing RLHF
Code generation checked by unit tests.	pass@k with the unbiased estimator	Draw n samples per problem, with n at least k and usually much larger, and apply the estimator. Drawing exactly k samples gives a noisy estimate.	No matching problem yet.
Clustering
Clusters with no ground-truth labels.	Silhouette score	It runs from -1 to 1 and favours compact, well-separated clusters, so it can undervalue long or nested shapes.	Silhouette score
Clusters compared with known labels.	Adjusted Rand index or normalised mutual information	The unadjusted Rand index is high even for random clusterings.	Mutual information
How you measure, for every row above
A small dataset where one split gives a noisy number.	k-fold cross-validation, reported as mean and standard deviationUse stratified folds for imbalance, grouped folds when one person has many rows and time-based splits for forecasts.	Random folds on time-ordered or grouped data leak information and inflate every metric.	K-fold split Stratified split Resampling
Deciding whether model B really beats model A.	A bootstrap confidence interval on the paired difference over the same test set	When the interval includes zero, the gap cannot be told apart from test-set noise.	Bootstrap confidence interval Confidence intervals
Measuring impact after launch.	An A/B test on the product metric, with guardrail metrics	An offline gain does not guarantee an online one, so the live test has the final say.	A/B testing A/B testing and safe rollout

Every row above is about what you report. The model is usually trained on a different quantity, and the next sections explain the choices that cause the most confusion. The formulas for every metric in the table are collected near the end of the page.

Which metric should you use for imbalanced classification?

For imbalanced classification, compare models with precision-recall AUC (average precision), then report recall at the precision your workflow can tolerate, or the F-beta score at your chosen threshold. Accuracy misleads because always predicting the majority class scores high, and ROC-AUC can look strong when negatives vastly outnumber positives.

The card below keeps one model fixed and changes only how common the positive class is. Pick a lower positive rate and watch which beads slide.

Same model, rarer positives

At its fixed threshold, this illustrative model catches 80% of positives and wrongly flags 5% of negatives. Only the share of positive cases changes.

Positive rate

Out of 10,000 cases the model flags 575, of which 80 are real positives and 495 are false alarms.

Moves with the positive rate

Accuracy0.95

A model that always predicts negative scores 0.99 here.

Precision0.14

F10.24

MCC0.32

PR-AUC0.47

Stays put

Recall0.80

ROC-AUC0.96

Balanced accuracy0.88

Value at the chosen positive rateValue at the other three rates

Metric values for one fixed model at four positive rates
Metric	50% positive	10% positive	1% positive	0.1% positive
Accuracy	0.88	0.94	0.95	0.95
Precision	0.94	0.64	0.14	0.02
F1	0.86	0.71	0.24	0.03
MCC	0.76	0.68	0.32	0.11
PR-AUC	0.96	0.81	0.47	0.18
Recall	0.80	0.80	0.80	0.80
ROC-AUC	0.96	0.96	0.96	0.96
Balanced accuracy	0.88	0.88	0.88	0.88
Accuracy of always predicting negative	0.500	0.900	0.990	0.999

When positives are rare, compare models with PR-AUC and then choose the threshold by a recall target or an alert budget. Practise both with Compute AUC and Compute Accuracy, Precision, Recall, F1.

Recall, ROC-AUC and balanced accuracy are computed within each class, so they describe the model alone and do not move. Precision mixes the two classes, so it falls as the positive rate falls. Precision is the probability that a flagged case is positive, which is Bayes' theorem with the positive rate as the prior. F1, MCC and PR-AUC all depend on precision, so they fall with it.

A random model scores a PR-AUC equal to the positive rate, so a PR-AUC of 0.47 at a 1% positive rate is far above chance even though it looks low. Always report the positive rate of the test set beside PR-AUC, because the same model scores differently on data with a different mix. When you need one number that stays honest on skewed data, the Matthews correlation coefficient uses all four cells of the confusion matrix and scores 0, by the usual convention, for a model that always predicts the majority class.

In interviews, the reason accuracy fails tends to matter more than the metric you name, which the guide to common ML interview mistakes covers.

Should you use ROC-AUC or PR-AUC?

Use PR-AUC when positives are rare and false alarms carry a cost, and use ROC-AUC when you need a ranking measure that does not change with the positive rate, such as comparing one model across datasets with different class mixes. ROC-AUC equals the chance that a random positive scores above a random negative, which the lesson on integration and AUC derives from the area under the curve.

ROC-AUC remains a valid ranking measure on imbalanced data. Because its false positive rate divides by the number of negatives, thousands of extra false alarms barely move it when negatives dominate. PR-AUC reacts to those false alarms directly. On rare-event problems, report both.

What is the difference between a loss function and an evaluation metric?

A loss function is the quantity gradient descent minimises during training, so it must be smooth enough to differentiate, while an evaluation metric is the number that reflects the decision the model supports. A fraud model is usually trained on cross-entropy and judged on recall at a set precision.

Some quantities appear in both roles. Log loss as a metric is the same quantity as binary cross-entropy as a loss, MSE is a common regression loss whose square root is reported as RMSE, and Dice appears as both a segmentation loss and an overlap score. The cross-entropy lesson explains why classification is trained on cross-entropy, and the guide on which loss function to use covers the training side of the choice.

How do you choose a classification threshold?

Choose the threshold on validation data from the cost of each error or from a capacity limit, such as how many alerts a team can review each day. A threshold of 0.5 is right only when the probabilities are calibrated and both errors cost the same.

With calibrated probabilities, the threshold that minimises expected cost is $C_{FP} / (C_{FP} + C_{FN})$ , where $C_{FP}$ and $C_{FN}$ are the costs of a false positive and a false negative. When the costs are hard to price, fix one side and read off the other. Set a recall target and report the precision that comes with it, or set an alert budget and report how many positives fit inside it. Never tune the threshold on the test set, because the reported score then overstates what the model will do. The classification metrics and thresholds topic walks through the trade-off on a full example.

When do you need log loss, Brier score or calibration error?

You need probability metrics when the probability itself drives a decision, such as a price, an expected cost or a risk score that a person reads. Ranking metrics such as ROC-AUC ignore calibration completely, so a model can rank cases well while saying 90% for events that happen 60% of the time.

Log loss punishes confident wrong answers heavily, and the Brier score is the mean squared error of the probabilities. Expected calibration error groups predictions into confidence bins and compares each bin's confidence with its accuracy. If calibration is poor, fit Platt scaling or isotonic regression on held-out data. The Expected Calibration Error and Isotonic Regression Calibration problems have you build both.

Should you use MAE or RMSE for regression?

Use MAE when you want the typical error in the target's own units and outliers should not dominate, and use RMSE when large misses are much worse than small ones. RMSE is never smaller than MAE, so a wide gap between the two tells you a few predictions are far off.

The two metrics favour different predictions. MAE is minimised by the median of the target and MSE by the mean, so a model tuned for one is not tuned for the other. R² compares the model with always predicting the mean, which makes it independent of the target's units, but it can be negative on held-out data. Avoid MAPE when actual values can be zero or close to it, and use time-based splits for any forecast.

Which metrics are used for ranking and recommendation?

Ranking systems use recall@K or hit rate@K for the retrieval stage, NDCG@K when relevance is graded and position matters, MRR when only the first relevant result counts, and MAP@K when several items are relevant. Recommenders also track catalog coverage, because a model can score well by showing everyone the same popular items.

Choose K to match what users actually see, such as the number of results above the fold. The retrieval and ranking metrics topic works through each metric on a single ranked list, and the NDCG problem has you implement the most common one.

How do you evaluate language models and generated text?

Language models are evaluated with perplexity on held-out text, accuracy on multiple-choice benchmarks, overlap scores such as BLEU and ROUGE against references, pairwise win rates for open-ended answers, and pass@k for code. Each one measures something different, so a report usually needs more than one.

Perplexity is the exponential of the mean negative log-likelihood per token, which the perplexity lesson works through. It is comparable only between models that share a tokenizer and an evaluation set. BLEU and ROUGE reward word overlap, so they can score a correct paraphrase poorly. Pairwise judges, whether human or model, tend to prefer longer answers, which is why the judge length debiasing problem exists, and preference data of this kind is also what RLHF trains on.

How do you know a metric difference is real?

A metric difference is real when it holds up beyond the noise of the test set, which you check with a bootstrap confidence interval on the paired difference between two models scored on the same examples. If the interval includes zero, the gap is not established.

On small datasets, report k-fold cross-validation as a mean with its standard deviation, using stratified folds for imbalanced labels and grouped folds when one person contributes many rows. Fix the metric before you look at results, and keep the test set for one final check. If the validation metric stalls while the training loss keeps falling, the guide to broken training runs lists what to check. The resampling and confidence interval lessons cover the statistics, and the A/B testing lesson covers the online test that settles whether a gain survives launch.

What are the formulas for the main evaluation metrics?

The formulas below use TP, FP, TN and FN for the cells of the confusion matrix, $p_i$ for a predicted probability, $y_i$ for a label and $\hat{y}_i$ for a numeric prediction. Each group links to the lesson that derives it.

Threshold metrics from the confusion matrix

Type I and II errors lesson

Accuracy: $\displaystyle \frac{TP + TN}{TP + TN + FP + FN}$
The share of all predictions that are correct.
Precision: $\displaystyle \frac{TP}{TP + FP}$
The share of flagged cases that are truly positive.
Recall (TPR): $\displaystyle \frac{TP}{TP + FN}$
The share of real positives the model catches.
False positive rate: $\displaystyle \frac{FP}{FP + TN} = 1 - \text{specificity}$
The share of negatives wrongly flagged.
F1: $\displaystyle F_1 = \frac{2PR}{P + R} = \frac{2\,TP}{2\,TP + FP + FN}$
The harmonic mean of precision P and recall R, which ignores true negatives.
F-beta: $\displaystyle F_\beta = \frac{(1 + \beta^2)\,P R}{\beta^2 P + R}$
A beta above 1 weights recall more, and a beta below 1 weights precision more.
Balanced accuracy: $\displaystyle \frac{1}{K}\sum_{k=1}^{K} \text{Recall}_k$
The mean recall over the K classes, which for two classes is the mean of TPR and TNR.
MCC: $\displaystyle \frac{TP \cdot TN - FP \cdot FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}$
A correlation between labels and predictions from -1 to 1, with 0 for random guessing.
Cohen's kappa: $\displaystyle \kappa = \frac{p_o - p_e}{1 - p_e}$
Observed agreement p_o corrected by the agreement p_e expected by chance.

Ranking and probability metrics

Integration and AUC lesson

ROC-AUC: $\displaystyle P(s^+ > s^-) + \tfrac{1}{2}P(s^+ = s^-)$
The chance that a random positive outscores a random negative, which equals the area under the ROC curve.
Average precision: $\displaystyle \text{AP} = \sum_n (R_n - R_{n-1})\,P_n$
Precision at each threshold weighted by the gain in recall, the usual estimate of PR-AUC.
Log loss: $\displaystyle -\frac{1}{N}\sum_{i=1}^{N}\big[y_i \log p_i + (1 - y_i)\log(1 - p_i)\big]$
Binary cross-entropy, which punishes confident wrong probabilities heavily.
Brier score: $\displaystyle \frac{1}{N}\sum_{i=1}^{N}(p_i - y_i)^2$
The mean squared error of the predicted probabilities.
ECE: $\displaystyle \sum_{b=1}^{B}\frac{|B_b|}{N}\,\big|\text{acc}(B_b) - \text{conf}(B_b)\big|$
The gap between confidence and accuracy, averaged over B probability bins.

Regression metrics

MAE: $\displaystyle \frac{1}{N}\sum_{i=1}^{N}|y_i - \hat{y}_i|$
The average absolute error, in the units of the target.
RMSE: $\displaystyle \sqrt{\frac{1}{N}\sum_{i=1}^{N}(y_i - \hat{y}_i)^2}$
The square root of MSE, which weights large errors more and is never below MAE.
R²: $\displaystyle R^2 = 1 - \frac{\sum_i (y_i - \hat{y}_i)^2}{\sum_i (y_i - \bar{y})^2}$
The share of variance explained relative to predicting the mean, which can be negative on test data.
MAPE: $\displaystyle \frac{100\%}{N}\sum_{i=1}^{N}\left|\frac{y_i - \hat{y}_i}{y_i}\right|$
The mean percentage error, which is undefined when an actual value is zero.
Pinball loss: $\displaystyle L_\tau(y, \hat{y}) = \max\big(\tau(y - \hat{y}),\ (\tau - 1)(y - \hat{y})\big)$
The loss for a forecast of quantile tau, averaged over examples.

Ranking, vision and clustering metrics

Recall@K: $\displaystyle \frac{|\text{relevant} \cap \text{top-}K|}{|\text{relevant}|}$
The share of relevant items that appear in the top K.
MRR: $\displaystyle \frac{1}{|Q|}\sum_{q=1}^{|Q|}\frac{1}{\text{rank}_q}$
The mean of one over the rank of the first relevant result, counting 0 when none appears.
NDCG@K: $\displaystyle \frac{\text{DCG@}K}{\text{IDCG@}K},\quad \text{DCG@}K = \sum_{i=1}^{K}\frac{2^{rel_i} - 1}{\log_2(i + 1)}$
Discounted gain divided by the gain of the ideal order, from 0 to 1.
IoU: $\displaystyle \frac{|A \cap B|}{|A \cup B|}$
The overlap between a predicted region A and the true region B.
Dice: $\displaystyle \frac{2|A \cap B|}{|A| + |B|} = \frac{2\,\text{IoU}}{1 + \text{IoU}}$
An overlap score that equals pixel-level F1 on binary masks.
Silhouette: $\displaystyle s(i) = \frac{b(i) - a(i)}{\max\big(a(i), b(i)\big)}$
Here a is the mean distance to the point’s own cluster and b is the mean distance to the nearest other cluster.

Language model metrics

Perplexity lesson

Perplexity: $\displaystyle \exp\Big(-\frac{1}{N}\sum_{t=1}^{N}\log p(x_t \mid x_{<t})\Big)$
The exponential of the mean negative log-likelihood per token.
BLEU: $\displaystyle \text{BP}\cdot\exp\Big(\sum_{n=1}^{4}\tfrac{1}{4}\log p_n\Big),\quad \text{BP} = \min\big(1,\ e^{1 - r/c}\big)$
Clipped n-gram precision p_n with a brevity penalty, where c is the candidate length and r the reference length.
pass@k: $\displaystyle \mathbb{E}\left[1 - \frac{\binom{n - c}{k}}{\binom{n}{k}}\right]$
The chance that at least one of k samples passes, estimated from n samples of which c pass.

What else do people ask about evaluation metrics?

Is accuracy a good metric for imbalanced data?

Accuracy is a poor metric for imbalanced data, because a model that always predicts the majority class scores the majority share. At a 1% positive rate that model reaches 99% accuracy while finding no positives at all.

Is F1 score good for imbalanced data?

F1 is better than accuracy on imbalanced data because it ignores true negatives. It still weighs precision and recall equally and depends on the threshold, so use F-beta when one error costs more, or report precision at a fixed recall.

What is a good ROC-AUC score?

A ROC-AUC of 0.5 means random ranking and 1.0 means perfect ranking, and what counts as good depends on the task and the current baseline. A high ROC-AUC can sit beside very low precision when positives are rare.

What is the difference between micro, macro and weighted averaging?

Micro averaging pools the counts from every class, so large classes dominate it. Macro averaging takes the plain mean of the per-class scores, so rare classes count fully. Weighted averaging weights each per-class score by how many examples the class has.

What is a good F1 score?

No single F1 value is good everywhere. Compare it with a baseline that predicts positive for every case, which scores 2p / (1 + p) at a positive rate p, and with the model you already have.

Practice this

Implementing a metric is the quickest way to learn where it breaks. Each of these problems has you build one from scratch, and the ML Evaluation Metrics Sprint mock assessment tests several of them against a clock.

Compute Accuracy, Precision, Recall, F1 has you build the threshold metrics from the four cells of the confusion matrix.
Compute AUC (Area Under ROC) has you turn scores into a ranking metric that needs no threshold.
Expected Calibration Error has you check whether predicted probabilities match observed rates.
NDCG (Normalized Discounted Cumulative Gain) has you score a ranked list with graded relevance.