Classification metrics and thresholds

A clinic screens a thousand people a year for a disease that about twenty of them have. The screening test gives each person a score, and the nurse has to pick a cutoff above which someone is called back for a proper examination. Set the cutoff low and almost every sick person is called back, along with hundreds of healthy people who spend an anxious week waiting. Set it high and the callbacks are mostly right, but a few sick people go home.

The cost of each mistake depends on what happens next. A callback for a second test is cheap, so the clinic can afford many false alarms. If the same score instead triggered surgery, every false alarm would be a healthy person on an operating table, and the cutoff would have to be far higher. The metrics below are the clinic’s counts, and the cutoff is the clinic’s dial.

Scale is where the clinic and the moderation model part company, because the clinic sees a thousand people and the nurse remembers the hard cases, while a moderation model sees ten million posts a day and no human sees its mistakes unless the counts are designed to show them.

A confusion matrix is a two-by-two table that sorts every example by what was true and what the model decided. A flagged violation is a true positive, a flagged allowed item is a false positive, a missed violation is a false negative, and an allowed item left alone is a true negative. Counting the four separately is the whole point, because a single accuracy number mixes them together.

In an illustrative set of 10,000 items containing 200 violations, a model might flag 160 violations and 80 allowed items while missing 40 violations. Precision is the share of flagged items that were violations, here 160 out of 240, about 66.7 percent. Recall is the share of violations the model found, here 160 out of 200, or 80 percent. The false positive rate is the share of allowed items wrongly flagged, here 80 out of 9,800, about 0.82 percent.

The counts only turn into costs once the action is attached. If flagged items go to a review team, the team receives 240 items a day and finds 160 violations, and the 80 false alarms cost reviewer minutes. If flagged items are removed automatically, 80 people lose a legitimate post with no human in the way. The same predictions have very different consequences under the two policies, so an evaluation that does not name the action has not finished.

Precision also depends on how common the thing is. Keep the same model, with 80 percent recall and a 0.82 percent false positive rate, and move it to a feed where only 20 of 10,000 items are violations. It now finds 16 violations and flags about 82 allowed items, so precision falls from 66.7 percent to about 16 percent without the model changing at all. A model evaluated on a test set with as many violations as allowed items will look far better than it behaves on rare events.

The threshold is the score above which the system treats an item as positive, and the operating point is the pair of precision and recall you get at that threshold. Lowering the threshold flags more items, which finds more violations and raises more false alarms together. Compare models at the threshold the product can afford, such as the number of items the review team can handle in a day, because a default cutoff of one half has nothing to do with that team.

Two summary numbers describe a model across all thresholds at once. ROC-AUC, the area under the receiver operating characteristic curve, is the chance that a randomly chosen positive scores higher than a randomly chosen negative, so 0.5 is coin flipping and 1.0 is perfect ordering. A precision-recall curve plots precision against recall as the threshold moves, and it is more honest on rare events because precision feels the weight of the many negatives while ROC-AUC does not. Neither summary replaces the counts at the threshold you will use.

A model can order items well and still be wrong about how sure it is. If the items it scores near 0.8 turn out to be violations only half the time, treating 0.8 as an 80 percent risk misleads any decision that weighs cost against probability. Auto-removing a post when the expected harm is high enough is exactly that kind of decision. Calibration is the property that a score of 0.8 means roughly 80 percent in practice, and a reliability plot checks it by grouping predictions by score and comparing each group’s average score with its observed rate.

Two scores grade probabilities directly. Log loss punishes confident wrong answers heavily, and the Brier score is the average squared gap between the predicted probability and the outcome. Both mix calibration with ordering, so a change in either can move them. Any correction, such as fitting a simple curve from raw scores to honest probabilities, is fitted on held-out data and tested on a later period. A change in how common violations are, or in how the training set was sampled, changes what a score means.

Every one of these numbers needs enough examples behind it. A precision of 66.7 percent measured on 240 flagged items can move by several points from one week to the next, and a group of 40 items says almost nothing, which is the subject of the error analysis topic.

Accuracy on a rare event is the confusion with the longest history. A model that flags nothing is 98 percent accurate on a feed where two percent of posts break the rules, and 99.8 percent accurate when the rate is 0.2 percent, and it has caught nothing either time. Precision and recall at the threshold you will use are the numbers to say out loud.

Models also get compared by ROC-AUC alone, when two models with the same area can behave very differently in the small high-score region a review queue uses, so compare them at the review capacity instead.

Precision moving when the thing becomes more or less common is the one people forget. A model shipped from a test set with ten percent positives to a feed with one percent will look as if it broke, when all that changed is how rare the thing is.

On moderation, fraud and spam problems, classification metrics are the most common follow-up, and they take three or four minutes of the evaluation phase in a 45-minute round. Open with the two kinds of mistake and what each costs under the action the product takes, then name precision and recall at the threshold that matches the review capacity.

Mention calibration only if the product uses the probability itself, and save the summary curves for when the interviewer asks, which is usually right after that opening.