06 · Evaluation and experiments

Retrieval and ranking metrics

3 min read

Classification metrics describe whether individual decisions are correct, while a ranked list also depends on which items are included and where they appear. Retrieval selects possible results from a larger collection, and ranking orders that selection. Evaluating the two stages separately helps identify whether useful items are missing entirely or simply appearing too far down the list.

Check whether relevant items reach the ranker

Recall@K is the fraction of known relevant items retrieved in the first K candidates. If a query has ten judged relevant items and retrieval returns seven of them in its candidate set, recall is 0.7. The ranker cannot promote the three missing items, however capable it is.

Precision@K measures how many of the first K results are relevant, so a display of five results containing three relevant items has precision@5 of 0.6. Both metrics depend on how relevance was judged and how much of the catalog was assessed. An item without a judgment may still be relevant, which limits what an evaluation with incomplete judgments can establish.

These illustrative relevance judgments show how useful items can be lost during retrieval or placed below the visible results during ranking.Scroll the figure horizontally to see every step.

Use position-sensitive metrics for ordering

Mean reciprocal rank averages 1/r over queries, where r is the position of the first relevant result; a query with no relevant result contributes zero under the chosen cutoff. It suits tasks where finding one good answer matters, but ignores additional relevant results after the first.

Normalized discounted cumulative gain, or NDCG, gives more credit to relevant items placed earlier and compares the result with the best ordering of the judged items. A gain converts a relevance grade into credit, while a position discount reduces that credit farther down the list. Using gain = 2^relevance − 1 and discount = log₂(position + 1), grades [3, 0, 2] give DCG@3 of 7 + 0 + 3/2 = 8.5. The ideal order [3, 2, 0] gives about 8.893, so NDCG@3 is about 0.956 under this convention.

Some implementations accept gain values directly, so reproducing this calculation requires supplying [7, 0, 3] instead of assuming that the exponential transform is applied internally. The result also depends on the number of positions evaluated and the treatment of queries without any judged relevant items. Keeping those choices consistent makes scores comparable between models.

gain(relevance) = 2^relevance − 1
DCG@K = Σ gain(relevance at position i) / log₂(i + 1)
NDCG@K = DCG@K / ideal DCG@K

Do not confuse logged clicks with unbiased relevance

Top positions receive more attention, and the previous model chose which items were exposed. Evaluating only on its clicks inherits that selection. Randomized exposure, independent judgments, or justified counterfactual estimators can help, but each requires careful data collection and assumptions.

Model comparisons need the same evaluation pool, with checks for frequent and rare queries as well as coverage, freshness, and eligibility. A stronger aggregate ranking score may still hide a problem for new items or a particular group of users. Finding those hidden failures is the purpose of error analysis, which we will examine next before moving to online experiments.