How do you know a recommender works?
Introduction
Project introduction
How do you know a recommender works?
A recommender is the part of a shop, a streaming service or a news app that picks a short list of items for each person, such as the ten products on a home page. The real test is an online experiment, where some customers see the new recommender and you measure what they buy. That is slow and costly, so teams first compare models offline, on purchases that already happened.
Offline evaluation hides some purchases, asks each model for a ranked list per customer, and checks how many hidden purchases appear near the top. Measures such as Recall@K, precision, mean reciprocal rank and NDCG turn that check into numbers. Measures such as catalog coverage and popularity bias show whether a model only ever recommends the bestsellers.
Why is it easy to get wrong?
The common recipe hides a random 20% of all purchases. A model then trains on purchases made after the ones it is asked to predict, and in shopping data it often sees half of the very order it has to complete. Press Split by time on the chart, which trains on orders before November and tests on November and December. The model that won the random split falls to fourth.
A second shortcut ranks each hidden purchase against only 100 random products instead of the whole catalog. Krichene and Rendle showed in 2020 that this can change which model looks best, and you will see that happen too. In this project you write the measures yourself, then use them on a year of real orders from an online gift shop.
Which recommender is best?
Recall@10, real ordersThe largest model, factorization with 512 factors, in amber, wins. Recall@10 is the share of a customer’s hidden purchases found in the top 10.
What you will build
01
Ranking metrics
Hit rate, precision and recall at K, mean reciprocal rank, and NDCG with graded relevance.
02
Beyond accuracy
Catalog coverage and popularity bias, which show who gets recommended at all.
03
An honest test
A time-based split, and the difference between full and sampled ranking.
Start with the top k
Compute hit rate, precision and recall for one customer's recommendations.