From Metrics to an Honest Test
Introduction
Project introduction
From Metrics to an Honest Test
You have written the tools of offline evaluation, which are hit rate, precision and recall at K, mean reciprocal rank, NDCG with graded relevance, catalog coverage with popularity bias, a time-based split, and full versus sampled ranking.
In the notebook they run on a year of real orders from a UK online gift shop, about 375,000 order lines from 4,058 customers over 2,394 products. You build five simple recommenders, which are two popularity lists, an item-to-item model of products bought together, and two matrix factorization models that describe every customer and product with a short list of 64 or 512 numbers, called factors. Then you score them every way you have learned. Six cells ask you to write one small piece and check it before you move on. The rest is written for you and explained.
What you will find out
On a random split the 512 factor model wins with a Recall@10 of about 0.175, and a list of the last 30 days’ bestsellers comes last. Trained on orders before November and tested on November and December, every score drops to between 0.02 and 0.06, the 64 factor model leads, and the 512 factor model falls to fourth, just below that bestseller list.
You will also find the reason. About 99% of the purchases the random split hides were bought in the same order as a purchase it kept for training. Then you will see that ranking against 100 random products inflates every score several times and puts the plain most popular list ahead of the 512 factor model, the opposite of the full ranking.
Which recommender is best?
Recall@10, real ordersThe largest model, factorization with 512 factors, in amber, wins. Recall@10 is the share of a customer’s hidden purchases found in the top 10.
How the notebook is organized
01
Random split
Write item-to-item similarity, run a first leaderboard, and add NDCG.
02
Time split
Split by time, tune a setting on October, check whether gaps are real, then find and fix the leak.
03
Beyond accuracy
Coverage, popularity bias, full against sampled ranking, and how often five tests agree.
Test five recommenders on real orders
Run the whole evaluation in the browser, about 45 minutes.