Retrieval and ranking pipelines

A librarian asked for a good novel for a long flight does not read every book in the building. She walks to two or three shelves she knows fit the request, pulls out perhaps 15 books, and then spends her attention reading the back covers of those 15 to pick three. The walk to the shelves is retrieval and the careful comparison of 15 books is ranking, and she could not afford to read 15 back covers for every shelf in the library.

The system version has the same two steps, with a cheap similarity search in place of the walk and a scoring model in place of reading the back covers. Two differences from the librarian matter for the design. The librarian knows the reader and the books, while the system knows only what people clicked. The librarian will also happily suggest a book that arrived yesterday, while a system built on clicks has never seen that book clicked and may never pull it from the shelf.

For an illustrative catalog of ten million videos, retrieval pulls about 1,000 candidates and ranking orders them down to the 20 that fit on the screen. The sizes are assumptions to be tested rather than rules. Retrieval is judged on whether the right items are somewhere in its 1,000, and ranking is judged on whether it puts the best of those 1,000 at the top, using richer information about this user, this item, and this moment.

The arithmetic says why one stage is not enough. Assume the ranking model scores a batch of 1,000 candidates in 20 milliseconds, which is a reasonable figure for a mid-sized model on a server with an accelerator, a chip built for model arithmetic. Scoring all ten million items would take 10,000 such batches, or 200 seconds, for one request. The page has about 200 milliseconds, so the heavy model can afford roughly one thousandth of the catalog, and something cheaper has to choose which thousandth.

Candidates come from several sources, such as nearest neighbors of items the user liked, channels they follow, items they viewed recently, and what is popular in their region. The sources are merged and duplicates removed before the expensive scoring. Each source gets a quota, for example 400 from similarity and 200 from each of the others, because without one the largest source crowds out the rest.

Retrieval is fast because most of its work happens before any request arrives. Each item is turned into an embedding, which is a list of a few hundred numbers chosen so that similar items end up with similar numbers. The embeddings are loaded into an index, a data structure built to find the nearest neighbors of a point quickly. For ten million items this preparation runs as a batch job, hourly or nightly, and is checked before it replaces the previous index.

When a request arrives, the serving path builds one embedding for the user or the query, asks the index for the thousand nearest items, fetches the features those items need, and runs the ranker. The index gives up a little accuracy for speed, so it is called approximate nearest neighbor search, and a lookup over ten million items finishes in a few milliseconds instead of the hundreds it would take to compare against every item. Each response records which model and which index version produced it, so a bad result next week can be traced.

A new item has no clicks and no embedding learned from clicks, so it needs another way into the candidate pool. Its text, its images, or its creator can produce an embedding before anyone has watched it, and a small quota of deliberate exposure gives it a chance to earn clicks. A new user gets the popular list or a list based on the little that is known, such as country and device, until enough history exists. Both cases need their own measurements, because an average over established users hides them completely.

A final step after ranking enforces rules the model is not trusted with, such as removing items the user already saw, allowing at most two videos from one creator, dropping anything out of stock, and applying safety policy. Some of these rules belong before retrieval instead, since there is no point scoring an item that will be removed anyway. Access rules in particular are enforced as rules, because a relevance score is never permission to show something.

Filtering after retrieval shrinks the pool unevenly. A user in a small market with strict content rules can lose 900 of the 1,000 candidates and be left with a top-20 page that holds three items. The design tracks how many eligible items survive each stage for each kind of user and refills from a safe source when the count drops, because three excellent results on a page built for 20 is a coverage failure however good the three are.

When results are poor, the first question is which stage lost the item. Retrieval is measured by recall at 1,000, meaning the fraction of items the user eventually chose that were somewhere in the thousand candidates, and ranking is measured separately by how high it placed the good candidates it was given. If recall is 95 percent at 1,000 and barely higher at 2,000, the pool is big enough and a smaller one may save money. If recall is 40 percent, a bigger ranker changes nothing, because the item it should promote never reached it.

The same split works for one bad recommendation. Tracing the request shows whether the item was eligible, whether retrieval pulled it, what score the ranker gave it, and whether a filter removed it, and that trace also shows where the request spent its time. The next topic uses that time trace to set a budget for the whole request.

Proposing one large model that scores everything is the mistake interviewers hear most. The 200 second arithmetic above is the answer, and an interviewer who hears it knows the candidate understands why the funnel exists.

The two kinds of recall also get mixed up. Retrieval recall asks whether the right items were in the candidate pool at all, while the ranking metrics from the evaluation section ask how well the pool was ordered. A team that only measures the final page cannot tell which stage to fix.

The similarity index is expected to handle new items. An embedding learned from clicks does not exist for an item with no clicks, so new items need content-based embeddings or a deliberate exposure quota, and a design that forgets this shows the same established items forever.

The last confusion is applying every filter after ranking and reading the page as healthy. A user in a small market can lose 900 of the 1,000 candidates to content rules, and a page built for 20 that holds three items is a coverage failure the average click rate never shows, so the survivors at each stage are counted for each kind of user.

In a recommendation or search question this topic takes five to eight minutes and is usually the center of the architecture discussion. Draw the funnel with three numbers on it, the catalog size, the candidate count, and the page size, and say what each stage is measured on. The candidate sources, the new-item path, and the filtering rules are each worth one sentence here and a deeper answer only if asked.