Walkthrough, video recommendation

The interviewer opens with the prompt. “Design the home feed for a video app. When someone opens the app they should see twenty videos chosen for them, and it should feel instant.” That is all the candidate is given, and the clock starts.

A good bookshop assistant does this in two moves. When a regular customer walks in, the assistant first pulls a shelf of thirty or forty books that might suit them, based on what they bought before. Only then does the assistant look at the customer, the weather and the time of day to pick the three to put in their hands. Pulling the shelf is the candidate stage, picking the three is the ranking stage, and the assistant never considers every book in the shop because there is no time.

Two things about the shop do not carry over. The shop holds ten thousand books while the app holds 800 million videos, so the first move has to be done by a machine ahead of time. And the assistant does not learn from where she placed books on the table, while the app does, because people click what is at the top of the feed whether or not it was the best choice, and that distorts what the model learns next.

The candidate confirms the surface first. “Is this the feed on opening the app, rather than the related videos beside one that is playing or the next video in autoplay?” The interviewer confirms it is the opening feed. The candidate asks how many videos exist and how many people open the app, and the interviewer gives 800 million videos and asks the candidate to assume the rest.

The candidate sets assumptions and converts them. “I will assume 50 million people open the app each day, about ten times each, which is 500 million requests a day or roughly 5,800 a second, with peaks maybe three times that. I will take instant to mean the feed appears within 200 milliseconds, and I will treat about 100 of those as mine, with the rest for the network and the phone.” The interviewer accepts both numbers and the candidate writes them in the corner of the board.

The candidate then states the task. “For each candidate video, the model predicts the probability that this person watches it for at least 30 seconds without skipping, and the feed is the twenty with the highest predicted value, with a few rules on top such as no two videos from the same creator.” The interviewer asks why 30 seconds rather than a click, and the candidate says a click rewards misleading thumbnails while a 30 second watch is closer to the person having wanted the video. It is also observed within a minute, so labels arrive fast.

The candidate draws the two stages and justifies them with numbers. “If I scored every video for every request, that is 800 million scores times 5,800 requests a second, which is about 4.6 trillion scores a second, and no fleet does that. If a first stage can find 500 likely candidates per request, ranking is 500 times 5,800, about 2.9 million scores a second, which a few dozen machines can handle. So the first stage has to be cheap and the second can be expensive.”

The interviewer asks how the first stage can be cheap when it still has to look at 800 million videos. The candidate explains that each video is turned ahead of time into an embedding, meaning a short list of numbers that places similar videos near each other. A search structure over those numbers can then return the few hundred nearest to a person’s own list of numbers in a few milliseconds. The expensive part, computing the embeddings, runs once a night rather than once a request.

The candidate adds the learning path. What was shown and what was watched is logged, and those logs train the next version of the ranking model. The whiteboard at the end of this phase is below, and its boxes are still labels rather than choices, since the interviewer has not asked for a component yet.

Asked what a training example is, the candidate defines it precisely. “One example is one video shown to one person at one moment, with the information available at that moment, and a label that is one if they watched for 30 seconds and zero otherwise. Only shown videos can be examples, because I cannot know what someone would have done with a video they never saw.” The interviewer asks how rare the ones are, and the candidate assumes people engage with fewer than one in fifty videos they see, so the positives are about two percent and the training set must keep all of them and sample the zeros, with the probabilities corrected afterwards.

Features are organised by source. The person brings their watch history, summarised as an embedding, plus language and country. The video brings its own embedding, its length, its age in hours and its creator. The moment brings the time of day, the device and what has already been watched in this session, which matters because someone who has watched three cooking videos in a row is not the same person as their monthly average. The candidate also adds one feature that compares the person and the video directly, the similarity between the video’s embedding and the person’s recent watches.

The interviewer asks whether the label can be trusted. The candidate raises position bias. “People watch what is at the top of the feed far more than what is at position twenty, whatever the video is. If I train on raw labels, the model learns that videos the old model placed at the top are good, which is partly true and partly the position talking.” The fix the candidate offers is to give the model the position as an input during training and set it to a fixed value, such as position one, when serving. The model then learns how much of each watch was the position and removes it at prediction time.

Ignoring the bias and treating every watch as equal is the weak answer here, and naming it with the position feature as the fix is a good one. The strong answer adds that the bias also hides good videos that were never shown high, so a small share of the feed should be given to candidates the model is unsure about, which is the exploration that cold start needs as well.

The candidate gives each stage a baseline and an upgrade. “For candidates, the baseline is the most watched videos in the person’s country over the last day, which needs no model and is what new users see anyway. The upgrade is a two-tower model, meaning one small network that turns the person into a list of numbers and another that turns a video into a list of the same length, trained so the two lists are close when the person watched the video. The video side is computed nightly and searched with an approximate nearest neighbour index, which gives up a little accuracy for a lookup in a few milliseconds.”

For the ranking model the baseline is gradient boosted trees, a model built from many small decision rules that handles mixed features well and is cheap to run, trained with cross-entropy, a loss that rewards probabilities close to what happened, on the 30 second label. The upgrade is a neural ranker with several outputs, predicting the watch, a like and a skip at once, combined by weights the product team chooses. The interviewer asks why not start with the neural ranker, and the candidate says the trees give a measured number in a week and the neural model has to beat that number to earn its cost.

The interviewer asks about a video uploaded an hour ago. The candidate names the problem before offering a fix. “A new video has no watches, so the two-tower model has nothing to place it by, and the ranking model has never seen it. That gap is what people call cold start.”

The candidate proposes three things together. The video tower takes content inputs, such as the title, the thumbnail and the creator, so a new video gets a rough embedding before anyone watches it. A small share of each feed, perhaps one slot in twenty, is reserved for new videos matched on content, so they collect their first watches. And the embeddings refresh more often than nightly for videos under a day old.

New people are the other half of cold start. The baseline of popular videos by country and language handles the first session, and the candidate adds that asking a new person to pick three interests on sign-up gives the person tower something to work with from the first request. At this phase the whiteboard gains sizes on the two stages, which the funnel below shows.

Measurement comes next, and the candidate gives each stage its own metric. “For the candidate stage I measure recall at 500, meaning the share of videos people watched that appeared somewhere in the 500 candidates, because a video the first stage misses can never be ranked. For the ranking stage I measure precision at twenty, the share of the twenty shown that were watched, and normalised discounted cumulative gain, which rewards putting the watched videos near the top rather than anywhere in the twenty.” The interviewer asks what the candidate expects, and the candidate says recall at 500 around 0.6 for a first two-tower model is a reasonable assumption, labelled as such.

Offline numbers are checked against online ones. The candidate proposes an experiment in which a small share of people get the new ranker, with watch time per session as the main measure and skip rate, dislikes and the share of videos from small creators as guardrails, meaning numbers that are not allowed to get worse. The interviewer asks why guardrails, and the candidate says a ranker can raise watch time by showing the same ten viral videos to everyone, which empties the feed for creators and bores people within a month.

Serving fits the 100 milliseconds the candidate claimed. Looking up the person’s features takes about 15 milliseconds, the nearest neighbour search about 25, ranking 500 candidates in one batch about 40, and assembling the feed with its rules about 20. The interviewer asks what absorbs the peak of 17,000 requests a second. The candidate adds a cache in front of the pipeline that keeps a person’s feed for a few minutes, so someone who opens the app three times in ten minutes costs one computation, and says the cache also returns yesterday’s popular videos if the ranker is down.

The interviewer asks what the candidate watches after launch, and the answer sorts the signals by speed. “Within the hour I watch the share of feeds served from the cache and the response time at the slowest one in a hundred requests. Within the day I watch watch time per session and the skip rate against last week. Within the week I watch coverage, meaning how many distinct videos appeared in feeds at all, because a falling number means the feed is collapsing onto a few popular videos.” The interviewer asks what the candidate does when coverage falls, and the candidate raises the exploration share and checks whether the retrain has been learning from its own placements.

Freshness comes from the stream of engagement events, which arrive continuously, so the ranking model retrains daily on the last few weeks of watches. The candidate adds that a daily retrain is only safe with a check that the new model matches the old one on data kept aside for testing, followed by a short period on a small share of traffic before it takes over. The interviewer asks about a trend that appears in an afternoon, and the candidate says the nightly embeddings are too slow for that. The candidate proposes a small live signal, the video’s watch count in the last hour, as a ranking feature that the daily model can use without being retrained.

The whiteboard at the end of the round has the cache in the serving path, the stream feeding a daily retrain and a monitoring box between them. The last minutes go to a deep dive about the reserved slot, where the interviewer asks how the candidate would know the exploration is worth its cost, and the candidate proposes counting how many of the videos that became popular this month got their first thousand watches through that slot.

A mid-level candidate is expected to reach two stages with a reason, a 30 second target, a training example defined as a shown video, a baseline and an upgrade for each stage, and offline metrics for both. Doing the funnel arithmetic aloud and fitting the serving steps into the deadline puts a mid-level candidate near the top of that band.

A senior candidate is expected to raise position bias and cold start before being asked, and to connect them, since the reserved exploration slot answers both. A senior candidate also gives each stage its own metric, names guardrails for the online experiment, and explains why the tree-based ranker is the right first model even though a neural one will replace it.

A staff candidate is expected to talk about the system a year from now. That includes how the feed can collapse onto popular videos and what coverage monitoring and exploration do about it, how a daily retrain on the model’s own placements compounds the bias unless position is recorded, how creators respond to the ranker and what that does to the catalog, and how to show the product team the cost of the weights they choose between watches, likes and skips. A staff candidate spends the least time on the model architecture of the three.