Walkthrough, content moderation

The interviewer opens with two sentences. “Design the content moderation system for a short video app. Every upload has to be checked before it goes live, and the check has to catch content that breaks our rules.” Nothing else is given, and the candidate has 45 minutes.

Airport security has the same shape. Every bag goes through a machine, the machine clears most bags on its own, a small share are pulled aside for a person to open, and the people who want to get something through study the machine and change what they pack. The machine is the model, the person opening bags is the human reviewer, the share of bags pulled aside is a threshold someone chose, and the people studying the machine are the reason the system cannot stay the same for long.

The picture stops matching at the number of bags. An airport handles a few thousand bags an hour and can afford a person for each one pulled aside, while a video app receives uploads at a rate that forces the designer to count how many a reviewer can read. That arithmetic appears in the evaluation phase and shapes the thresholds.

The candidate asks what counts as harmful before anything else. “When you say content that breaks the rules, is that one category or several, and do I need to say which rule was broken?” The interviewer says there are several categories, such as violence and spam, but for today one label meaning the upload breaks any rule is enough. The candidate then asks about the rate of uploads and how long an upload may wait, and the interviewer offers no numbers and asks the candidate to assume some.

The candidate states assumptions aloud. “I will assume 100 million uploads a day, which is about 1,200 a second, and that an upload can wait up to a few minutes before it is published. I will also assume that fewer than one upload in a thousand breaks a rule, and that there is a review team whose size I will have to estimate.” The interviewer confirms the first two and says the one in a thousand figure is roughly right.

The task statement follows. “The model receives one upload, with its picture, its sound and its caption, and returns a probability that the upload breaks a rule. A very high probability removes the upload automatically, a very low one publishes it, and the band in between goes to a reviewer.” The interviewer asks why there are three outcomes rather than two, and the candidate explains that a reviewer can only read a limited number of uploads a day, so the middle band has to be sized to the team and the two extremes handled automatically.

Three candidate openings are graded below. The weak one picks a model before the task exists, the good one states the task, and the strong one states the task and already names the constraint that will shape every later phase.

The candidate draws six boxes in three minutes and talks while drawing. “An upload arrives and is held. One component combines the picture, the sound and the caption into a single record, because a harmless picture with a harmful caption is still harmful and the model has to see both together. The risk model scores that record. A set of decision rules turns the score into remove, review or publish, and reviewers handle the middle band. Their decisions become the labels for the next version of the model.”

The interviewer asks where the upload sits while it waits. The candidate says it sits in a queue, meaning a line of waiting items that the scoring machines take from in order, and that the queue is also what protects the system when uploads spike during a live event. The interviewer accepts the queue in one sentence and does not ask for more, which is the usual treatment of infrastructure in this round.

The figure is the whiteboard at the end of this phase. Nothing in it has been chosen yet, and that is the point of a sketch, because every later phase will point at one of these boxes and fill it in.

The interviewer asks where the labelled examples come from. The candidate names three sources and ranks them by trust. “The most reliable labels are reviewer decisions, because a trained person looked at the upload. Appeals are next, because when a creator appeals a removal and wins, that overturns a label. User reports on published content are the noisiest, because many reports are mistaken, but they are the only source for harm the model missed and published.” The interviewer asks how long a label takes to arrive, and the candidate says minutes for a reviewer decision but days for an appeal, so a training set built today is missing the appeals from this week.

The interviewer then asks what a model trained on the raw data would learn. The candidate does the arithmetic aloud. “At 100 million uploads a day and one in a thousand harmful, there are about 100,000 harmful uploads and 99.9 million harmless ones. A model that says harmless to everything is 99.9 percent accurate and catches nothing, so accuracy is useless here and I will need to keep every harmful example and sample the harmless ones, or weight the harmful ones more heavily in training.” The interviewer asks what sampling does to the probabilities, and the candidate says they come out too high and must be corrected before the thresholds are set, which earns a nod.

Features come next, and the candidate avoids a long list by naming sources of signal. The upload itself gives the frames, a transcript of the sound and the caption text. The account gives its age, its past strikes and how many uploads it has made in the last hour, because a new account uploading fifty videos in an hour is a different risk from a two-year-old account uploading one. The network gives how many other accounts have posted the same clip, which catches coordinated spam that no single upload reveals.

A weak answer lists twenty features from the caption alone. A good answer covers the upload and the account. A strong answer adds the network signal and says which features an attacker can change cheaply, such as the caption, and which they cannot, such as account age, because the hard-to-fake signals are the ones that keep working after attackers adapt.

The candidate starts with a baseline. “The baseline is a list of known bad clips, matched by a fingerprint of the video, meaning a short code computed from its frames, plus a list of banned words in the caption. It catches re-uploads of content already removed, which is a large share of the problem, and it costs almost nothing per upload.” The interviewer asks what the baseline misses, and the candidate says anything new, anything with the harm in the sound, and anything an attacker has edited enough to change the fingerprint.

For the model itself the candidate reuses existing pieces. “I would not train a vision model from scratch. I would take a pretrained image encoder, meaning a model someone already trained on millions of pictures to turn an image into a list of numbers, and run it on a few frames. I would do the same with a pretrained text encoder on the caption and the transcript, and feed those lists of numbers into one small classifier that outputs the probability.” The interviewer asks why one classifier rather than three, and the candidate returns to the harmless picture with the harmful caption, which three separate models would each score as safe.

The interviewer asks about the loss, meaning the number the training process tries to make small. The candidate names cross-entropy, which measures how far the predicted probability is from the true label, with the harmful examples weighted more heavily so that the rare class is not ignored. The candidate adds that heavy weighting pushes probabilities up, so calibration, meaning checking that uploads scored 0.3 break a rule about 30 percent of the time, will matter before the thresholds are set.

The whiteboard changes at this phase only inside the risk model box, which now contains two encoders feeding one classifier. The figure shows that change on its own, because redrawing the whole pipeline would spend minutes on arrows that have not moved.

The interviewer asks how the candidate would know the model is good. The candidate separates two numbers. “Recall is the share of harmful uploads the model catches, and precision is the share of what it flags that is harmful. I would report both at the thresholds I would use rather than one summary number, because the business cares about how much harm gets through and how many creators are wrongly removed, and those are recall and precision in plain words.” The interviewer asks which matters more, and the candidate says it depends on the category, with violence favouring recall and spam favouring precision, and that one label hides that difference.

The thresholds are set by the review team, and the candidate works the numbers out aloud. “Suppose the model flags two percent of uploads for review, which is 2 million a day. If a reviewer reads about 100 uploads an hour over an eight-hour shift, that is 800 a day, so 2 million needs 2,500 reviewers. If the team has 250 people, I can send them 200,000 a day, which is 0.2 percent of uploads, and the review band has to be set that narrow.” The funnel below shows those three numbers.

The interviewer asks what happens to the other 1.8 million the model was unsure about. The candidate explains the three bands. Uploads above the high threshold are removed automatically and the creator can appeal. Uploads in the narrow review band are held for a person. Everything below the review band is published, and a share of it is harmful, which is the miss rate the business accepted when it chose the team size. Saying that trade-off aloud is what the interviewer is listening for.

Serving is short because the deadline is generous. At 1,200 uploads a second with a few minutes of allowed wait, the scoring machines can take uploads from the queue in batches of a few hundred, which is far cheaper per upload than scoring one at a time. The candidate adds that the fingerprint baseline runs first, so a known bad clip never reaches the encoders, and that if the model is down the upload stays held in the queue rather than being published unchecked.

The interviewer asks how the candidate would notice the model getting worse. The candidate names signals that arrive at different speeds. “Within an hour I can see the flag rate, meaning the share of uploads the model sends to each band, and a sudden drop means attackers have found a gap. Within a day I can see how often reviewers agree with the model on the uploads they read. Within a week I can see how many removals were overturned on appeal and how many published uploads were reported by users.” The interviewer asks which one would make the candidate retrain, and the candidate says a falling reviewer agreement rate, because it means the model’s middle band has stopped matching what people judge harmful.

Retraining is frequent by design, and the candidate proposes a weekly retrain on the latest reviewer decisions. Before release, the new model must do at least as well as the old one on last week’s labels kept aside for testing, and after release it runs on a small share of traffic before it takes over. The interviewer asks about content the model has never seen, such as a new kind of harm appearing after a news event. The candidate says the user report stream is the only early signal, so a cluster of reports on published uploads with similar fingerprints should open a review task even before any label exists.

The whiteboard gains a monitoring box and a retrain path, shown below. The interviewer uses the remaining minutes on a deep dive about attackers, asking what happens when they learn the caption filter. The candidate answers that caption signals decay first, that account age and network signals are harder to fake, and that the weekly retrain is the mechanism that keeps up. The candidate adds a warning that a retrain on labels from a model that is already missing things will learn to miss them too, so the user reports have to be in every training set.

A mid-level candidate is expected to reach the end of the six phases with a working design. That means a stated task, a sketch, labels from reviewer decisions, a sensible model that reuses pretrained encoders, precision and recall at the chosen thresholds, and a weekly retrain. Doing the imbalance arithmetic without being asked and naming the three outcome bands puts a mid-level candidate near the top of that band.

A senior candidate is expected to find the constraints before the interviewer points at them. The reviewer capacity arithmetic, the observation that sampling distorts probabilities and calibration must be checked, and the ranking of label sources by trust are senior moves, because each one changes a decision in a later phase. A senior candidate also says which signals attackers can fake cheaply and chooses features accordingly.

A staff candidate is expected to reason about the system changing itself over time and about the people around it. That includes the warning that retraining on the current model’s own decisions teaches it to repeat its misses, and the case for keeping user reports in every training set. It also includes the idea that different categories want different thresholds, and a view on how the review team and the model should grow together over a year. A staff candidate spends less time on the model than a mid-level candidate does and more on the loops around it.