02 · Framing the problem
From business goal to ML task
10 min read
A business goal is the change a company wants to see, such as more guests coming back. A model cannot learn that directly, so the designer has to translate it into an ML task, which is one specific question with one kind of answer that past records can teach. This translation decides what the model predicts, what counts as a correct answer, and how the product acts on the prediction, and a mistake here cannot be fixed by a better model.
An everyday example
The owner of a small restaurant wants more guests to come back. That is the business goal, and it is a good one, but nobody on the staff can act on it directly during a Friday service. The host can act on which tables to confirm by phone, the kitchen can act on how many portions to prepare, and the waiter can act on which dish to suggest. Each of those is a decision, and each decision could use a prediction.
Translating the goal means choosing one of those decisions and stating the question the prediction answers. That question might be whether this booking will arrive, how many portions of the special will sell tonight, or which three dishes the waiter should mention to this table. The restaurant then checks the business goal by watching return visits over months, which is slow, while it checks each prediction against what happened that same evening.
Translate the goal into a target the model can learn
The target is the outcome you ask the model to predict, and the label is the recorded value of that outcome for one past example. If the restaurant chooses the no-show decision, the target is whether a booking arrives, and the label for each past booking is a yes or a no written down after the evening. A training example is then the information known when the booking was made, such as party size, the day of the week and how far ahead it was booked, together with that label.
The model learns to estimate the probability of the label, and its training loss is the number that measures how far its estimates sit from the labels during training. That gives three different things three different names. The business goal is return visits, the ML task is predicting no-shows, and the loss is the training score. Improving the loss does not guarantee the business goal, which is why the sections below measure them separately.
A target that is easy to count is usually standing in for the outcome you care about, and the gap matters. A video feed that records clicks can train on them, yet a click says only that a title was tempting. Watching for at least a defined length of time is closer to enjoyment, though a fixed length favors long videos, so even the better target still has to be checked against the experience the feed was meant to improve.
Five kinds of task a model can do
Once the decision is chosen, the question it asks falls into one of a small number of task types, and naming the type tells everyone what shape the answer has and how it will be scored. Classification answers a yes-or-no or which-of-these question with a category, such as whether a booking will arrive. Regression answers a how-much question with a number, such as how many portions of the special will sell.
Ranking answers a which-order question by sorting a set of items so the best ones sit at the top, such as the three dishes most likely to please this table out of thirty on the menu. Retrieval answers a find-me-things-like-this question by pulling a few hundred items out of a very large collection quickly. For a restaurant chain, that might mean finding dishes similar to the one a guest praised last time out of every dish the chain has ever served. Generation answers a write-or-draw-this question by producing new text, images or sound, such as a description for a new dish on the menu.
Retrieval and ranking usually work together, with retrieval cutting a huge collection down to a few hundred candidates cheaply and ranking ordering those few hundred carefully. The restaurant’s menu of thirty dishes needs no retrieval step, and that is where the restaurant example stops matching large systems. A video service with ten million videos cannot score them all for every viewer, so it retrieves a few hundred first and ranks only those.
| Task type | Question it answers | Restaurant example | Shape of the answer |
|---|---|---|---|
| Classification | Which category is this | Will this booking arrive | A category or a probability |
| Regression | How much or how many | Portions of the special tonight | A number |
| Ranking | In what order | Three dishes for this table | A sorted list |
| Retrieval | Which few out of very many | Dishes like the one they praised | A small set from a huge one |
| Generation | Write or draw this | A description for a new dish | New text, image or sound |
A prediction needs a decision rule
A probability on its own does nothing. The host still has to decide whom to phone, and the rule that turns a probability into an action is called the decision policy. The policy needs more than the score, because the two possible mistakes cost different amounts. Phoning a guest who was coming anyway costs a little goodwill, while an unconfirmed table that stays empty costs the whole evening’s takings for that table.
A small cost calculation shows how the policy follows from those costs. Assume an empty table costs 100 units, an unnecessary confirmation call costs 5 units, and a correct decision costs nothing. For a no-show probability p, leaving the booking alone has an expected cost of 100 times p, and calling has an expected cost of 5 times one minus p. Calling is cheaper once p is above 5 divided by 105, which is about 4.76 percent, so the host should phone any booking the model puts above roughly one chance in twenty.
This arithmetic relies on calibrated probabilities, which means that bookings the model scores at 10 percent fail to arrive about 10 percent of the time. An uncalibrated model can still rank bookings well while its numbers mean nothing as probabilities, and then the threshold of 4.76 percent is a threshold on nonsense. The costs are also simplified, since a complete policy would weigh the chance that a call changes the guest’s mind and the limited time the host has for calls.
Leave alone expected cost = 100 × p
Call to confirm expected cost = 5 × (1 − p)
Call when 100p > 5 × (1 − p)
105p > 5
p > 5 / 105 ≈ 4.76%Measure the business effect separately
The model’s score on past bookings says how good its guesses are, and only a measurement on actual evenings says whether the restaurant gained anything. Choose one primary outcome and a few guardrails before the policy goes live. For the restaurant, the primary outcome is empty tables per evening, and the guardrails are complaints about confirmation calls and the number of regulars who stop booking.
A guardrail is a measurement that is allowed to stay flat and must not get worse. When several outcomes matter, the policy has to say how they trade, because adding click, watch and dislike scores with made-up weights produces behavior nobody intended. Begin with one primary outcome and firm limits on the guardrails, and test the trade-offs in a controlled comparison where some evenings, or some users, get the new policy and the rest do not.
The business goal itself sits furthest away in time. Return visits take months to count and are moved by the food and the weather as well as by the host’s calls. The restaurant therefore uses empty tables per evening as its faster signal and checks every quarter that fewer empty tables did go with more returning guests. With the input, target and decision defined, the next question is how much work the system must do and how fast, which the next topic estimates.
Where people get confused
Offering the loss function as the business goal is the confusion interviewers hear most. A candidate who says the goal is to minimize the training loss has named the model’s score, and the interviewer wanted to hear about return visits, the no-show decision and the label that links them.
Close behind it is picking the easy-to-count target without checking what it stands in for. Clicks, opens and calls made are all activity, and a model that is excellent at predicting activity can pull the product away from the outcome that mattered, as the video feed that learned to promote tempting titles shows.
The task type also gets chosen before the decision. A team that announces a classification model for dish suggestions has to work backwards to a decision that needs a yes or no, when the waiter’s decision needs an order, which makes it a ranking task. Name the decision first and let the task type follow from the shape of the answer it needs.
In the interview
Near the start of a 45-minute design interview, inside the framing phase, about five minutes go to this translation. It is one of the places where interviewers separate candidates who have shipped something from candidates who have only trained models. A plain way to bring it up is to say that you want to state the business goal, the decision, the ML task and the label before touching the model, and then to check that the four agree.
Say the task type out loud, and if the natural task is ranking, say what you would retrieve first and why. If the interviewer pushes on the threshold or on the cost of each mistake, the cost arithmetic above is a 30-second answer. Everything else about measurement can wait, and a sentence saying you would return to evaluation in the deep dive is enough to park it.
Interview questions
The model got better at predicting clicks, yet the live experiment shows people coming back less often. Can both results be right at the same time?
Yes, and I would expect them to disagree sometimes. The offline result says the model predicts clicks on past data well, and the live result says the decisions built on those predictions made the product worse for people. A click is a stand-in for enjoyment, and a model that gets very good at predicting clicks starts favoring tempting titles that disappoint after the click.
I would first rule out a broken experiment or a serving bug, then look at what changed in the feed, such as more short attention-grabbing items or less variety. If those checks hold up, I would change the target to something closer to the goal, such as watching for at least 30 seconds, and rerun the comparison before declaring either number wrong.
You are asked to design the system that suggests dishes to a table. Which type of ML task is it?
The waiter’s decision is which three dishes to mention, so the answer has to be an order over the menu, which makes this a ranking task. I would score each dish for this table with a probability that the table orders it and sort by that score, which means the model underneath can be a classifier even though the task as a whole is ranking.
With thirty dishes there is no retrieval step, because scoring thirty items is cheap. If this were a chain with ten thousand dishes across its history, I would add a retrieval step that pulls a few hundred plausible dishes before ranking them. I would say that out loud so the interviewer knows I see the difference in scale.
Given the model’s probabilities, how would you decide which bookings the host phones?
I would write down the cost of each mistake and let the threshold fall out of them. If an empty table costs about 100 units and an unnecessary call costs about 5, calling is worth it once the no-show probability is above 5 divided by 105, which is roughly 5 percent. The host then phones everyone the model scores above that.
That only works if the probabilities are calibrated, meaning bookings scored at 10 percent fail about 10 percent of the time, so I would check calibration on held-out bookings first. I would also cap the number of calls at what the host can manage in an afternoon and phone the highest-scoring bookings first.
What to practice next
Design these systems on the canvas and get your design graded.
This problem lets you compute accuracy, precision and recall for a set of yes-or-no predictions, which is the first check on a classification target before any threshold is chosen.