01 · Understanding ML systems
When machine learning is the right tool
6 min read
A product goal is the improvement we want people to notice when they use a product, written down before any model is mentioned. Deciding whether machine learning is the right tool means tracing that goal to one decision the product makes, asking what a prediction would add to that decision, and comparing the learned approach with a plain rule. The answer decides whether a model gets built at all, and if so, what it is for.
Start with the person’s need
A neighborhood clinic books about 200 appointments a day, and roughly one patient in ten never arrives, so around 20 slots a day sit empty while other people wait weeks to be seen. The practice manager asks for an AI that fixes no-shows. That sentence names a tool and hides the need, which is the empty slots and the people who could have filled them. Every number in this example is an assumption chosen to keep the arithmetic easy to follow.
Begin by describing the experience you want to improve in words that make sense without mentioning a model. For the clinic, the people affected are the patients waiting for an appointment and the receptionists who plan each day. Three things keep that description honest.
- Who needs help. Name the people affected and the moment they are in when the product touches them. At the clinic, that is the receptionist planning tomorrow’s list at four in the afternoon and the patient who could have taken an empty slot.
- What they are trying to do. Describe the task in their words, which here is filling the day’s appointments, and resist describing the feature you plan to build.
- What gets in their way. Look for one specific difficulty, such as wasted effort, repeated mistakes, or information that arrives too late. The clinic learns about a no-show at the moment the patient fails to appear, which is too late to offer the slot to anyone else.
Find the decision the product can change
Turn the difficulty into a product goal that a person could check, such as cutting empty slots from 20 a day to 10. Then find a decision the product makes that could move that number. The clinic could decide which patients get a reminder call the day before, and the receptionists have time for about 30 calls, so the decision is which 30 of 200 patients to phone.
That decision gives a prediction its job. A model that estimates each patient’s chance of not arriving lets the receptionist spend the 30 calls on the 30 likeliest no-shows instead of the first 30 names on the list. Ask what the product would do differently if the prediction changed. If the clinic would phone the same 30 people regardless, the prediction has no job and the model should not be built.
Keep the product goal and the model’s task apart
The product goal is the change people should notice, which for the clinic is fewer empty slots. The model’s task is the single output the decision needs, which is a chance of not arriving for each booked patient. These are measured on different things. The task is scored on past appointments where the outcome is known, and the goal is counted on the clinic’s calendar in the weeks after the calls begin.
Keeping both written down lets the team explain the chain from a better prediction to a better day at the clinic, and it exposes the places where the chain can break. The model can get better at spotting no-shows while the slots stay empty, because a reminder call only helps if the patient can still cancel in time for someone else to book. Two more things belong next to the goal when you write it down.
- Evidence of progress. Decide which observable change would show the experience improved. For the clinic, the count of empty slots per day is that evidence, and the count of calls made is only activity.
- Effects to protect against. Name what must not get worse. A patient who always arrives should not be phoned every week until they stop answering, and no group of patients should be singled out because of where they live.
Check whether learning from data is the right tool
Machine learning is worth considering when the decision depends on patterns that are hard to write down as rules and when records exist that carry those patterns. A plain rule is the right tool when it is good enough, and it should be tried first because it is cheaper to build, easier to explain and impossible to break by retraining.
For the clinic, a rule might say to phone anyone who has missed an appointment before. If that catches 8 of the 20 daily no-shows, the rule has done 40 percent of the job for the price of a spreadsheet filter. A model trained on two years of past appointments, around 100,000 of them, might learn that bookings made over a month ahead, Monday mornings and first visits all raise the chance of a no-show. With the same 30 calls it might catch 14 of the 20. The difference of 6 filled slots a day is the benefit the model has to justify.
Against that benefit sits the cost of gathering and cleaning the appointment records, running the model every afternoon, watching it for mistakes, and retraining it when the clinic’s patients change. The table compares the two tools on the questions that decide between them.
Two confusions show up at this stage. The first is treating the request for AI as the requirement, which skips the need and usually produces a model with no decision to serve. The second is assuming that a pattern people cannot describe must exist in the records. If the clinic has never recorded booking dates or first visits, the model has nothing to learn those patterns from, and the honest answer is to start recording them and use the rule in the meantime.
When the decision is clear and the records support it, the next step is to understand what a model does with those records. The next topic explains training and prediction, the two kinds of work involved in producing a model from past appointments and using it on tomorrow’s list.
| Question | A written rule | A learned model |
|---|---|---|
| Where the logic comes from | A person writes it down | Found in past records |
| What it needs to exist | Someone who knows the pattern | Enough records with known outcomes |
| How it fails | Misses patterns nobody thought of | Learns a pattern that stops being true |
| What it costs to keep | An edit when the rule changes | Monitoring and retraining |