Training and prediction

A parent drives a child to school every morning. After a few months the parent can look at the clock and the rain and say the trip will take 25 minutes today. Nobody taught them a formula. Hundreds of past drives, each with a start time, the weather and the time it took, built a feel for how those things go together.

That is training and prediction in miniature. The months of drives are the examples, the feel for how long it takes is the learned pattern, and this morning’s guess is the prediction. The analogy breaks in one place that matters for design. The parent learns a little more on every drive, while a model in a product stops learning the moment training ends and keeps making predictions with the same fixed pattern until someone trains it again.

Training feeds the model thousands of examples, each pairing the information available at the time with the outcome that followed. Learning from pairs like that is called supervised learning, because the known outcomes supervise the guesses. The outcome recorded for each example is called the label, and the pieces of information the model is given are called its inputs, or features.

The model makes a guess for each example, compares it with the label, and adjusts its internal numbers to shrink the gap. It repeats this over the whole set of examples again and again, and the result of all that adjusting is a trained model. For a delivery app, the examples would be completed deliveries with their route distance, the traffic at departure and the minutes the trip took.

Before the model is trusted, it is tested on examples that were kept out of training. A model that has memorized its training deliveries will do well on them and badly on the held-out ones, and the held-out score is the one that predicts how it will behave on next week’s deliveries. If 50,000 completed deliveries are available, keeping 5,000 aside for this test still leaves 45,000 to learn from, and the test score is the one the team reports.

Prediction, which engineers also call inference, hands the trained model one new input and takes back an estimate. When a driver sets off, the app sends the route distance and the current traffic to the model and gets back a number of minutes. The true duration is unknown at that moment, so the product treats the number as an estimate with an error around it. Showing the customer a window such as 30 to 40 minutes serves them better than a single figure.

The model does not learn from the request. The same trained model answers thousands of requests in a row with the same fixed pattern, and recording what happened afterward does not change it. Training can run again later on the new deliveries, and that is a separate job with its own schedule.

The two activities also run at different speeds. Training over 45,000 deliveries might take an hour on one machine, once a night. A single prediction has to come back while the customer is looking at the screen, so within about half a second. At dinner time the app might ask for 500 of them a minute.

These numbers are assumptions for this example, and the shape they show is general. Training is heavy work done rarely with nobody waiting for it, while prediction is light work done constantly with someone waiting.

Put together, the delivery app has two paths that share one model. The top path runs at night and turns completed deliveries into a trained model. The bottom path runs all day and turns each new departure into an estimate. The paths meet at the trained model, which the night path produces and the day path reads.

The paths need different information. The night path needs the outcome of each delivery, which is only known after the driver arrives. The day path needs facts available at departure, such as the distance and the traffic right now, and it cannot use anything learned later. A design that lets the day path see information from after departure will look accurate in testing and fail with paying customers, which is a trap the data topics return to.

Over weeks, the picture is a long stretch of predictions with an occasional retraining. The model trained on Sunday night answers every request until the next retraining, and the deliveries it predicted during the week become the examples for that retraining. New roads, a new depot or a change in the kind of customers slowly make the Sunday model worse, and retraining is how the system catches up.

This separation is what makes the system designable. The team can choose how often to retrain, where the heavy training work runs, how many machines answer predictions and what happens when the model is unavailable, because the two jobs can be planned apart.

The belief that takes longest to unlearn is that the model keeps learning while it is in use. Unless the team builds a retraining job, the model’s pattern is frozen at training time, and a system that has answered a million requests knows exactly as much as it did on the day it was trained.

A strong training score also gets read as proof that the product will work. The score on held-out examples is the honest one, and even it describes last month’s deliveries under last month’s conditions, which is why the product keeps comparing estimates with what happened.

Treating a prediction as a fact is the mistake that reaches customers. An estimate of 35 minutes is a best guess with an error around it, and a product that promises 35 minutes to the customer takes on a risk the model never claimed to cover.

Training needs completed examples with known outcomes, while prediction needs whatever is known at the moment the product asks. The trained model is the only thing the two activities share, and everything else about them, including their timing, their machines and their data, can be decided separately.

We still need to decide how quickly the estimate must appear and how close it must be for customers to trust it. The next topic, clarifying requirements, turns a product goal into expectations like those that the system has to meet.