Training pipelines and versioning

A bakery that sells the same loaf every morning does not improvise it. The baker follows a recipe card, writes the batch number and the flour delivery on it, bakes at the stated temperature, and tastes one loaf from the batch before the rest go on the shelf. If a customer complains on Thursday, the card for Wednesday night says which flour and which oven were used.

When the bakery grows, the owner is tempted to buy a second oven. The night baker points out that the mixer takes an hour per batch and the oven forty minutes, so a second oven would sit cold while everyone waits for dough. The pipeline questions below are the bakery’s questions, namely what went into this batch, where the night’s time is spent, and who decided the loaf was fit to sell.

A baker can judge a loaf in a bite, which is where the analogy breaks, because a model’s quality has to be measured against the current model on data kept aside for testing and on the groups of users where it is most likely to fail. That test is itself a program whose inputs have to be recorded too.

A training run should record which snapshot of the data it used, how the labels were defined and where the data was split into training and test parts. It should also record how each input was computed, which version of the code ran, every setting, and the model it started from if any. The evaluation results and the finished model are recorded next to those inputs. A random seed, the starting number for the random choices training makes, helps, but it does not promise identical results on different hardware, so two runs that differ slightly are normal and two that differ a lot are a bug.

The finished model is a bundle, and engineers call each saved piece an artifact. Next to the learned numbers, called weights, it includes the vocabulary that maps words to numbers, the scaling constants for numeric inputs and the list of categories the model knows. All of these are saved together under one version, because a model loaded with last week’s vocabulary reads every word wrongly while reporting no error at all.

Before adding machines, measure where a run spends its time. Suppose each training step waits 200 milliseconds for the next batch of examples to be read and prepared and then spends 50 milliseconds computing on the graphics processor. The processor is idle four fifths of the time, and a second processor would idle in the same way. The fix is more loading workers or a faster data format, which can bring the step down to around 60 milliseconds with no new hardware.

When compute is the bottleneck, the usual fix is data parallel training. Several processors each hold a full copy of the model, each reads a different batch, and they average their learning signals before every update so the copies stay identical. The averaging means the machines have to talk to each other, which takes time, so eight processors rarely give an eight-fold speedup. The combined batch is also eight times larger, which changes how the model learns and may need a different learning rate, the size of each adjustment. When one copy of the model does not fit on a single processor, the model itself is split across machines, which is called model parallelism and is harder to get right.

Long runs fail, so the state is saved often enough to restart. A twelve-hour run that saves every 30 minutes loses at most half an hour to a crash, and if each save is 2 gigabytes, keeping the last three costs 6 gigabytes of disk. The save has to include the state of the optimizer, the routine that adjusts the weights, the step counter and the position in the data, because a weights-only save that restarts from the first example retrains on data it has already seen. The restart is worth testing once on purpose rather than discovering during a real crash.

A training job that finishes has proved only that the code ran. Before the candidate is allowed near users, the pipeline checks that its inputs still match the format it expects. It checks that the quality beats the current model on the held-out set and on the important slices, such as new users or one language. It also checks that the saved pieces load together and that the model answers within the response time budget. A candidate with a better average can still lose on a group the product cares about.

A model registry is the catalogue that holds every numbered version with its record, and an alias is a name such as production that points at one version. Deployment resolves the alias and logs the exact version it loaded, so an incident on Thursday can be traced to a version and its training record the way the bakery traces a loaf to a card. The previous version’s full bundle is kept, because moving the alias back is the fastest rollback there is and it only works if the old pieces still exist.

The pipeline has to be able to say no. A candidate that fails a check is recorded and rejected while the current model keeps serving, which is what separates a successful training job from a model that is ready to use.

Two of the confusions here come from mistaking a part for the whole. A saved weights file is believed to be the model, when it is one piece of the bundle, and without the vocabulary, the scaling constants and the code version that produced them, the file cannot be loaded correctly, and the failure is silent because numbers still come out. A finished run is treated as a release in the same spirit. Finishing proves the code ran, and the release checks, the alias and the kept previous version are what make the result safe to put in front of people.

The other confusion is assuming that more processors mean a faster run. When loading data is the bottleneck, which it is in most first attempts, extra processors idle the way the first one did. The measurement in the section above costs far less than the hardware.

In a standard round the training pipeline is a sentence or two when you describe how the model is retrained, two or three minutes at most, and it grows only when the problem is about training at scale. Say that each run records its data snapshot, code version and settings, that a candidate is checked against the current model on the important slices before an alias moves, and that the previous version stays ready for rollback.

If the interviewer asks about distributed training, give the bottleneck measurement before the machine count, since a doubled processor count that failed to speed anything up is the usual way the subject arrives.