05 · Models and training
Training pipelines and versioning
3 min read
Choosing a model and loss tells us what to train, but a system needs a reliable way to repeat that work as data changes. A training pipeline connects data preparation, model fitting, evaluation, and the recording of a candidate version. Tracking the inputs and results of each run makes comparisons explainable and helps prevent an unsuitable model from reaching the application.
Define the inputs and outputs of a run
A run should identify the dataset snapshot, label policy, split boundaries, feature definitions, code revision, configuration, and initial model or checkpoint. Record evaluation results and the final artifact alongside these inputs. A random seed is useful, but by itself does not guarantee identical results across hardware or nondeterministic kernels.
An artifact is a saved output of the run that is needed to use or inspect the model later. Alongside learned weights, this can include vocabularies, scaling parameters, category mappings, and other preprocessing state. Versioning these together preserves the compatible set required to transform an incoming request into a prediction.
Find the bottleneck before distributing
Before adding more training hardware, measure where the run spends its time. An accelerator may be idle because input loading or preprocessing cannot supply data quickly enough, in which case additional devices will wait on the same bottleneck. Improving or parallelizing that preparation step lets you establish whether more model compute would actually shorten the run.
In data parallel training, workers each hold a model replica, process different batches, and synchronize gradients before updating their parameters. Communication overhead and uneven work can limit the speedup, while the larger effective batch size may require changes to optimization. Model parallelism distributes parts of the model across devices, which can help when a full replica cannot fit or be computed efficiently on one device.
Checkpoint enough state to resume correctly, which can include optimizer, scheduler, step counter, and data-order state as well as weights. Checkpoint frequency trades storage and pause overhead against work lost after a failure. Test resume behavior instead of assuming that a saved weight file restores the whole run.
Separate training success from release approval
A job finishing successfully establishes that code ran. Release checks should also verify input schemas, output signatures, quality against the incumbent, important slices, artifact compatibility, and serving performance. A candidate with a better average score can still fail if it degrades an important user segment or misses the latency limit.
A registry records immutable versions and their metadata; a named alias can identify the candidate or currently selected version. Deployment should resolve and log the actual version used. Keep the previous compatible artifact set available so an alias change or rollout can be reversed without reconstructing old training state.
The pipeline should be able to reject an incomplete or unsuitable candidate while keeping the current model available. This separates a successful training job from evidence that its output is ready to use. The next topics examine that evidence, beginning with how classification metrics connect prediction errors to the decisions the application makes.