Reusing pretrained models

A home cook making a tart can laminate puff pastry by hand, which takes a day of folding and chilling, or buy a frozen sheet that a factory made for everyone. The bought sheet is good, it is ready in minutes, and almost every cook uses it. The cook still decides the filling, the shape and the baking time, and those are where the tart becomes theirs.

The bought sheet has costs that are easy to miss. It comes in one size, so it may need trimming or may not fit the tin. Its recipe belongs to the factory, which can change it without telling anyone. It was made for general tarts, so a cook with an unusual dish may find it does the wrong thing. A pretrained model is the frozen sheet, and the filling and shaping are the small part you train yourself.

At the kitchen door the comparison stops, because a pastry sheet is used once, while a pretrained model is called millions of times a day, so its size and speed become a running cost rather than a one-off. The sections below take the price per call, the fit to your data and the dependence on the supplier in turn.

A pretrained model usually does one general job well. A text encoder turns a sentence into a list of a few hundred numbers, called an embedding or a vector, in which sentences with similar meaning end up close together, and an image encoder does the same for pictures. Those numbers become the input to a small model of your own that learns your task, such as whether a support ticket is urgent, from your labels.

The cheapest use is to freeze the encoder, meaning its numbers never change, and train only the small model on top. The middle option is fine-tuning, where training is allowed to adjust the encoder a little so its numbers suit your data, at the price of more computing time and more labels. The most expensive is training the encoder from scratch, which is rarely the right call unless your data looks nothing like the public data it was trained on, such as medical scans or machine logs.

The saving shows up in labels. Assume a from-scratch text classifier needs about a million labelled tickets to reach a useful accuracy, while a frozen encoder with a small model on top reaches the same accuracy with about ten thousand. At two minutes of a reviewer’s time per label, that is the difference between roughly 33,000 hours and 330 hours of labelling. That gap is the difference between a project that happens and one that does not.

A pretrained encoder is usually large, and large means slow and memory hungry. Assume a text encoder takes 30 milliseconds per item on one graphics processor, while the small model on top takes under a millisecond. If the request budget for ranking is 40 milliseconds and there are 300 candidates, encoding them live is out of the question, and the design has to compute the item vectors ahead of time and store them.

Precomputing has its own arithmetic. Two million catalogue items at 30 milliseconds each is 60,000 seconds, or about 17 hours on one processor, so a nightly refresh needs around eight processors to finish in two hours. Storage is a smaller problem than the processor time. Two million vectors of 256 numbers at 4 bytes each is about 2 gigabytes, which fits in memory on one serving machine. The query side, the thing the person typed, still has to be encoded live, and that single 30-millisecond call has to fit inside the budget.

A smaller encoder cuts both numbers, and a technique called distillation trains a small model to copy a large one’s outputs so that most of the quality survives at a fraction of the cost. When the encoder sits behind a paid interface rather than on your machines, the price per call replaces the processor arithmetic. A product that makes ten million calls a day at a tenth of a cent each is paying 10,000 dollars a day.

Inside a large company the pretrained model is frequently a table rather than a file. One team trains a vector for every user from everything the user has done, and the ads, search and recommendation teams each read those vectors as inputs to their own models. It is the same reuse, and it saves each team from learning the user from scratch.

The price of that reuse is dependence on the team that owns the table. If the embedding team retrains with a new method, every vector changes meaning overnight, and a model downstream that was trained on the old vectors is now reading a language it was never taught. The 2015 Google paper on hidden technical debt calls this an unstable data dependency and names versioning as the cure. In practice that means the embedding team publishes a numbered version, downstream models pin the version they were trained on, and a switch is a deliberate retrain rather than a surprise.

Freshness is the other half. A user vector refreshed nightly knows nothing about what the person did an hour ago, so a design that needs recent behaviour pairs the shared vector with a few live inputs of its own. New users have no vector at all, and the downstream model needs a default for them and a rule for when a vector is old enough to be dropped.

A public model comes with a licence, which is the owner’s statement of what you may do with it, and some licences forbid commercial use or require that anything built on the model be shared back. The check takes a minute, and skipping it has unwound products. A model behind a paid interface has terms too, usually about what data you may send and whether the provider may keep it, which matters when the inputs are customer messages.

The second check is whether the model has seen data like yours. An encoder trained on web text handles customer emails well and legal contracts moderately, and a general image model trained on photographs struggles with X-ray films. The quick test is to freeze the encoder, train the small model on top and compare it with the baseline. If the frozen encoder barely beats the popularity list, fine-tune it, and if fine-tuning still fails, the data is far enough from the public world to justify training your own.

The last check is the one newcomers skip. Reusing a model means someone else decides when it changes, so write down which version you use and test the new one before switching, the same way you would for a shared embedding. A pinned version that is retested on a schedule is a dependency you control, and an unpinned one changes under you at a time of someone else’s choosing.

Newcomers assume that reusing is cheating and that an interviewer wants to see a model trained from scratch, when interviewers expect reuse. Every reference walkthrough of a search or moderation system starts from a pretrained encoder, and a candidate who proposes training a language model from nothing for a support assistant has signalled that they do not know what that costs.

A pretrained model is also thought to be free at serving time because the training happened elsewhere. The training was free, and every call still pays the full size of the model, so a 30-millisecond encoder inside a 40-millisecond budget is a design problem whoever trained it.

The last confusion is treating a shared embedding as a fixed fact about the user. It is the output of another team’s model, which changes when they retrain, goes stale between refreshes and is missing for anyone new.

The moment the interviewer asks what the model is, reuse comes up, and two or three minutes of the modelling phase go on it. Say which kind of pretrained encoder you would start from, whether you would freeze or fine-tune it and why, and give the serving arithmetic that decides between encoding ahead of time and encoding live.

If the company has a shared user embedding, say you would use it, name the version pin and the default for new users, and move on. The follow-ups test whether the reuse was understood or assumed.