Fallbacks and model rollouts

When the oven in a restaurant kitchen fails at seven in the evening, the kitchen does not close and does not keep taking orders for roast chicken. It serves the dishes that need no oven, tells the waiters which ones those are, and keeps the oven items off the menu until the repair is done. A good kitchen decided this before the oven ever failed, and the cold dishes it falls back on are ones it already knows how to make well.

The same kitchen tries a new dish as a special on a few tables before printing it on the menu, and it keeps the old recipe card in the drawer in case the new one disappoints. The special is a canary, the recipe card is the rollback, and the menu change is the rollout. On judgment the kitchen and the service differ, because a head chef can improvise on the night while a service running at 2,000 requests a second cannot, so every degraded answer and every stop rule has to be written down and tested in advance.

The right fallback depends on what the prediction was going to do. A recommendation page can show a cached list of eligible items or the popular list, and a user who sees a slightly worse page has lost little. A payment system whose fraud model times out cannot guess, so it falls back to a small set of hand-written rules, asks the customer for an extra verification step, or sends the payment to review. A document assistant that cannot find evidence says so instead of writing a confident answer. These are product policies with different consequences, and the engineering question is only how to make each one reliable.

For each input the service depends on, the design states whether its absence allows a reduced answer or stops the action. A missing recent-preference feature permits a less personal recommendation, while a missing permission check never permits showing restricted content from a cache. Writing that distinction down per dependency keeps a drop in prediction quality separate from a broken access rule, which is the line a reviewer will ask about after an incident.

A fallback is tested the way the main path is tested. If a feature can be missing at serving time, the model is trained and evaluated with that feature missing some of the time, because a model that has never seen a blank in that field can return an extreme score when it does. A separate small model for the degraded case is sometimes easier to trust than hoping a large model copes with inputs it never saw.

Teams also break things on purpose in a safe environment, switching off a dependency and watching what the service returns and how fast. The test checks the timeouts, how much load lands on the fallback path, how old a cached answer is allowed to be, and whether the service returns to the healthy path smoothly when the dependency comes back. A popular-items cache that receives all the traffic at once can itself fall over, so the capacity plan includes the degraded mode, and the fraction of requests served by the fallback is watched as a daily number.

A new model version reaches users in stages so a problem is caught while it is small. In shadow mode the new model scores every request alongside the old one but its answers are never shown, so the team can compare scores and latency on live traffic at no risk to users. A canary then serves the new version to a small slice of traffic, such as one percent, with the old version serving the rest, and the slice grows only while the agreed health and quality signals stay inside their limits. The last step is full traffic with the old version kept ready to take over.

The size of the canary decides what it can detect. At one percent of 40 million daily requests the canary sees about 2,800 requests in ten minutes, which is enough to notice a crash, a timeout, or a spike in errors almost at once. A change of a tenth of a percent in click rate would need days at that slice, so the canary catches breakage and the longer experiment from the evaluation section catches quality.

A rollout has a written stop rule and a named person or an automatic check that is allowed to halt it. A typical rule is the p99 latency, the time the slowest one in a hundred requests takes, rising above 250 milliseconds, or the error rate rising above half a percent. Without that rule the slice grows on schedule while someone argues about whether the dashboard shows a problem, which is how a one-percent problem becomes a hundred-percent one.

A rollback returns the service to the previous version when a release causes a problem, and it only works if the previous version still fits the rest of the system. A model is one piece of a set that also includes the definitions of its features, the vocabulary that maps words or categories to numbers, and the embedding index it searches, the stored lists of numbers that stand for items. The earlier lessons on feature consistency apply here, because old model weights given a new vocabulary produce confident nonsense. The whole set is kept together under one version name, so rolling back restores exactly what was tested.

A rollback changes what is served from now on and does not undo what already happened, so every logged prediction carries its version. The fraction of requests on the fallback path and the number of rollbacks in a quarter are both recorded, because a service can return a response for every request and still be serving the degraded answer to everyone. The same ideas carry over when the output is generated text, and the next topic looks at the extra serving work that language models bring.

Newcomers assume there is a universally safe default, such as approving every payment when the fraud model is down so that checkout keeps working. There is no safe default that holds for every product, since approving everything invites fraud and blocking everything loses sales, and the only honest answer is the policy the product owner wrote down in advance.

Shadow mode and a canary get mixed up. Shadow mode shows users nothing new and can only measure scores and speed, while a canary shows a slice of users the new answer and can measure what they do. A team that wants to know whether users click more needs the canary and then the experiment, and shadow mode cannot tell them.

A rollback is thought of as pressing a button. It is only that if the previous model, its features, and its index were kept together as one set, and a team that saved the weights alone finds at the worst moment that nothing else still matches them.

Near the end of the design, fallbacks and rollouts take two to three minutes, and in a payments or safety question the fallback is the design and takes longer. Give one sentence per dependency, saying what the service returns when it is missing, and one sentence on how a new version reaches users, such as shadow, then a one-percent canary with a latency stop rule. Offer the rollback detail only if asked.