Which Optimizer to Use in Deep Learning, and Why
A decision table for choosing between AdamW, SGD with momentum and the other common optimizers, with starting learning rates and memory per weight.
Which optimizer should you use in deep learning?
Use AdamW by default for transformers and for any new project where you cannot tune much, starting near a learning rate of 3e-4 with warmup. Use SGD with momentum 0.9 and a decaying schedule for convolutional image classifiers when you can tune it. Tune the learning rate before switching optimizers.
| Your situation | Use | Starting settings | Why | Learn and practice |
|---|---|---|---|---|
| Training or fully fine-tuning a transformer or language model | AdamW | Peak learning rate near 3e-4 for small models and lower for large ones, or 1e-5 to 5e-5 when fully fine-tuning a pretrained model. Betas 0.9 and 0.95 for large pretraining runs, while the default beta2 of 0.999 is fine for small models. Weight decay 0.1 for pretraining and about 0.01 for fine-tuning. Linear warmup, then cosine or warmup-stable-decay. Clip the global gradient norm at 1.0. | Per-weight step sizes cope with the very different gradient scales of embeddings, attention and normalization layers, and decoupled weight decay regularizes every weight at the same rate. | |
| LoRA or QLoRA fine-tuning of a large model | AdamW on the adapter weights only | Learning rate 1e-4 to 2e-4, weight decay 0 to 0.01 and a short warmup. | Only the small adapter matrices are trained, so they tolerate a higher learning rate than full fine-tuning, and optimizer state exists only for those few weights. | |
| Convolutional network for image classification, with time to tune | SGD with momentum 0.9, often Nesterov | Learning rate 0.1 at batch size 256, scaled linearly with batch size. Weight decay 1e-4 at ImageNet scale or 5e-4 at CIFAR scale. Cosine or step decay, with a short warmup for large batches. | Tuned SGD with momentum often reaches equal or better test accuracy than Adam on these models, and it stores one buffer per weight instead of two. | |
| New architecture or quick baseline with little tuning time | Adam or AdamW | Learning rate 1e-3, betas 0.9 and 0.999, epsilon 1e-8, which are the library defaults. | Adam is less sensitive than SGD to the starting learning rate, so the first run usually trains. | |
| Sparse features such as large embedding tables, click-through or bag-of-words models | AdaGrad or Adam | AdaGrad learning rate 0.01 to 0.1, or Adam at 1e-3. | Rarely seen features keep a large step because their accumulated squared gradient stays small. AdaGrad’s step size only ever shrinks, so it can stall on long dense runs. | |
| Recurrent networks or reinforcement learning with a shifting objective | Adam or RMSprop | RMSprop learning rate near 1e-3 with a decay of 0.9 to 0.99, or Adam at 1e-3 to 3e-4. | A moving average of squared gradients adapts when the gradient scale drifts during training. | |
| Small dataset with a smooth, deterministic loss that fits in one batch | L-BFGS | Full batch, a history of 10 to 20 gradient pairs and a line search. | It builds a curvature estimate from recent gradient differences and needs far fewer iterations, but mini-batch noise and dropout break that estimate. | |
| Convex classical model on a huge dataset, such as linear or logistic regression | Mini-batch SGD with a decaying learning rate | Learning rate 0.01 to 0.1, decayed over training. | Each step is cheap, and on a convex loss a suitably decaying step size converges to a global minimum. | |
| Optimizer state does not fit in GPU memory | 8-bit AdamW, Adafactor or Lion, or shard the state across GPUs | Keep the AdamW settings for 8-bit states. For Lion, use a learning rate 3 to 10 times smaller and weight decay 3 to 10 times larger than for AdamW. | AdamW stores two extra values per weight, and these options store fewer of them, store them in fewer bytes or split them across devices. | |
| Distributed training at a very large batch size | LARS for SGD-style training or LAMB for Adam-style training | Warmup plus a learning rate scaled up with the batch size. | A per-layer trust ratio of weight norm to update norm keeps every layer’s step in proportion, which keeps huge batches stable. |
Every value in the table is a common starting point to sweep around, and none of them is guaranteed to be best for your model. The update rules behind each optimizer are derived step by step in the Adam, RMSprop and AdaGrad lesson and the momentum and Nesterov lesson, so this page only covers how to choose between them.
Should you use Adam or SGD?
Use Adam, or better AdamW, when you want a run that trains well on the first try, and use SGD with momentum when you have the time to tune a learning rate schedule and the model is a convolutional network. The two differ on four practical points.
- Adam usually drives the training loss down faster in the early epochs, because it divides each weight's step by a running estimate of that weight's gradient size.
- Adam is more forgiving of a poor learning rate. SGD can stall at a learning rate that is too low and diverge at one that is only a few times too high.
- On image classification with convolutional networks, carefully tuned SGD with momentum often reaches equal or slightly better test accuracy. A common explanation is that it settles in flatter regions of the loss surface, which the loss landscapes lesson discusses.
- Adam stores two extra values per weight while SGD with momentum stores one, which matters once the model reaches billions of parameters.
For transformers the question is mostly settled in favor of AdamW. Their gradients differ in scale by orders of magnitude between the embeddings, the attention projections and the normalization gains, and plain SGD handles that spread poorly in practice.
What is the difference between Adam and AdamW?
AdamW subtracts weight decay from the weights directly, while Adam with a weight decay setting adds an L2 penalty to the gradient before the adaptive step. In Adam the penalty term is divided by along with the rest of the gradient, so weights with a history of large gradients are regularized much less than the rest. AdamW, in the form most libraries implement, keeps the decay outside the fraction.
Here and are the bias-corrected running averages of the gradient and the squared gradient, is the learning rate and is the weight decay. Every weight now shrinks by the same fraction per step whatever its gradient history, which is why a weight decay value tuned for AdamW transfers between models more predictably. The derivation is in the AdamW section of the adaptive rates lesson, and the regularization lesson covers why weight decay helps in the first place. The update rules for the other optimizers are collected in the optimization section of the ML math formula sheet.
In practice, choose AdamW whenever you want weight decay with an adaptive optimizer, and leave biases and normalization gains out of the decay, which is standard for transformers.
When is SGD with momentum the better choice?
SGD with momentum is the better choice for convolutional image models trained from scratch when you can afford a few runs to tune the schedule, and for convex models such as logistic regression on very large datasets. In both settings its simplicity helps, because there is one learning rate to tune and no adaptive scaling that could interact with regularization.
Pair it with a learning rate schedule, since constant-rate SGD rarely reaches its best accuracy. Cosine decay and step decay are the usual choices, and the SGD and learning rate schedules lesson compares them. Nesterov momentum evaluates the gradient at the look-ahead point and is a small, common improvement over classical momentum at the same setting of 0.9.
What learning rate should you start with for each optimizer?
Start Adam and AdamW at 1e-3 for small networks and near 3e-4 for transformers, start SGD with momentum near 0.1 for convolutional networks at batch size 256, and treat each as the center of a sweep. Learning rates do not transfer between optimizers, since a good SGD rate is often around a hundred times a good Adam rate because Adam's step is already normalized by the gradient size.
| Optimizer | Extra state per weight | Common starting values | Main weakness |
|---|---|---|---|
| SGD | None | Learning rate 0.01 to 0.1 | Slow along narrow valleys and very sensitive to the learning rate. |
| SGD with momentum | 1 (velocity) | Momentum 0.9, learning rate 0.01 to 0.1 | Needs a tuned learning rate schedule to do its best. |
| Nesterov momentum | 1 (velocity) | Momentum 0.9 | Carries the same tuning burden as plain momentum. |
| AdaGrad | 1 (sum of squared gradients) | Learning rate 0.01 | The step size decays towards zero on long runs. |
| RMSprop | 1 (average of squared gradients) | Learning rate 1e-3, decay 0.9 to 0.99 | The base form has no momentum and no bias correction. |
| Adam | 2 (m and v) | Learning rate 1e-3, betas 0.9 and 0.999, epsilon 1e-8 | Its weight decay setting acts as L2 and is scaled unevenly by the adaptive step. |
| AdamW | 2 (m and v) | Learning rate 1e-3, or 3e-4 for transformers, weight decay 0.01 to 0.1 | Two state values per weight cost memory on large models. |
| Nadam | 2 (m and v) | Learning rate 1e-3 to 2e-3 | Gains over Adam are usually small. |
| Lion | 1 (momentum) | Betas 0.9 and 0.99, learning rate 3 to 10 times below AdamW | Sensitive to batch size and learning rate. |
| Muon | 1 (momentum) | Momentum 0.95 with Nesterov, five Newton-Schulz steps | Applies only to 2D hidden weight matrices, so it runs alongside AdamW. |
| Adafactor | Row and column statistics per matrix | Often run without momentum and with a relative step size | Less stable than AdamW without careful settings. |
| L-BFGS | 2m vectors for a history of m | History 10 to 20, full batch, line search | Needs full-batch, low-noise gradients. |
The batch size moves the learning rate as well. A common rule for SGD is to multiply the learning rate by k when you multiply the batch size by k, with a warmup, while Adam-style optimizers often scale closer to the square root of that ratio. Treat both as starting points, and note that past the critical batch size a bigger batch buys little extra speed whichever optimizer you use. The learning rate and batch size scaling problem fits these relationships from data.
How much memory does each optimizer need?
In mixed precision training, AdamW needs about 16 bytes per weight before activations, which is about 112 GB for a 7B parameter model. SGD with momentum and Lion need about 12 bytes per weight, and plain SGD needs about 8. Pick a model size and a precision below, then open a row to see what each optimizer stores.
- 16 bytes per weight. AdamW keeps the same m and v as Adam. Its weight decay acts on the weights directly and needs no extra memory. AdamW section of the lesson
On a 7B model in mixed precision, AdamW needs about 112 GB for weights, gradients and optimizer state, against about 56 GB for plain SGD.
Mixed precision does not shrink this bill, because the optimizer state and a master copy of the weights stay in fp32, so the fp32 toggle above shows the same 16 bytes per weight for AdamW. The mixed precision lesson explains why. The real saving in mixed precision is in activations and speed. When the state does not fit, the options are to store it in 8 bits, to factor it as Adafactor does, to keep only one buffer as Lion does, or to shard it across GPUs, which the ZeRO memory accounting problem works through stage by stage. Adapter fine-tuning sidesteps most of the cost, because the LoRA optimizer only holds state for the small adapter matrices.
What should you tune before switching optimizers?
Tune the learning rate, the schedule and the warmup before you switch optimizers, because these usually matter more than the choice between two reasonable optimizers. Work through them in this order.
- Sweep the learning rate on a log scale, for example 1e-4, 3e-4, 1e-3 and 3e-3 for Adam, and pick the best value on validation loss.
- Add a linear warmup over the first few percent of steps for any adaptive optimizer, because the early estimates of the squared gradient are noisy. The warmup and linear decay problem builds one.
- Decay the learning rate towards the end of training with a cosine or warmup-stable-decay schedule.
- Clip the global gradient norm, usually at 1.0 for transformers, as in the gradient clipping problem.
- Lower beta2 from 0.999 to 0.95 if a large transformer shows loss spikes, since a shorter average reacts faster to sudden large gradients.
- Retune the learning rate and weight decay from scratch whenever you do switch optimizers, because the old values belong to the old update rule.
With gradient accumulation the optimizer choice stays the same, but the step, the schedule and the weight decay advance once per accumulated batch, as the gradient accumulation lesson shows. If the loss is flat, spiking or turning into NaN, work through the guide to a training loss that is not decreasing before you blame the optimizer.
Are newer optimizers such as Lion and Muon worth trying?
Lion and Muon are worth trying once a tuned AdamW baseline exists, because their gains are real in some reported settings but depend on careful tuning. Neither one should be the first optimizer on a new project.
Lion steps by the sign of an interpolated momentum, so every coordinate moves by the same amount. It keeps one buffer instead of two, but it needs a learning rate 3 to 10 times smaller than AdamW and a weight decay 3 to 10 times larger, and it is sensitive to batch size.
Muon is applied only to the 2D weight matrices in the hidden layers. It takes the Nesterov momentum update for each matrix and orthogonalizes it with a few Newton-Schulz iterations, while AdamW still trains the embeddings, the output layer, the biases and the normalization gains. The Muon update problem implements the orthogonalization step.
What else do people ask about choosing an optimizer?
Is Adam better than SGD?
Adam is not better in every case. It reaches a low training loss faster and with less tuning, which makes it the safer default. Well-tuned SGD with momentum often matches or beats it on test accuracy for convolutional image classifiers, while AdamW is the standard for transformers.
What are the default Adam hyperparameters?
The usual defaults are a learning rate of 0.001, beta1 of 0.9, beta2 of 0.999 and epsilon of 1e-8. These come from the original Adam paper and are the defaults in PyTorch. Transformers usually start lower, near 3e-4, with warmup.
What is the difference between Adam and AdamW?
AdamW subtracts weight decay from the weights directly, outside the adaptive step. Adam with a weight decay setting adds an L2 term to the gradient instead, so the adaptive denominator shrinks the decay for weights with large gradients and the regularization becomes uneven.
Which optimizer uses the least memory?
Plain SGD uses the least, because it keeps no state between steps. SGD with momentum, RMSprop, AdaGrad and Lion keep one value per weight, while Adam and AdamW keep two. Adafactor and 8-bit optimizer states cut Adam-style memory further.
Does batch size change which optimizer you should use?
Batch size mostly changes the learning rate, and the optimizer can usually stay the same. Larger batches need a larger learning rate and a warmup. At very large batch sizes, layer-wise optimizers such as LARS for SGD and LAMB for Adam keep training stable.
Can switching optimizers fix a model that is not learning?
Switching optimizers rarely fixes a run that is not learning. A wrong learning rate, a data bug or a mismatched loss is a far more common cause, so check those first and only compare optimizers after a learning rate sweep for each one.
Practice this
Writing an optimizer step yourself is the quickest way to see why the rows of the decision table differ. Each problem below implements one update rule, and the Optimization study plan and the timed Activations, Losses, and Optimizers mock set go further.
- Implement Adam Optimizer Step has you write the moving averages and bias correction behind the default choice.
- Implement AdamW (Decoupled Weight Decay) has you move weight decay outside the adaptive step.
- Implement Nesterov Momentum (NAG) has you build the update recommended for convolutional image models.
- Warmup + Linear Decay LR Schedule has you write the schedule that adaptive optimizers depend on.