Most modern training pipelines start with a warmup phase where the learning rate gradually increases from zero, followed by a decay phase where it gradually decreases. This prevents early instability from large updates while still allowing the optimizer to escape sharp minima later.
Given a base learning rate, warmup steps, total steps, and the current step, compute the learning rate at that step.
Warmup phase (current_step < warmup_steps): the learning rate increases linearly from 0 to base_lr.
lr=base_lr×warmup_stepscurrent_stepDecay phase (current_step >= warmup_steps): the learning rate decreases linearly from base_lr to 0.
lr=base_lr×total_steps−warmup_stepstotal_steps−current_stepInput: base_lr = 0.1, warmup_steps = 10, total_steps = 100, current_step = 5
Output: 0.05
Explanation: Step 5 is halfway through the linear warmup.
Input: base_lr = 0.1, warmup_steps = 10, total_steps = 100, current_step = 55
Output: 0.05
Use the warmup fraction when the current step is below the warmup boundary.
Use the remaining-step fraction during linear decay.
Accepts: number
Accepts: number
Accepts: number
Accepts: number
Most modern training pipelines start with a warmup phase where the learning rate gradually increases from zero, followed by a decay phase where it gradually decreases. This prevents early instability from large updates while still allowing the optimizer to escape sharp minima later.
Given a base learning rate, warmup steps, total steps, and the current step, compute the learning rate at that step.
Warmup phase (current_step < warmup_steps): the learning rate increases linearly from 0 to base_lr.
lr=base_lr×warmup_stepscurrent_stepDecay phase (current_step >= warmup_steps): the learning rate decreases linearly from base_lr to 0.
lr=base_lr×total_steps−warmup_stepstotal_steps−current_stepInput: base_lr = 0.1, warmup_steps = 10, total_steps = 100, current_step = 5
Output: 0.05
Explanation: Step 5 is halfway through the linear warmup.
Input: base_lr = 0.1, warmup_steps = 10, total_steps = 100, current_step = 55
Output: 0.05
Use the warmup fraction when the current step is below the warmup boundary.
Use the remaining-step fraction during linear decay.
Accepts: number
Accepts: number
Accepts: number
Accepts: number