Cosine annealing smoothly decreases the learning rate following a half-cosine curve. Unlike linear decay, it slows down the rate of decrease near the start and end, spending more training time at moderate learning rates. This schedule is widely used in vision and language model training.
Given a base learning rate, a minimum learning rate, total steps, and the current step, compute the learning rate.
At step 0 the cosine term equals 1, so lr = base_lr. At step = total_steps the cosine term equals -1, so lr = min_lr.
Input: base_lr = 0.1, min_lr = 0, total_steps = 100, current_step = 0
Output: 0.1
Explanation: At step zero, the cosine factor selects the base learning rate.
Input: base_lr = 0.1, min_lr = 0, total_steps = 100, current_step = 100
Output: 0.0
Convert the step fraction into an angle by multiplying it by pi.
Scale one plus the cosine between the base and minimum rates.
Accepts: number
Accepts: number
Accepts: number
Accepts: number
Cosine annealing smoothly decreases the learning rate following a half-cosine curve. Unlike linear decay, it slows down the rate of decrease near the start and end, spending more training time at moderate learning rates. This schedule is widely used in vision and language model training.
Given a base learning rate, a minimum learning rate, total steps, and the current step, compute the learning rate.
At step 0 the cosine term equals 1, so lr = base_lr. At step = total_steps the cosine term equals -1, so lr = min_lr.
Input: base_lr = 0.1, min_lr = 0, total_steps = 100, current_step = 0
Output: 0.1
Explanation: At step zero, the cosine factor selects the base learning rate.
Input: base_lr = 0.1, min_lr = 0, total_steps = 100, current_step = 100
Output: 0.0
Convert the step fraction into an angle by multiplying it by pi.
Scale one plus the cosine between the base and minimum rates.
Accepts: number
Accepts: number
Accepts: number
Accepts: number