Cosine annealing smoothly decreases the learning rate following a half-cosine curve. Unlike linear decay, it slows down the rate of decrease near the start and end, spending more training time at moderate learning rates. This schedule is widely used in vision and language model training.
Given a base learning rate, a minimum learning rate, total steps, and the current step, compute the learning rate.
At step 0 the cosine term equals 1, so lr = base_lr. At step = total_steps the cosine term equals -1, so lr = min_lr.
Input:
base_lr = 0.1, min_lr = 0.0, total_steps = 100, current_step = 0
Output:
0.1
At step 0, cos(0) = 1, so lr = 0 + 0.5 * 0.1 * (1 + 1) = 0.1.
Input:
base_lr = 0.1, min_lr = 0.0, total_steps = 100, current_step = 50
Output:
0.05
At the halfway point, cos(pi/2) = 0, so lr = 0 + 0.5 * 0.1 * (1 + 0) = 0.05.
You will need math.cos and math.pi. The formula is a single expression with no branching.
The (base_lr - min_lr) factor scales the cosine curve to oscillate between base_lr and min_lr instead of 1 and -1.
Sign in to take notes on this problem
Accepts: number
Accepts: number
Accepts: number
Accepts: number
Cosine annealing smoothly decreases the learning rate following a half-cosine curve. Unlike linear decay, it slows down the rate of decrease near the start and end, spending more training time at moderate learning rates. This schedule is widely used in vision and language model training.
Given a base learning rate, a minimum learning rate, total steps, and the current step, compute the learning rate.
At step 0 the cosine term equals 1, so lr = base_lr. At step = total_steps the cosine term equals -1, so lr = min_lr.
Input:
base_lr = 0.1, min_lr = 0.0, total_steps = 100, current_step = 0
Output:
0.1
At step 0, cos(0) = 1, so lr = 0 + 0.5 * 0.1 * (1 + 1) = 0.1.
Input:
base_lr = 0.1, min_lr = 0.0, total_steps = 100, current_step = 50
Output:
0.05
At the halfway point, cos(pi/2) = 0, so lr = 0 + 0.5 * 0.1 * (1 + 0) = 0.05.
You will need math.cos and math.pi. The formula is a single expression with no branching.
The (base_lr - min_lr) factor scales the cosine curve to oscillate between base_lr and min_lr instead of 1 and -1.
Sign in to take notes on this problem
Accepts: number
Accepts: number
Accepts: number
Accepts: number