Cosine Annealing LR Scheduler
Cosine Annealing LR Scheduler
Cosine annealing smoothly decreases the learning rate following a half-cosine curve. Unlike linear decay, it slows down the rate of decrease near the start and end, spending more training time at moderate learning rates. This schedule is widely used in vision and language model training.
Given a base learning rate, a minimum learning rate, total steps, and the current step, compute the learning rate.
Formula
lr=min_lr+21(base_lr−min_lr)(1+cos(total_stepsπ⋅current_step))At step 0 the cosine term equals 1, so lr = base_lr. At step = total_steps the cosine term equals -1, so lr = min_lr.
Examples
Input:
base_lr = 0.1, min_lr = 0.0, total_steps = 100, current_step = 0
Output:
0.1
At step 0, cos(0) = 1, so lr = 0 + 0.5 * 0.1 * (1 + 1) = 0.1.
Input:
base_lr = 0.1, min_lr = 0.0, total_steps = 100, current_step = 50
Output:
0.05
At the halfway point, cos(pi/2) = 0, so lr = 0 + 0.5 * 0.1 * (1 + 0) = 0.05.
Hint 1
You will need math.cos and math.pi. The formula is a single expression with no branching.
Hint 2
The (base_lr - min_lr) factor scales the cosine curve to oscillate between base_lr and min_lr instead of 1 and -1.
Requirements
- Apply the cosine annealing formula exactly as specified
- Support a non-zero minimum learning rate
- Return base_lr at step 0 and min_lr at step = total_steps
Constraints
- base_lr > min_lr >= 0
- total_steps > 0, 0 <= current_step <= total_steps
- Return a single float
- Time limit: 300 ms
Try Similar Problems
Log in to take notes on this problem
Accepts: number
Accepts: number
Accepts: number
Accepts: number
Cosine Annealing LR Scheduler
Cosine Annealing LR Scheduler
Cosine annealing smoothly decreases the learning rate following a half-cosine curve. Unlike linear decay, it slows down the rate of decrease near the start and end, spending more training time at moderate learning rates. This schedule is widely used in vision and language model training.
Given a base learning rate, a minimum learning rate, total steps, and the current step, compute the learning rate.
Formula
lr=min_lr+21(base_lr−min_lr)(1+cos(total_stepsπ⋅current_step))At step 0 the cosine term equals 1, so lr = base_lr. At step = total_steps the cosine term equals -1, so lr = min_lr.
Examples
Input:
base_lr = 0.1, min_lr = 0.0, total_steps = 100, current_step = 0
Output:
0.1
At step 0, cos(0) = 1, so lr = 0 + 0.5 * 0.1 * (1 + 1) = 0.1.
Input:
base_lr = 0.1, min_lr = 0.0, total_steps = 100, current_step = 50
Output:
0.05
At the halfway point, cos(pi/2) = 0, so lr = 0 + 0.5 * 0.1 * (1 + 0) = 0.05.
Hint 1
You will need math.cos and math.pi. The formula is a single expression with no branching.
Hint 2
The (base_lr - min_lr) factor scales the cosine curve to oscillate between base_lr and min_lr instead of 1 and -1.
Requirements
- Apply the cosine annealing formula exactly as specified
- Support a non-zero minimum learning rate
- Return base_lr at step 0 and min_lr at step = total_steps
Constraints
- base_lr > min_lr >= 0
- total_steps > 0, 0 <= current_step <= total_steps
- Return a single float
- Time limit: 300 ms
Try Similar Problems
Log in to take notes on this problem
Accepts: number
Accepts: number
Accepts: number
Accepts: number