Generate a warmup-stable-decay learning-rate schedule. Let T be total_steps, w warmup_steps, s stable_steps, d=T−w−s the decay length, ηmax peak_learning_rate, and r final_ratio. For zero-based warmup index t:
ηt=ηmaxwt+1.The next s rates equal ηmax. For zero-based decay index j, normalized progress is:
pj={1,j/(d−1),d=1,d>1.The decay rate is:
ηw+s+j=ηmax[r+(1−r)21+cos(πpj)].Here, η denotes learning rate, cos is cosine and π is pi. A zero-length warmup or stable phase contributes no entries. The final rate is rηmax, including a one-step decay. Return a float64 PyTorch tensor on the CPU with shape (total_steps,), in training-step order.
Input: total_steps = 6, warmup_steps = 2, stable_steps = 2, peak_learning_rate = 1, final_ratio = 0
Output: [0.5,1,1,1,1,0]
Explanation: Two warmup steps reach the peak, two stable steps hold it, and two decay steps include the peak and zero endpoints.
Input: total_steps = 4, warmup_steps = 0, stable_steps = 2, peak_learning_rate = 0.5, final_ratio = 0.2
Output: [0.5,0.5,0.5,0.1]
Input: total_steps = 3, warmup_steps = 1, stable_steps = 1, peak_learning_rate = 2, final_ratio = 0.25
Output: [2,2,0.5]
torch.arange can generate the indices of a phase.
torch.cos evaluates cosine elementwise.
torch.cat joins one-dimensional tensors.
Accepts: number
Accepts: number
Accepts: number
Accepts: number
Accepts: number
Generate a warmup-stable-decay learning-rate schedule. Let T be total_steps, w warmup_steps, s stable_steps, d=T−w−s the decay length, ηmax peak_learning_rate, and r final_ratio. For zero-based warmup index t:
ηt=ηmaxwt+1.The next s rates equal ηmax. For zero-based decay index j, normalized progress is:
pj={1,j/(d−1),d=1,d>1.The decay rate is:
ηw+s+j=ηmax[r+(1−r)21+cos(πpj)].Here, η denotes learning rate, cos is cosine and π is pi. A zero-length warmup or stable phase contributes no entries. The final rate is rηmax, including a one-step decay. Return a float64 PyTorch tensor on the CPU with shape (total_steps,), in training-step order.
Input: total_steps = 6, warmup_steps = 2, stable_steps = 2, peak_learning_rate = 1, final_ratio = 0
Output: [0.5,1,1,1,1,0]
Explanation: Two warmup steps reach the peak, two stable steps hold it, and two decay steps include the peak and zero endpoints.
Input: total_steps = 4, warmup_steps = 0, stable_steps = 2, peak_learning_rate = 0.5, final_ratio = 0.2
Output: [0.5,0.5,0.5,0.1]
Input: total_steps = 3, warmup_steps = 1, stable_steps = 1, peak_learning_rate = 2, final_ratio = 0.25
Output: [2,2,0.5]
torch.arange can generate the indices of a phase.
torch.cos evaluates cosine elementwise.
torch.cat joins one-dimensional tensors.
Accepts: number
Accepts: number
Accepts: number
Accepts: number
Accepts: number
