Ridge regression (L2 regularization) adds a penalty term to the ordinary least squares objective to prevent overfitting. The regularization term shrinks the weights toward zero, which is especially useful when features are correlated or when the number of features is large relative to the number of samples.
Given a feature matrix X, a target vector y, and a regularization parameter lambda, compute the ridge regression weights using the closed-form solution.
Where I is the d x d identity matrix and lambda controls the regularization strength.
Input:
X = [[1, 0], [0, 1]], y = [3, 5], lam = 1.0
Output:
[1.5, 2.5]
X^T X = [[1,0],[0,1]]. Adding lambda*I gives [[2,0],[0,2]]. Inverse is [[0.5,0],[0,0.5]]. X^T y = [3,5]. Result: [1.5, 2.5]. The weights are shrunk from the OLS solution [3, 5] toward zero.
Input:
X = [[1, 1], [1, 2], [1, 3]], y = [3, 5, 7], lam = 0.0
Output:
[1.0, 2.0]
With lambda = 0, ridge regression reduces to ordinary least squares. The perfect fit y = 1 + 2x gives weights [1, 2].
Build a d x d identity matrix I = [[1 if i==j else 0 for j in range(d)] for i in range(d)]. Scale it by lambda and add to X^T X. Then invert and multiply by X^T y, just like ordinary least squares.
The only difference from ordinary least squares is adding lambda * I to X^T X before inverting. This one extra step provides regularization.
Sign in to take notes on this problem
Accepts: array
Accepts: array
Accepts: number
Ridge regression (L2 regularization) adds a penalty term to the ordinary least squares objective to prevent overfitting. The regularization term shrinks the weights toward zero, which is especially useful when features are correlated or when the number of features is large relative to the number of samples.
Given a feature matrix X, a target vector y, and a regularization parameter lambda, compute the ridge regression weights using the closed-form solution.
Where I is the d x d identity matrix and lambda controls the regularization strength.
Input:
X = [[1, 0], [0, 1]], y = [3, 5], lam = 1.0
Output:
[1.5, 2.5]
X^T X = [[1,0],[0,1]]. Adding lambda*I gives [[2,0],[0,2]]. Inverse is [[0.5,0],[0,0.5]]. X^T y = [3,5]. Result: [1.5, 2.5]. The weights are shrunk from the OLS solution [3, 5] toward zero.
Input:
X = [[1, 1], [1, 2], [1, 3]], y = [3, 5, 7], lam = 0.0
Output:
[1.0, 2.0]
With lambda = 0, ridge regression reduces to ordinary least squares. The perfect fit y = 1 + 2x gives weights [1, 2].
Build a d x d identity matrix I = [[1 if i==j else 0 for j in range(d)] for i in range(d)]. Scale it by lambda and add to X^T X. Then invert and multiply by X^T y, just like ordinary least squares.
The only difference from ordinary least squares is adding lambda * I to X^T X before inverting. This one extra step provides regularization.
Sign in to take notes on this problem
Accepts: array
Accepts: array
Accepts: number