Home / MLT / Week 6

Week 6 · Ridge and LASSO: taming overfitting

Official syllabus line: Supervised Learning - Regression - Ridge/LASSO

About 70 minQuiz 2 material (see exam radar)BSCS2007 · Arun Rajkumar

Before you start: MLT week 5 (linear regression), MLF week 8 (convex functions help but are not required).

Watch first: the animated story of this weekAbout 1-2 minutes of real 3D animation with a friendly voice. Pause any time.

Read the words of the video

Imagine a very eager student changing their answers to fit every little detail of practice questions, even noise. They fit practice data beautifully but fail on new questions. Our computer models can do this too, learning too much noise. We give the model a 'leash' to keep its answers sensible.
We add a penalty for large 'weights' – the numbers the model learns. Ridge regression uses a soft leash: it gently pulls all weights smaller, but never to zero. LASSO regression uses a sharp leash: it can pull some weights exactly to zero, turning off unhelpful features.
Let's see an example. If a model's best 'weight' for a feature is 3 without a penalty. Ridge, with a penalty, will shrink it, maybe to 1 or 0.3, but always keeps it positive. LASSO, however, with a strong enough penalty, can make that weight exactly zero, completely removing that feature.
For your exam, remember that Ridge never makes weights exactly zero; LASSO can. Both add penalties to stop models from learning noise and overfitting. We choose the right 'leash strength', called lambda, by finding what works best on new data. Great job learning this complex idea!

The one-minute story

Think of it like this

A model with too much freedom is like a very eager student who changes their answer to fit every little detail of the practice questions, including noise. Their weights (the numbers the model learns) swing wildly: +900 here, −870 there, cancelling each other out. It fits the practice data beautifully and falls apart on a new question.

The cure is a leash on the weights. We add a penalty for large weights to the thing we are minimising. Now the model must be useful and modest.

Ridge puts a soft leash: every weight is pulled gently towards zero, but none is ever cut loose entirely.
LASSO puts a sharp leash: it can pull some weights exactly to zero, which switches those features off. It chooses which features matter.

Same idea, two flavours, two different personalities.

By the end of this week you can:

explain overfitting and why large weights are a warning sign;
derive and use the ridge solution $(X^TX+\lambda I)^{-1}X^T\mathbf y$ ;
explain LASSO and why it gives sparse weights;
interpret both as MAP estimation (Gaussian prior vs Laplace prior);
pick $\lambda$ by cross-validation and reason about bias and variance.

Words you need

Regularisation - adding a penalty to discourage complicated models.
$\lambda$ - the penalty strength. $\lambda=0$ means no penalty.
$\ell_2$ penalty $\|\mathbf w\|^2=\sum w_j^2$ (ridge). $\ell_1$ penalty $\|\mathbf w\|_1=\sum|w_j|$ (LASSO).
Shrinkage - making the weights smaller than the least-squares ones.
Sparse - mostly zeros.
Bias - systematic error from a model being too simple. Variance - sensitivity of the model to the particular training sample.

Idea 1: Why penalise?

If two features are almost copies of each other, least squares can give them huge opposite weights that nearly cancel. If $X^TX$ is singular (columns dependent) it is even worse: there are infinitely many equally good solutions. Penalising the size of the weights fixes both problems: it prefers small, stable weights.

Idea 2: Ridge regression ( $\ell_2$ penalty)

Minimise fit + penalty:

\mathcal L_{\text{ridge}}(\mathbf w)=\|\mathbf y-X\mathbf w\|^2+\lambda\|\mathbf w\|^2

Take the gradient and set it to zero: $-2X^T(\mathbf y-X\mathbf w)+2\lambda\mathbf w=0$ , giving

\boxed{\mathbf w_{\text{ridge}}=(X^TX+\lambda I)^{-1}X^T\mathbf y}

Things to notice:

Adding $\lambda I$ adds $\lambda$ to every eigenvalue of $X^TX$ , so for any $\lambda>0$ the matrix is invertible, even if $X^TX$ was singular. Ridge always has a unique answer.
$\lambda\to0$ : back to least squares. $\lambda\to\infty$ : $\mathbf w\to\mathbf 0$ .
The objective is strictly convex (MLF week 8) so there is exactly one minimum.
In the simplest case of one feature, $\mathbf w_{\text{ridge}}=\dfrac{\sum x_iy_i}{\sum x_i^2+\lambda}$ : the penalty just inflates the denominator, shrinking $w$ .

Solved example: 1. Ridge in one dimension

Question. Data $x=(1,2,3)$ , $y=(2,3,5)$ . Fit $y=wx$ by ridge with $\lambda=0,\;1,\;14$ .

Solution. $\sum xy=23$ and $\sum x^2=14$ , so $w=\dfrac{23}{14+\lambda}$ .

$\lambda$	$w$
0 (plain least squares)	$\frac{23}{14}=1.643$
1	$\frac{23}{15}=1.533$
14	$\frac{23}{28}=0.821$

As $\lambda$ grows the slope is pulled towards zero, but never reaches it.

Solved example: 2. Ridge rescues a singular problem

Question. Two features are exact copies: $X=\begin{pmatrix}1&1\\2&2\\3&3\end{pmatrix}$ and $\mathbf y=(2,4,6)^T$ . Least squares has no unique answer. What does ridge with $\lambda=1$ give?

Solution. $X^TX=\begin{pmatrix}14&14\\14&14\end{pmatrix}$ is singular (determinant 0): any $w_1+w_2=2$ fits perfectly. With ridge: $X^TX+I=\begin{pmatrix}15&14\\14&15\end{pmatrix}$ , invertible, and $X^T\mathbf y=(28,28)^T$ . By symmetry $w_1=w_2=w$ and $15w+14w=28$ , so $w=\frac{28}{29}\approx0.966$ for each. Ridge splits the credit equally between the twin features, and shrinks the total slightly (from $2$ to $1.93$ ).

Idea 3: LASSO ( $\ell_1$ penalty)

\mathcal L_{\text{lasso}}(\mathbf w)=\|\mathbf y-X\mathbf w\|^2+\lambda\|\mathbf w\|_1

The absolute value has a sharp corner at 0 so the objective is not differentiable there, and there is no closed form in general (we use iterative methods). But it has a famous property: it sets some weights exactly to 0, a built-in feature selector.

For one feature whose unpenalised best weight is $z$ , minimising $(w-z)^2+\lambda|w|$ gives the soft-threshold:

w_{\text{lasso}}=\operatorname{sign}(z)\,\max\!\Big(|z|-\frac\lambda2,\;0\Big)

and the ridge version of the same problem, $(w-z)^2+\lambda w^2$ , gives $w_{\text{ridge}}=\dfrac z{1+\lambda}$ .

Solved example: 3. Ridge vs LASSO on one number

Question. Unpenalised best $z=3$ . Compare ridge and LASSO for $\lambda=2$ and $\lambda=8$ .

Solution. Ridge: $\frac3{1+2}=1$ and $\frac3{1+8}=0.333$ . LASSO: $\max(3-1,0)=2$ and $\max(3-4,0)=\mathbf0$ .

For $\lambda=8$ , LASSO has set the weight to exactly zero while ridge still keeps a small non-zero value.

The geometry (why LASSO makes zeros)

Equivalent view: minimise the squared error subject to a budget on the weights: $\|\mathbf w\|_2^2\le c$ (ridge) or $\|\mathbf w\|_1\le c$ (LASSO). The error has elliptical contour lines. The ridge budget region is a round disc; the LASSO region is a diamond with corners on the axes. The first contour to touch a smooth disc touches it anywhere; but it very often first touches the diamond at a corner, and a corner sits on an axis, where one weight is exactly 0.

Idea 4: The Bayesian meaning

From MLT week 5: Gaussian noise gives the squared-error fit term. Add a prior on the weights:

Prior on $\mathbf w$	MAP objective	Method
Gaussian $\mathcal N(0,\tau^2)$	squared error $+\lambda\\|\mathbf w\\|_2^2$ , $\lambda=\sigma^2/\tau^2$	Ridge
Laplace $\propto e^{-\|w\|/b}$	squared error $+\lambda\\|\mathbf w\\|_1$	LASSO

A Laplace density has a sharp spike at 0, so it "believes in zeros", which explains the sparsity. A tighter prior (small $\tau$ ) means a bigger $\lambda$ .

Idea 5: Choosing $\lambda$ and the bias-variance trade-off

Small $\lambda$ : flexible, low bias, high variance (overfits).
Large $\lambda$ : stiff, high bias, low variance (underfits).
Training error rises with $\lambda$ ; validation error is U-shaped: best somewhere in the middle. Pick $\lambda$ by cross-validation.
Standardise the features first (the penalty treats all weights equally, so features on different scales would be punished unfairly), and normally do not penalise the intercept.
Elastic net mixes both penalties.

Your turn

Your turn: 1. Ridge in one dimension

Data $x=(1,2)$ , $y=(3,5)$ . Compute the ridge weight for $y=wx$ with $\lambda=5$ (check the $\lambda=0$ case too).

Need a hint?

$w=\sum xy/(\sum x^2+\lambda)$ .

Show the full solution

$\sum xy=3+10=13$ , $\sum x^2=1+4=5$ . $\lambda=0$ : $w=2.6$ . $\lambda=5$ : $w=\frac{13}{10}=\mathbf{1.3}$ .

Your turn: 2. Soft-threshold

The unpenalised best weights for three features are $4,\,1.5,\,-3$ . What are the LASSO weights (per-feature problem) with $\lambda=4$ ? And ridge with $\lambda=1$ ?

Need a hint?

Soft-threshold at $\lambda/2=2$ .

Show the full solution

LASSO: $4\to2$ ; $1.5\to0$ (since $1.5<2$ ); $-3\to-1$ . Ridge ( $\div2$ ): $2,\;0.75,\;-1.5$ . LASSO zeroed the middle feature; ridge did not.

Your turn: 3. Direction of the effect

For each, say whether it goes up or down as $\lambda$ increases: (a) training error (b) the size of the weights (c) variance of the predictions (d) bias.

Need a hint?

More penalty means a stiffer model.

Show the full solution

(a) up, (b) down, (c) down, (d) up.

Your turn: 4. Which statements are true?

(a) Ridge always gives a unique solution for $\lambda>0$ . (b) LASSO can set weights exactly to zero. (c) Ridge is MAP with a Laplace prior. (d) Ridge has a closed-form solution.

Need a hint?

Review the table in Idea 4.

Show the full solution

(a) True. (b) True. (c) False: ridge is a Gaussian prior; LASSO is Laplace. (d) True: $(X^TX+\lambda I)^{-1}X^T\mathbf y$ .

Your turn: 5. Choosing lambda

Validation errors for $\lambda=0,\,0.1,\,1,\,10,\,100$ are $9.0,\,5.1,\,3.2,\,4.0,\,8.5$ . Which $\lambda$ would you choose and why?

Need a hint?

Smallest validation error.

Show the full solution

$\lambda=1$ (error $3.2$ ). The curve is U-shaped: smaller $\lambda$ overfits, larger underfits.

Exam radar

MLT Quiz 2 summaries mention ridge in 7 papers. In the End Term: ridge regression in 20 papers (and "Ridge Regression (L2 Regularization)" in 13), overfitting in 19, underfitting in 15. LASSO is named in the course syllabus, so expect it too.

Typical: "which of the following is the ridge solution?", "what happens to the weights as $\lambda\to\infty$ ?", "which regulariser produces sparse weights?", "ridge as MAP with which prior?", and a one-dimensional ridge calculation.

Watch out - real exam trap

Ridge never gives exact zeros; LASSO can.
Ridge penalty is on squared weights; the intercept is usually not penalised.
Bigger $\lambda$ means more bias, less variance.
Gaussian prior $\leftrightarrow$ ridge; Laplace prior $\leftrightarrow$ LASSO (easy to swap by accident).
Features must be on comparable scales before regularising.

Quick quiz

The ridge regression solution is...

(X^T X)^-1 X^T y
(X^T X + lambda I)^-1 X^T y
(X X^T)^-1 y
X^T y / lambda

Ridge adds lambda times the identity to X^T X before inverting.

As lambda grows very large, ridge weights tend to...

infinity
the least-squares weights
zero
one

The penalty dominates and pulls all weights towards zero.

Which penalty tends to produce exactly-zero weights?

l2 (ridge)
l1 (LASSO)
no penalty
none of them

The l1 penalty has a corner at zero that can trap weights there.

Select all TRUE statements. (select all that apply)

Ridge is MAP estimation with a Gaussian prior on the weights.
Increasing lambda increases the bias of the model.
Ridge regression can make a singular X^T X problem worse.
LASSO has a simple closed-form solution for every data set.

Adding lambda I makes the matrix invertible, and LASSO generally needs an iterative solver.

Recap

Remember this

Ridge: $\|\mathbf y-X\mathbf w\|^2+\lambda\|\mathbf w\|^2\Rightarrow\mathbf w=(X^TX+\lambda I)^{-1}X^T\mathbf y$ . Always invertible; shrinks smoothly; Gaussian prior.
LASSO: $\|\mathbf y-X\mathbf w\|^2+\lambda\|\mathbf w\|_1$ . Sparse weights; no closed form; Laplace prior; soft-threshold in 1-D.
$\lambda\uparrow$ : bias $\uparrow$ , variance $\downarrow$ , training error $\uparrow$ . Choose $\lambda$ by cross-validation; standardise features.

Formulas to keep

Ridge: $\mathbf w=(X^TX+\lambda I)^{-1}X^T\mathbf y$ ; 1-D: $\frac{\sum xy}{\sum x^2+\lambda}$ ; per-feature shrink $\frac z{1+\lambda}$
LASSO soft-threshold: $\operatorname{sign}(z)\max(|z|-\lambda/2,0)$
MAP: Gaussian prior $\to\lambda=\sigma^2/\tau^2$ (ridge); Laplace prior $\to$ LASSO

Connected weeks (learn them together)

MLF week 8 - Bowls, not bumpy hills: convexity: Ridge and LASSO are convex problems, so a global best exists and can be found.
MLP week 4 - Polynomial regression and regularisation: Ridge and LASSO: the theory, then the code.

Practise with real exam questions

These come straight from past papers (with the real figures and the official answer key). Open 18 real questions on this week All Quiz 2 papers

Week 6 · Ridge and LASSO: taming overfitting

The one-minute story

Words you need

Idea 1: Why penalise?

Idea 2: Ridge regression (ℓ2\ell_2ℓ2​ penalty)

Idea 3: LASSO (ℓ1\ell_1ℓ1​ penalty)

The geometry (why LASSO makes zeros)

Idea 4: The Bayesian meaning

Idea 5: Choosing λ\lambdaλ and the bias-variance trade-off

Your turn

Exam radar

Quick quiz

Recap

Practise with real exam questions

Idea 2: Ridge regression ( $\ell_2$ penalty)

Idea 3: LASSO ( $\ell_1$ penalty)

Idea 5: Choosing $\lambda$ and the bias-variance trade-off