Home / Study together / Weeks 5-8 together

MLF + MLT + MLP: weeks 5-8 together

Weeks 5 to 8 (with a week-4 bridge): MLF tools, MLT algorithms and MLP code in the best order.

Weeks 1-4 together Weeks 5-8 together Weeks 1-8 together Weeks 1-12 together

The big idea of weeks 5 to 8

Think of it like this

In weeks 1 to 4 you built the tools (vectors, derivatives, projection, eigenvectors) and the first unsupervised algorithms. Weeks 5 to 8 are the second chapter: now we have labels, and we learn to predict. The same tools come back wearing new names:

least squares is a projection (MLF 3) solved with a symmetric matrix (MLF 5);
ridge is that same solution made safe, and SVD (MLF 6) shows exactly how it shrinks;
gradient descent (MLF 7) is how a computer walks to the answer, and convexity (MLF 8) is why it always arrives;
k-NN, trees and Naive Bayes (MLT 7, 8) are the first classifiers, and Naive Bayes needs Bayes' rule from MLF's probability weeks.

MLP then makes you type all of it in scikit-learn.

Tip

Where this sits in the exams. Quiz 2 is taken to cover weeks 5 to 8 in each of the three courses (the official pages do not print the quiz boundary, so this split is inferred from the question topics). Do the lessons in the braided order below and you prepare all three Quiz 2 papers at once. Week 4 (eigenvalues, EM) is the bridge: skim MLF week 4 first if eigenvalues feel rusty.

Dependency map (the tool, and where it is used next)

MLF idea (the tool)	Where weeks 5 to 8 of MLT and MLP use it
W3 projection, normal equations $X^TX\mathbf w=X^T\mathbf y$	MLT W5 least squares is this; MLP W3 `LinearRegression` solves it
W5 symmetric matrices, real eigenvalues, PSD	$X^TX$ is symmetric PSD, so least squares has a solid solution; MLT W6 adds $\lambda I$ to push every eigenvalue up
W6 SVD and low-rank	MLT W6 shows ridge shrinks each SVD direction by $\sigma^2/(\sigma^2+\lambda)$ ; MLP W4 `Ridge`/`Lasso`
W7 gradient descent	MLP W3 `SGDRegressor` (batch and stochastic); MLP W5 logistic regression trained by GD
W8 convexity	least squares and ridge are convex, so one minimum and GD cannot get stuck; LASSO is convex but has a corner (MLT W6)
W10 Bayes rule (borrowed early)	MLT W8 Naive Bayes; MLP W6 probabilities and thresholds
MLT W7 k-NN and trees	MLP W9 `KNeighbors...`, `DecisionTree...` (the code comes later)

The braided plan

Sixteen sessions. The MLP sessions are short: run the code and compare with the hand calculation from the session before.

#	Session	Read	Why now	Time
1	Symmetric matrices	MLF week 5	$X^TX$ and covariance matrices are symmetric; this explains why they behave	80 min
2	SVD and PCA	MLF week 6	one tool that explains PCA, ridge and low-rank	90 min
3	Least squares as regression	MLT week 5	MLF 3's projection, now with data and noise, plus its Bayesian reading	90 min
4	Linear regression in code	MLP week 3 (first half)	type what you just solved by hand	45 min
5	Gradient descent	MLF week 7	the computer's way to the same answer	80 min
6	GD and SGD in sklearn	MLP week 3 (second half)	`eta0`, `learning_rate`, batch vs stochastic	45 min
7	Convexity	MLF week 8	tells you when GD is guaranteed to win	80 min
8	Ridge and LASSO	MLT week 6	uses sessions 1, 2, 5 and 7 together	90 min
9	Polynomial and regularised models	MLP week 4	`PolynomialFeatures`, `Ridge`, `Lasso`, direction of `alpha`	50 min
10	Bayes rule, borrowed early	MLF week 10 (conditional probability and Bayes only)	Naive Bayes needs exactly this	40 min
11	k-NN and decision trees	MLT week 7	distance (MLF 1) as a classifier; entropy as a splitting rule	80 min
12	Naive Bayes	MLT week 8	Bayes rule plus an independence assumption	80 min
13	Logistic regression in code	MLP week 5	`LogisticRegression`, `C`, probabilities	45 min
14	Binary classifier metrics	MLP week 6	confusion matrix, precision, recall, PR curve	60 min
15	Multiclass	MLP week 7	OvR needs $K$ classifiers, OvO needs $K(K-1)/2$	45 min
16	SVMs in sklearn	MLP week 8	`SVC`, `kernel`, `C`, `gamma` (the theory is MLT week 10, later)	50 min

Do each lesson's quick quiz and real exam questions straight after. Tick Mark this week as done so the home page tracks you.

Five braided problems (solve them, then check)

Your turn: 1. Least squares, then a safer version (MLF 3 and 5 meet MLT 5 and 6)

Data: $x=1,2,3$ and $y=2,4,7$ , one feature, no intercept. (a) Find the least-squares weight $w$ . (b) Find the ridge weight with $\lambda=1$ using $w=(X^TX+\lambda I)^{-1}X^T\mathbf y$ . (c) Which is smaller, and why?

Need a hint?

Here $X^TX$ is a single number, $\sum x^2$ .

Show the full solution

$X^TX=1+4+9=14$ and $X^T\mathbf y=2+8+21=31$ . (a) $w=\frac{31}{14}\approx\mathbf{2.214}$ . (b) $w=\frac{31}{14+1}=\frac{31}{15}\approx\mathbf{2.067}$ . (c) The ridge weight is smaller: adding $\lambda$ to the denominator shrinks the weight towards zero. That is the whole idea of ridge: a slightly worse fit on the training data, in return for a steadier weight.

Your turn: 2. Why ridge never fails to invert (MLF 5 meets MLT 6)

Two features are exact copies, so $X^TX=\begin{pmatrix}3&2\\2&3\end{pmatrix}$ is replaced in a bad case by $A=\begin{pmatrix}2&2\\2&2\end{pmatrix}$ . (a) Is $A$ invertible? Give its eigenvalues. (b) Find the eigenvalues and determinant of $A+\lambda I$ for $\lambda=1$ .

Need a hint?

For a symmetric matrix the eigenvalues add $\lambda$ when you add $\lambda I$ .

Show the full solution

(a) $\det A=4-4=0$ , so $A$ is not invertible. Trace $=4$ and determinant $=0$ give eigenvalues $\mathbf{4}$ and $\mathbf{0}$ . (b) $A+I=\begin{pmatrix}3&2\\2&3\end{pmatrix}$ with determinant $9-4=\mathbf5\neq0$ . Its eigenvalues are $4+1=5$ and $0+1=1$ (product $5$ ✓). Adding $\lambda I$ lifts every eigenvalue by $\lambda$ , so none can be $0$ . That is why ridge works even when features are copies of each other.

Your turn: 3. How much does ridge shrink each direction? (MLF 6 meets MLT 6)

In the SVD of $X$ , ridge multiplies the least-squares answer along a direction with singular value $\sigma$ by $\dfrac{\sigma^2}{\sigma^2+\lambda}$ . For $\lambda=1$ , compute the factor for $\sigma=3$ and for $\sigma=1$ . Which direction is shrunk more?

Need a hint?

Plug in. Compare with 1 (no shrinkage).

Show the full solution

$\sigma=3$ : $\frac{9}{9+1}=\mathbf{0.9}$ . $\sigma=1$ : $\frac{1}{1+1}=\mathbf{0.5}$ . The small-singular-value direction is shrunk far more. Small $\sigma$ means the data barely varies in that direction, so the weight there is the least trustworthy: ridge quietly distrusts it. Least squares has factor $1$ everywhere.

Your turn: 4. Gradient descent on the same problem (MLF 7 and 8 meet MLP 3)

For the data of Problem 1, the loss is $L(w)=\sum(wx_i-y_i)^2=14w^2-62w+69$ . (a) Find $L'(w)$ and $L''(w)$ and say why GD is guaranteed to find the minimum. (b) Do one GD step from $w_0=0$ with $\eta=0.01$ . (c) What is the largest $\eta$ that still converges?

Need a hint?

$L''>0$ everywhere means convex.

Show the full solution

(a) $L'(w)=28w-62$ , $L''(w)=28>0$ everywhere, so $L$ is convex with one minimum, at $L'=0$ : $w=\frac{62}{28}=\frac{31}{14}$ , the same answer as before. (b) $w_1=0-0.01\times(28\cdot0-62)=\mathbf{0.62}$ . (c) For a quadratic with curvature $c=28$ , GD converges when $\eta<\frac2c=\frac2{28}\approx\mathbf{0.0714}$ . In scikit-learn this is exactly why too large an eta0 makes SGDRegressor blow up.

Your turn: 5. Bayes rule in one line (MLF 10 meets MLT 8)

$P(\text{spam})=0.4$ . The word "free" appears in $50\%$ of spam and $10\%$ of non-spam. An email contains "free". Find $P(\text{spam}\mid\text{free})$ .

Need a hint?

Bayes: posterior $=$ likelihood $\times$ prior, divided by the total.

Show the full solution

Numerator: $0.5\times0.4=0.20$ . Other branch: $0.1\times0.6=0.06$ . So $P=\frac{0.20}{0.20+0.06}=\frac{0.20}{0.26}\approx\mathbf{0.769}$ . Naive Bayes does this once per word and multiplies the likelihoods, pretending the words are independent.

What the weeks become in each other (cheat-sheet)

Idea	In MLF	In MLT	In MLP
Normal equations	W3 projection	W5 least squares	`LinearRegression`
Adding $\lambda I$	W5 eigenvalue shift	W6 ridge	`Ridge(alpha=...)`
Sparse solutions	W8 convex, non-smooth	W6 LASSO	`Lasso(alpha=...)`
Walk downhill	W7 gradient descent	W5 to W6 iterative solvers	`SGDRegressor(eta0, learning_rate)`
Probability	W10 Bayes	W8 Naive Bayes	`GaussianNB`, `predict_proba`
Many classes	-	W9 to W10	W7 OvR $K$ , OvO $K(K-1)/2$

The idea

Counting skills that pay every time in Quiz 2. Number of fits in a grid search $=$ combinations $\times$ folds ( $+1$ refit). OvR trains $K$ models, OvO trains $K(K-1)/2$ . Larger ridge/LASSO penalty $\Rightarrow$ smaller weights: alpha up, C down is the stronger regularisation. Learn these as three one-line rules.

Short on time?

The idea

4-day crash version (Quiz 2 only). Day 1: MLF 5 and 6 (focus on PSD and SVD facts). Day 2: MLT 5 and 6 plus MLP 3 and 4 (the ridge story). Day 3: MLF 7 and 8 plus MLP 5 and 6 (GD, convexity, metrics). Day 4: MLT 7 and 8 plus MLP 7 and 8 (classifiers). Finish every day with that week's real exam questions.

After the sessions, sit one full Quiz 2 paper per course from Past papers under a timer. Then continue to weeks 1 to 8 together or weeks 1 to 12 together.