Home / Study together / Weeks 1-4 together

MLF + MLT, weeks 1-4, learned together

One braided path through both courses, so each idea arrives just before you need it.

Weeks 1-4 together Weeks 5-8 together Weeks 1-8 together Weeks 1-12 together

Why learn them together?

Think of it like this

Imagine two friends building a treehouse. One (MLF) makes the tools: saws, hammers, rulers. The other (MLT) builds the treehouse and keeps shouting, "Now I need a saw!" If the second friend starts before the saw exists, they get stuck. If the first friend makes every tool before anyone builds anything, you forget why you made them.

So we braid: make a tool, use it straight away, make the next tool, use that. Weeks 1 to 4 of the two courses fit together almost perfectly this way.

Here is the dependency map. An arrow means "you need the left idea to understand the right one".

MLF idea (the tool)	When MLT uses it (the treehouse)
Week 1: vectors, length, distance, dot product	MLT W1: data points are vectors; W3: k-means uses distance; W2: kernels are dot products
Week 2: derivative = 0 at the best point	MLT W3: the k-means centre is the mean; W4: MLE; Bayes
Week 3: projection ("closest point = perpendicular drop")	MLT W1: PCA's reconstruction is a projection
Week 4: eigenvalues and eigenvectors	MLT W1: principal components are eigenvectors; W2: kernel PCA
(later) MLF W5 symmetric/PSD, W6 SVD	deepens MLT W1 (covariance is symmetric PSD; PCA = SVD) and W2 (valid kernels are PSD)
(later) MLF W10 to W12 probability, MLE, EM	MLT W4 (MLE, Bayes, GMM, EM) uses them; this page gives you a first taste

Tip

Quiz 1 for both courses covers weeks 1 to 4. MLF Quiz 1 papers test the intro, calculus, least squares/projection and eigenvalues. MLT Quiz 1 papers test PCA, kernel PCA, k-means, MLE, Beta-Bernoulli and GMM/EM. If you follow this page, you are preparing for both Quiz 1s at once.

The braided plan

Nine sessions. Each one is a lesson on this site; times are rough. Do them in this order.

#	Session	Read	Why now	Time
1	What is ML, and vectors	MLF week 1	everything is built from vectors and dot products	60 to 90 min
2	The setup of PCA (stop after the covariance matrix)	MLT week 1, up to "Idea 1: The setup"	you now see why we want projections and eigenvectors, so the next two sessions have a purpose	40 min
3	Calculus	MLF week 2	derivative = 0 is needed for k-means and MLE	70 to 90 min
4	Matrices, projection, least squares	MLF week 3	projection is half of PCA	90 min
5	Eigenvalues and eigenvectors	MLF week 4	eigenvectors are the other half of PCA	80 min
6	PCA, finished	MLT week 1	the payoff: it should all click	80 min
7	Kernel PCA	MLT week 2	PCA only uses dot products, so swap them for kernels	80 min
8	k-means	MLT week 3	uses distance (S1) and "mean minimises squared distance" (S3)	80 min
9	MLE, Bayes, GMM and EM	MLT week 4	uses derivative = 0 (S3); gives the first taste of MLF's probability weeks	110 min

Two tips: after each session do its "Your turn" problems and the quick quiz; and tick "Mark this week as done" at the bottom of each lesson so the home page keeps your progress.

Four braided problems (solve them, then check)

Each of these is one problem seen through both courses. Try to answer before opening the solution.

Your turn: 1. A shadow is a reconstruction (MLF week 3 meets MLT week 1)

In MLT week 1 the data points $(0,-1),\,(3,5),\,(6,2)$ were centred and had first principal component $\mathbf w=\frac1{\sqrt2}(1,1)$ . (a) Write the projection matrix $P$ onto that line (MLF week 3). (b) Use $P$ to reconstruct the centred point $(0,3)$ and find the reconstruction error. (c) Say why this equals the squared distance from $(0,3)$ to the line $y=x$ .

Need a hint?

$P=\mathbf w\mathbf w^T$ since $\|\mathbf w\|=1$ .

Show the full solution

(a) $P=\frac12\begin{pmatrix}1&1\\1&1\end{pmatrix}$ . Check: $P^2=P$ , $P^T=P$ , trace $=1$ (rank 1). (b) $P\,(0,3)^T=\frac12(3,3)=(1.5,\,1.5)$ . Error vector $(0,3)-(1.5,1.5)=(-1.5,\,1.5)$ , squared length $2.25+2.25=\mathbf{4.5}$ . (c) The reconstruction is the projection onto the line, i.e. the closest point on the line, so the error is the squared perpendicular distance from the point to the line. (The other centred points: $(-3,-3)$ lies on the line so its error is 0, and $(3,0)$ has error $4.5$ by symmetry. The average $\frac{0+4.5+4.5}3=3$ equals the discarded eigenvalue $\lambda_2=3$ .)

Your turn: 2. Eigenvalues are variances (MLF week 4 meets MLT week 1)

The covariance matrix is $C=\begin{pmatrix}6&3\\3&6\end{pmatrix}$ . Without solving a quadratic, use the trace and determinant facts from MLF week 4 to find its eigenvalues, then give the variance explained by PC1.

Need a hint?

Sum of eigenvalues = trace; product = determinant.

Show the full solution

Trace $=12$ , determinant $=36-9=27$ . We need two numbers with sum $12$ and product $27$ : $9$ and $3$ . So $\lambda_1=9$ and $\lambda_2=3$ . Variance explained by PC1 $=\frac9{12}=\mathbf{75\%}$ . Note the trace is the total variance: $12$ , the sum of the two feature variances $6+6$ . (Eigenvector for 9: $(C-9I)\mathbf v=0\Rightarrow\mathbf v=(1,1)$ .)

Your turn: 3. Derivative = 0 is everywhere (MLF week 2 meets MLT weeks 3 and 4)

(a) For the numbers $1,\,2,\,3$ find the value $m$ that minimises $\sum(x_i-m)^2$ . (b) Why is that the k-means centre update? (c) A coin shows 7 heads and 3 tails; find the $p$ that maximises $p^7(1-p)^3$ .

Need a hint?

Differentiate and set to zero in both.

Show the full solution

(a) $\frac{d}{dm}\sum(x_i-m)^2=-2\sum(x_i-m)=0\Rightarrow m=\bar x=\mathbf2$ . (b) In k-means, for fixed cluster members the best centre minimises the sum of squared distances, which by (a) is the mean of the members; so the update step moves each centre to the mean. (c) Maximise $\ell=7\ln p+3\ln(1-p)$ : $\ell'=\frac7p-\frac3{1-p}=0\Rightarrow7(1-p)=3p\Rightarrow p=\mathbf{0.7}$ , the MLE. The same move, "set the derivative to zero", solved three different problems.

Your turn: 4. A dot product wearing a hat (MLF week 1 meets MLT week 2)

Take $\mathbf x=(1,2)$ and $\mathbf y=(3,1)$ . (a) Compute the dot product (MLF week 1). (b) Compute the polynomial kernel $k(\mathbf x,\mathbf y)=(\mathbf x\cdot\mathbf y)^2$ . (c) Verify it with the feature map $\phi(\mathbf x)=(x_1^2,\sqrt2x_1x_2,x_2^2)$ .

Need a hint?

A kernel is a dot product after lifting.

Show the full solution

(a) $\mathbf x\cdot\mathbf y=3+2=5$ . (b) $k=5^2=25$ . (c) $\phi(\mathbf x)=(1,\,2\sqrt2,\,4)$ , $\phi(\mathbf y)=(9,\,3\sqrt2,\,1)$ : dot product $9+12+4=25$ ✓. So the kernel is exactly the week-1 dot product, taken in a bigger space, without ever computing $\phi$ .

What each MLF idea becomes in MLT (cheat-sheet)

MLF idea	New name in MLT	One-line check
Distance $\\|\mathbf a-\mathbf b\\|$	k-means assignment; Euclidean distance	nearest centre = smallest squared distance
Dot product	kernel $k(\mathbf x,\mathbf y)$	Gram matrix must be PSD
Derivative $=0$	MLE; k-means centroid = mean; ridge/least-squares solutions	set $\partial/\partial\theta=0$
Projection matrix $P=\mathbf w\mathbf w^T$	PCA reconstruction $WW^T\mathbf x$	error $=$ discarded eigenvalues
Eigenvalues of a symmetric matrix	variances of the principal components	trace $=$ total variance
Rank	number of non-zero PCA eigenvalues	data on a line $\Rightarrow$ one non-zero
Least squares $X^TX\mathbf w=X^T\mathbf y$	linear regression (MLT week 5)	residual $\perp$ columns

Short on time?

The idea

3-day crash version (Quiz 1 only): Day 1: MLF W2 ("Idea 3" and "Idea 4" only: gradient, linear approximation, directional derivative) + MLF W3 (projection and least squares). Day 2: MLF W4 + MLT W1. Day 3: MLT W3 (k-means) + MLT W4 (Beta posterior, EM step). Finish each day with that week's quick quiz and real exam questions.

7-day version: follow the nine sessions, two per day, with the real-question pages on Days 5 to 7.

When you finish a lesson, open Practise with real exam questions at its bottom. When you finish all nine, go to the Past papers and sit one full Quiz 1 paper per subject under a timer.

Next: weeks 5 to 8 together, weeks 1 to 8 together or weeks 1 to 12 together. All four plans are listed on the Study together page.