Home / MLF / Week 4

Week 4 · The special directions of a matrix

Official syllabus line: Linear Algebra - Eigenvalues and eigenvectors

About 80 minQuiz 1 material (see exam radar)BSCS2004 · Harish Guruprasad Ramaswamy, Arun Rajkumar, Prashanth LA

Before you start: Week 3 (matrices as machines, rank). Solving a quadratic equation

x^2+bx+c=0

Watch first: the animated story of this weekAbout 1-2 minutes of real 3D animation with a friendly voice. Pause any time.

Read the words of the video

Imagine stretching a rubber sheet with arrows drawn on it. Most arrows twist and turn, pointing to new places after the stretch.
But a few special arrows only get longer or shorter, still pointing exactly the same way. These are called eigenvectors.
How much each special arrow grows or shrinks along its path is its eigenvalue. This number tells us the exact stretch factor.
Mathematically, applying a matrix 'A' to an eigenvector 'v' just scales 'v' by a number, lambda. The direction stays the same!
To find these special stretch numbers, we solve an equation: the determinant of the matrix minus lambda times the identity matrix must be zero.
For a two-by-two matrix, there's a neat shortcut! The sum of the eigenvalues is the trace, and their product equals the determinant.
For example, if the sum of two numbers is seven and their product is ten, the numbers are easily found as two and five by a simple quadratic equation.
If you know a matrix's eigenvalues, you can quickly find eigenvalues for its inverse, powers, or when a constant is added.
If a matrix has a zero eigenvalue, it squashes space flat and cannot be reversed or 'un-squashed.' It is not invertible.
When we have enough independent eigenvectors, we can rewrite the matrix in a special way to easily calculate high powers, saving huge effort.
Beware of common traps! Inverse eigenvalues are reciprocals, not negatives. Also, a repeated eigenvalue does not always mean a matrix cannot be diagonalized.
So, eigenvectors show special directions, and eigenvalues show how much things stretch. Together, they simplify complex matrix operations and transformations.

The one-minute story

Think of it like this

Take a rubber sheet with a grid drawn on it and stretch it: pull it wider in one direction, squeeze it in another, maybe shear it. Now draw many arrows on the sheet before the stretch.

After the stretch, almost every arrow has been turned to point somewhere new. But a few special arrows only get longer or shorter, still pointing exactly the way they did (or exactly backwards). Those lucky arrows are the eigenvectors of the stretch. By how much each one grew or shrank is its eigenvalue.

Why care? Because along its special directions a complicated matrix behaves like plain multiplication by a number. Learn the special directions and the matrix stops being scary. Everything in PCA (MLF week 6, MLT week 1) is "find the special directions of the data".

By the end of this week you can:

say what an eigenvalue and eigenvector are, and find them for $2\times2$ matrices;
use the trace and determinant shortcuts to get eigenvalues quickly;
find eigenvalues of $A^{-1}$ , $A^k$ , $A+cI$ and $A^T$ without recomputing;
decide when a matrix is diagonalisable and use $A=PDP^{-1}$ to compute powers.

Words you need

Eigenvector $\mathbf v$ - a non-zero vector whose direction $A$ does not change.
Eigenvalue $\lambda$ - the stretch factor of that eigenvector: $A\mathbf v=\lambda\mathbf v$ .
Characteristic equation - $\det(A-\lambda I)=0$ ; its roots are the eigenvalues.
Trace - the sum of the diagonal entries of a matrix.
Determinant - a single number that says how a matrix scales area (or volume); zero means it squashes space flat.
Diagonal matrix - zeros everywhere except the diagonal. Easy to multiply and raise to powers.
Diagonalisable - can be rewritten as $PDP^{-1}$ with $D$ diagonal.

Idea 1: The definition

A\mathbf v=\lambda\mathbf v,\qquad\mathbf v\neq\mathbf 0

Read it aloud: " $A$ applied to $\mathbf v$ gives the same arrow, just scaled by $\lambda$ ." We insist $\mathbf v\ne\mathbf 0$ because the zero vector satisfies this for every $\lambda$ and tells us nothing.

$\lambda>1$ : stretched. $0<\lambda<1$ : shrunk. $\lambda<0$ : flipped to point the opposite way. $\lambda=0$ : squashed to nothing.

How to find them

Rewrite $A\mathbf v=\lambda\mathbf v$ as $(A-\lambda I)\mathbf v=\mathbf 0$ . A non-zero $\mathbf v$ can only be sent to zero if $A-\lambda I$ squashes space flat, i.e. its determinant is zero:

\det(A-\lambda I)=0

For a $2\times2$ matrix this becomes a short quadratic with two handy facts:

\lambda^2-(\operatorname{trace}A)\,\lambda+\det A=0

The idea

The two free facts (true for every square matrix):

sum of eigenvalues $=$ trace of $A$
product of eigenvalues $=$ determinant of $A$

Use them to find a missing eigenvalue in seconds, or to sanity-check yours.

Solved example: 1. Eigenvalues and eigenvectors of a

2\times2

Question. Find the eigenvalues and eigenvectors of $A=\begin{pmatrix}4&1\\2&3\end{pmatrix}$ .

Solution.

Trace $=4+3=7$ , determinant $=4\cdot3-1\cdot2=10$ . The equation is $\lambda^2-7\lambda+10=0$ , which factorises as $(\lambda-5)(\lambda-2)=0$ . So $\lambda_1=5$ , $\lambda_2=2$ .
For $\lambda=5$ : $A-5I=\begin{pmatrix}-1&1\\2&-2\end{pmatrix}$ . Solving $(A-5I)\mathbf v=0$ : $-v_1+v_2=0$ , so $v_2=v_1$ . Take $\mathbf v_1=(1,1)$ .
For $\lambda=2$ : $A-2I=\begin{pmatrix}2&1\\2&1\end{pmatrix}$ . Then $2v_1+v_2=0$ , so $v_2=-2v_1$ . Take $\mathbf v_2=(1,-2)$ .
Check: $A(1,1)=(5,5)=5(1,1)$ and $A(1,-2)=(2,-4)=2(1,-2)$ .

Any non-zero multiple of an eigenvector is also an eigenvector, so $(2,2)$ works just as well as $(1,1)$ .

Idea 2: Shortcuts that save marks

If $A\mathbf v=\lambda\mathbf v$ , then, with the same eigenvector $\mathbf v$ :

Matrix	Its eigenvalue
$A^k$	$\lambda^k$
$A^{-1}$ (if it exists)	$1/\lambda$
$A+cI$	$\lambda+c$
$cA$	$c\lambda$
$A^T$	the same $\lambda$ (eigenvectors may change)
$p(A)$ for a polynomial $p$	$p(\lambda)$

Two more:

Triangular and diagonal matrices: the eigenvalues are simply the diagonal entries.
$A$ is invertible $\iff$ no eigenvalue is 0 $\iff\det A\ne0$ . Because $\det A=\prod\lambda_i$ , a single zero eigenvalue makes the determinant zero.

Solved example: 2. Quick puzzles that appear again and again

(a) A $2\times2$ matrix has trace 3 and determinant 2. Find its eigenvalues.

Solve $\lambda^2-3\lambda+2=0$ : $(\lambda-1)(\lambda-2)=0$ , so $\lambda=1,\,2$ .

(b) The eigenvalues of a $3\times3$ matrix $A$ are $1,\,\tfrac12,\,\tfrac14$ . Find $\det\big((A^{-1})^T\big)$ .

The eigenvalues of $A^{-1}$ are $1,\,2,\,4$ . Transposing does not change eigenvalues. The determinant is the product: $1\cdot2\cdot4=\mathbf 8$ .

(c) A $3\times3$ matrix has eigenvalues $2,\,3,\,-1$ . Find $\operatorname{trace}(A)$ , $\det(A)$ and the eigenvalues of $A^2-I$ .

Trace $=2+3-1=4$ . Determinant $=2\cdot3\cdot(-1)=-6$ . For $A^2-I$ use $p(\lambda)=\lambda^2-1$ : $3,\;8,\;0$ .

The last one shows $A^2-I$ is singular, because one eigenvalue is 0.

Idea 3: When eigenvalues are complex, repeated, or not enough

Complex eigenvalues. A rotation by $90^\circ$ , $\begin{pmatrix}0&-1\\1&0\end{pmatrix}$ , turns every arrow, so no real arrow stays on its line. Its characteristic equation $\lambda^2+1=0$ gives $\lambda=\pm i$ . Real matrices can have complex eigenvalues; real symmetric matrices cannot (week 5).

Repeated eigenvalues. The number of times $\lambda$ appears as a root is its algebraic multiplicity. The number of independent eigenvectors it has is its geometric multiplicity, which is never bigger.

The idea

Diagonalisable means you can find enough eigenvectors: $n$ independent ones for an $n\times n$ matrix. Then, with $P=[\mathbf v_1\;\cdots\;\mathbf v_n]$ and $D=\operatorname{diag}(\lambda_1,\dots,\lambda_n)$ :

A=PDP^{-1}\qquad\Longrightarrow\qquad A^k=PD^kP^{-1}

Eigenvectors for different eigenvalues are always independent. Trouble only arises with repeated eigenvalues.

Watch out - real exam trap

"The eigenvalue is repeated, so it is not diagonalisable" is wrong. So is "...so it is diagonalisable".

The identity matrix $I$ has eigenvalue 1 twice and is already diagonal. Diagonalisable.
The shear $\begin{pmatrix}1&1\\0&1\end{pmatrix}$ has eigenvalue 1 twice, but $A-I=\begin{pmatrix}0&1\\0&0\end{pmatrix}$ only has one independent eigenvector $(1,0)$ . Not diagonalisable.

A repeated eigenvalue by itself tells you nothing about diagonalisability. Check how many independent eigenvectors it has.

Solved example: 3. Using

A=PDP^{-1}

to find

A^3

Question. For $A=\begin{pmatrix}4&1\\2&3\end{pmatrix}$ from Example 1, compute $A^3$ .

Solution. $P=\begin{pmatrix}1&1\\1&-2\end{pmatrix}$ (columns are the eigenvectors), $D=\begin{pmatrix}5&0\\0&2\end{pmatrix}$ . Then $D^3=\operatorname{diag}(125,\,8)$ and $P^{-1}=\frac13\begin{pmatrix}2&1\\1&-1\end{pmatrix}$ .

A^3=PD^3P^{-1}=\frac13\begin{pmatrix}258&117\\234&141\end{pmatrix}=\begin{pmatrix}86&39\\78&47\end{pmatrix}

Check by brute force: $A^2=\begin{pmatrix}18&7\\14&11\end{pmatrix}$ , and $A^2A=\begin{pmatrix}86&39\\78&47\end{pmatrix}$ . ✓. For $A^{100}$ the shortcut would be effortless while brute force would be hopeless: just replace $125$ and $8$ with $5^{100}$ and $2^{100}$ .

Solved example: 4. A matrix that is not diagonalisable

Question. Is $B=\begin{pmatrix}2&1\\0&2\end{pmatrix}$ diagonalisable?

Solution. It is triangular, so the eigenvalues are the diagonal entries: $2$ and $2$ (repeated). Solve $(B-2I)\mathbf v=0$ : $B-2I=\begin{pmatrix}0&1\\0&0\end{pmatrix}$ gives $v_2=0$ , so the only eigenvectors are multiples of $(1,0)$ . We need two independent eigenvectors and have only one. Not diagonalisable.

Your turn

Your turn: 1. Triangular shortcut

Find the eigenvalues of $\begin{pmatrix}3&1\\0&4\end{pmatrix}$ and of $\begin{pmatrix}7&0&0\\2&-1&0\\5&9&6\end{pmatrix}$ .

Need a hint?

Both are triangular.

Show the full solution

Read off the diagonals: $3,\,4$ and $7,\,-1,\,6$ .

Your turn: 2. Eigen facts

A $3\times3$ matrix $A$ has eigenvalues $2,\,4,\,-1$ . Find: (a) $\det A$ (b) trace $A$ (c) eigenvalues of $A^{-1}$ (d) eigenvalues of $A+3I$ (e) is $A$ invertible?

Need a hint?

Sum and product give trace and determinant. Then apply each function to each eigenvalue.

Show the full solution

(a) $2\cdot4\cdot(-1)=-8$ . (b) $2+4-1=5$ . (c) $\frac12,\,\frac14,\,-1$ . (d) $5,\,7,\,2$ . (e) Yes, because no eigenvalue is 0.

Your turn: 3. A symmetric matrix

Find the eigenvalues and eigenvectors of $S=\begin{pmatrix}1&2\\2&1\end{pmatrix}$ . What do you notice about the eigenvectors?

Need a hint?

Trace 2, determinant $1-4=-3$ .

Show the full solution

$\lambda^2-2\lambda-3=0\Rightarrow(\lambda-3)(\lambda+1)=0$ , so $\lambda=3,\,-1$ . For $3$ : $S-3I=\begin{pmatrix}-2&2\\2&-2\end{pmatrix}$ gives $\mathbf v=(1,1)$ . For $-1$ : $S+I=\begin{pmatrix}2&2\\2&2\end{pmatrix}$ gives $\mathbf v=(1,-1)$ . They are perpendicular ( $1-1=0$ ). That is no accident: it is the spectral theorem of week 5.

Your turn: 4. Missing eigenvalue

A $3\times3$ matrix has trace 6, determinant 6, and one eigenvalue equal to 1. Find the other two.

Need a hint?

The other two, call them $a,b$ , satisfy $1+a+b=6$ and $1\cdot a\cdot b=6$ .

Show the full solution

$a+b=5$ and $ab=6$ , so they are the roots of $x^2-5x+6=0$ : $2$ and $3$ . The eigenvalues are $1,\,2,\,3$ .

Your turn: 5. Complex or real?

Without computing, can $\begin{pmatrix}0&-4\\1&0\end{pmatrix}$ have real eigenvalues?

Need a hint?

Find $\lambda^2-\text{trace}\cdot\lambda+\det$ .

Show the full solution

Trace $0$ , determinant $0\cdot0-(-4)(1)=4$ . The equation $\lambda^2+4=0$ has $\lambda=\pm2i$ . No real eigenvalues: this matrix turns every real arrow.

Exam radar

Eigenvalues and eigenvectors appear in 20 of the 53 MLF Quiz 1 papers and in 29 of the 30 Quiz 2 papers that have a concept summary and in nearly every End Term. Diagonalisation appears in 7 Quiz 1 and 6 Quiz 2 papers.

Past End Terms keep returning to: the trace 3, determinant 2 puzzle (8 times), $\det((A^{-1})^T)$ from eigenvalues $1,\,0.5,\,0.25$ with answer 8 (7 times), eigenvalues of $A+kI$ and of powers of $A$ , and properties of symmetric and positive definite matrices (next week).

Watch out - real exam trap

Eigenvalues of $A^{-1}$ are reciprocals, not negatives.
Eigenvalues of $A+cI$ shift by $c$ ; they do not multiply by anything.
Product of eigenvalues is the determinant, sum is the trace. Do not swap.
$A$ and $A^T$ share eigenvalues, but not necessarily eigenvectors.
"Repeated eigenvalue" does not decide diagonalisability.

Quick quiz

A 2 by 2 matrix has trace 5 and determinant 6. Its eigenvalues are...

1 and 6
2 and 3
5 and 6
3 and 3

They are the roots of x^2 - 5x + 6 = 0, which are 2 and 3.

A has eigenvalues 2 and 5. The eigenvalues of A inverse are...

-2 and -5
1/2 and 1/5
2 and 5
4 and 25

The eigenvalues of A inverse are reciprocals.

Which matrix is NOT diagonalisable?

the identity matrix
[[1, 1], [0, 1]]
[[2, 0], [0, 3]]
[[1, 2], [2, 1]]

The shear has the repeated eigenvalue 1 with only one independent eigenvector. The identity and the diagonal matrix are already diagonal; the symmetric matrix is always diagonalisable.

Select all that are TRUE. (select all that apply)

The sum of the eigenvalues equals the trace.
A matrix with a zero eigenvalue is not invertible.
Eigenvectors for different eigenvalues are always perpendicular.
A repeated eigenvalue means the matrix cannot be diagonalised.

Perpendicularity is only guaranteed for symmetric matrices. Repeated eigenvalues do not by themselves rule out diagonalisation.

Recap

Remember this

$A\mathbf v=\lambda\mathbf v$ : along special directions the matrix only stretches. Find $\lambda$ from $\det(A-\lambda I)=0$ , then $\mathbf v$ from $(A-\lambda I)\mathbf v=0$ .
$\sum\lambda_i=\operatorname{trace}$ , $\prod\lambda_i=\det$ . Triangular matrix: eigenvalues are the diagonal.
$A^{-1}\to1/\lambda$ , $A^k\to\lambda^k$ , $A+cI\to\lambda+c$ , $A^T\to$ same $\lambda$ . Invertible $\iff$ no zero eigenvalue.
Diagonalisable $\iff$ $n$ independent eigenvectors; then $A=PDP^{-1}$ and $A^k=PD^kP^{-1}$ . Repeated eigenvalues alone decide nothing.

Formulas to keep

$A\mathbf v=\lambda\mathbf v$ , $\det(A-\lambda I)=0$
$2\times2$ : $\lambda^2-(\operatorname{tr}A)\lambda+\det A=0$
$\sum\lambda_i=\operatorname{tr}A$ , $\prod\lambda_i=\det A$
$A^{-1}:1/\lambda$ ; $A^k:\lambda^k$ ; $A+cI:\lambda+c$ ; $A^T:\lambda$
$A=PDP^{-1}$ , $A^k=PD^kP^{-1}$

Connected weeks (learn them together)

MLT week 1 - PCA: finding the best view of your data: Principal components ARE eigenvectors of the covariance matrix.
MLT week 2 - Kernel PCA: curved views of data: Kernel PCA is eigen-analysis of a kernel matrix.

Practise with real exam questions

These come straight from past papers (with the real figures and the official answer key). Open 55 real questions on this week All Quiz 1 papers