Home / MLF / Week 5

Week 5 · Symmetric matrices: the well-behaved ones

Official syllabus line: Linear Algebra - Symmetric matrices

About 80 minQuiz 2 material (see exam radar)BSCS2004 · Harish Guruprasad Ramaswamy, Arun Rajkumar, Prashanth LA

Before you start: Week 4 (eigenvalues and eigenvectors, diagonalisation).

Watch first: the animated story of this weekAbout 1-2 minutes of real 3D animation with a friendly voice. Pause any time.

Read the words of the video

Last week, some math shapes had crooked special directions. This week, we meet the well-behaved ones! Imagine a matrix that's a mirror image across its middle line. It's like stretching a grid, such as a window pane, perfectly straight in two directions, with no strange twisting. Everything stays neat and tidy.
For these special symmetric matrices, everything is tidy! Their special numbers, called eigenvalues, are always real. Even better, their special directions, or eigenvectors, are always perfectly perpendicular to each other. This neat behaviour is why these matrices are super important in machine learning, making data analysis much simpler.
Let's try one! A symmetric matrix like 'two, one; one, two' has two special numbers: three and one. And its special directions? They are indeed perfectly perpendicular. We can rewrite the matrix using these perpendicular directions and the special numbers, making it much easier to understand and work with. It's a neat trick!
For your exams, remember that symmetric matrices always have real eigenvalues and perpendicular eigenvectors. Also, positive definite matrices have all positive eigenvalues, like a bowl shape! This tidy family is super important for understanding data in machine learning, helping us solve complex problems with confidence and clarity. You've got this!

The one-minute story

Think of it like this

Last week you met matrices whose special directions can be crooked: two eigenvectors could lean towards each other at any odd angle, and some matrices had no real special directions at all.

This week is about the well-behaved family. Imagine a matrix that is a mirror image of itself across its diagonal: the number in row 1, column 2 equals the number in row 2, column 1. That is a symmetric matrix. It is like stretching a sheet along two perpendicular axes, like the grid of a window, with no weird shearing.

Everything is tidy for symmetric matrices: all the eigenvalues are real numbers, the special directions are perpendicular, and the matrix can always be rewritten using a clean rotation. Because data analysis keeps producing symmetric matrices (covariance, similarity, curvature), this tidy family is the one that matters most in machine learning.

By the end of this week you can:

state and use the spectral theorem for symmetric matrices;
recognise orthogonal matrices and why they are special;
decide whether a quadratic form or symmetric matrix is positive definite, semi-definite, negative definite or indefinite;
handle the complex cousins: Hermitian and unitary matrices.

Words you need

Symmetric matrix - $A=A^T$ .
Orthogonal matrix $Q$ - square, with perpendicular unit-length columns: $Q^TQ=I$ . It rotates or reflects without stretching.
Orthonormal - perpendicular and length 1.
Quadratic form - the expression $\mathbf x^TA\mathbf x$ , a "squared" function of the entries of $\mathbf x$ .
Positive definite (PD) - $\mathbf x^TA\mathbf x>0$ for every non-zero $\mathbf x$ .
Conjugate - of a complex number $a+bi$ is $a-bi$ .
Hermitian matrix - $A^*=A$ , where $A^*$ means "transpose and conjugate". The complex version of symmetric.
Unitary matrix - $U^*U=I$ . The complex version of orthogonal.

Idea 1: Symmetric matrices and the spectral theorem

The idea

Spectral theorem. If $A$ is a real symmetric matrix, then

all its eigenvalues are real;
eigenvectors for different eigenvalues are perpendicular;
$A$ is always diagonalisable, and with orthonormal eigenvectors:

A=Q\Lambda Q^T,\qquad Q^TQ=I

Here $Q$ has the unit-length eigenvectors as columns and $\Lambda$ holds the eigenvalues. Compare with week 4: $A=PDP^{-1}$ . For symmetric matrices the awkward $P^{-1}$ turns into the easy $Q^T$ .

Why are the eigenvectors perpendicular? Take $A\mathbf v=\lambda\mathbf v$ and $A\mathbf w=\mu\mathbf w$ with $\lambda\ne\mu$ . Since $A$ is symmetric, $(A\mathbf v)\cdot\mathbf w=\mathbf v\cdot(A\mathbf w)$ . The left side is $\lambda(\mathbf v\cdot\mathbf w)$ , the right side is $\mu(\mathbf v\cdot\mathbf w)$ . So $(\lambda-\mu)(\mathbf v\cdot\mathbf w)=0$ . Because $\lambda\neq\mu$ , we must have $\mathbf v\cdot\mathbf w=0$ .

Orthogonal matrices

A matrix $Q$ with $Q^TQ=I$ has these properties:

columns are perpendicular unit vectors, so $Q^{-1}=Q^T$ (inverting is free);
$\|Q\mathbf x\|=\|\mathbf x\|$ and $(Q\mathbf x)\cdot(Q\mathbf y)=\mathbf x\cdot\mathbf y$ : lengths and angles are preserved;
$\det Q=\pm1$ , and every eigenvalue has $|\lambda|=1$ .

Rotations and reflections are the orthogonal matrices.

Solved example: 1. Orthogonally diagonalise a symmetric matrix

Question. Write $S=\begin{pmatrix}2&1\\1&2\end{pmatrix}$ as $Q\Lambda Q^T$ .

Solution.

Trace $4$ , determinant $3$ : $\lambda^2-4\lambda+3=0$ , so $\lambda=3$ and $\lambda=1$ (both real, as promised).
$\lambda=3$ : $S-3I=\begin{pmatrix}-1&1\\1&-1\end{pmatrix}$ gives $(1,1)$ . $\lambda=1$ : $S-I=\begin{pmatrix}1&1\\1&1\end{pmatrix}$ gives $(1,-1)$ .
They are perpendicular: $1\cdot1+1\cdot(-1)=0$ . Normalise by dividing each by $\sqrt2$ .
$Q=\dfrac1{\sqrt2}\begin{pmatrix}1&1\\1&-1\end{pmatrix}$ , $\Lambda=\begin{pmatrix}3&0\\0&1\end{pmatrix}$ , and $S=Q\Lambda Q^T$ .

Check one entry: the top-left of $Q\Lambda Q^T$ is $\frac12(3\cdot1+1\cdot1)=2$ . ✓

Idea 2: Quadratic forms and definiteness

For a symmetric $A$ , the quadratic form $q(\mathbf x)=\mathbf x^TA\mathbf x$ is a function that curves like a bowl, a dome, or a saddle. For $A=\begin{pmatrix}a&b\\b&c\end{pmatrix}$ it is $ax^2+2bxy+cy^2$ .

Think of it like this

Put a marble on a surface. If the surface curves up in every direction, the marble settles at the bottom: positive definite (a bowl). If it curves down in every direction, the marble rolls off the top: negative definite (a dome). If it curves up one way and down another, you have a saddle (a horse's saddle, or a Pringle): indefinite.

Level curves (contour lines) of two quadratic forms, seen from above.

The idea

Name	Condition	All eigenvalues	Shape
Positive definite (PD)	$\mathbf x^TA\mathbf x>0$ for all $\mathbf x\ne0$	all $>0$	bowl
Positive semi-definite (PSD)	$\mathbf x^TA\mathbf x\ge0$	all $\ge0$	bowl with a flat valley
Negative definite	$\mathbf x^TA\mathbf x<0$	all $<0$	dome
Negative semi-definite	$\le0$	all $\le0$	dome with a flat ridge
Indefinite	takes both signs	mixed signs	saddle

Why do eigenvalues decide it? Write $A=Q\Lambda Q^T$ and let $\mathbf y=Q^T\mathbf x$ . Then $\mathbf x^TA\mathbf x=\mathbf y^T\Lambda\mathbf y=\lambda_1y_1^2+\dots+\lambda_ny_n^2$ . A sum of squares weighted by the $\lambda_i$ . If every $\lambda_i>0$ it is positive whatever $\mathbf y$ is; if some are negative you can choose $\mathbf y$ to make it negative.

Quick tests that avoid finding eigenvalues

$2\times2$ : PD $\iff$ $a>0$ and $\det A>0$ . Indefinite if $\det A<0$ .
Leading minors (Sylvester): PD $\iff$ all leading principal minors (the determinants of the top-left $1\times1$ , $2\times2$ , $3\times3$ , ... blocks) are $>0$ .
PSD is subtler: all principal minors must be $\ge0$ , not just the leading ones.
A PD matrix has a positive diagonal, and every PD matrix is invertible.

Solved example: 2. Classify three quadratic forms

Question. Decide the definiteness of $A=\begin{pmatrix}2&1\\1&2\end{pmatrix}$ , $B=\begin{pmatrix}1&2\\2&1\end{pmatrix}$ and $C=\begin{pmatrix}1&2\\2&4\end{pmatrix}$ .

Solution.

$A$ : $a=2>0$ and $\det=4-1=3>0$ , so PD. (Its eigenvalues are 3 and 1.)
$B$ : $\det=1-4=-3<0$ , so indefinite. (Eigenvalues $3$ and $-1$ .)
$C$ : $\det=4-4=0$ and trace $=5$ , so eigenvalues $5$ and $0$ : positive semi-definite, not definite, because $\mathbf x=(2,-1)$ gives $\mathbf x^TC\mathbf x=0$ .

Solved example: 3. Leading minors for a

3\times3

Question. Is $M=\begin{pmatrix}2&-1&0\\-1&2&-1\\0&-1&2\end{pmatrix}$ positive definite?

Solution. Leading minors: $\det[2]=2>0$ ; $\det\begin{pmatrix}2&-1\\-1&2\end{pmatrix}=3>0$ ; $\det M=4>0$ . All positive, so yes, PD. (Its eigenvalues are $2-\sqrt2,\;2,\;2+\sqrt2$ , all positive.)

Where you will see this again.

The Hessian of a function is PD at a minimum (week 7).
A function is convex exactly when its Hessian is PSD everywhere (week 8).
A covariance matrix is always PSD (it measures variances, which cannot be negative), which is why PCA works (week 6).
A function is a valid kernel exactly when its Gram matrix is PSD (MLT week 2).

Idea 3: The complex family

Some papers use complex numbers. All you need: $i$ is a number with $i^2=-1$ ; a complex number is $z=a+bi$ ; its conjugate is $\bar z=a-bi$ ; and $z\bar z=a^2+b^2=|z|^2$ , a real non-negative number.

For a complex matrix, the conjugate transpose $A^*$ flips the matrix over its diagonal and conjugates every entry. The real-world matrices you know are the special case where conjugating changes nothing.

Real	Complex	Definition	Key properties
symmetric	Hermitian	$A^*=A$	diagonal entries real; eigenvalues real; eigenvectors for different eigenvalues orthogonal; unitarily diagonalisable
orthogonal	unitary	$U^*U=I$	$\\|U\mathbf x\\|=\\|\mathbf x\\|$ ; every eigenvalue has $\|\lambda\|=1$ ; columns orthonormal

Schur's theorem (appears in Quiz 2): every square matrix can be written $A=UTU^*$ with $U$ unitary and $T$ upper triangular, and the eigenvalues of $A$ sit on the diagonal of $T$ . Hermitian and unitary matrices are special cases where $T$ turns out to be diagonal.

Solved example: 4. Hermitian, unitary, neither?

Question. Classify $H=\begin{pmatrix}2&1+i\\1-i&3\end{pmatrix}$ and $U=\dfrac1{\sqrt2}\begin{pmatrix}1&i\\i&1\end{pmatrix}$ .

Solution.

$H^*$ : transpose gives $\begin{pmatrix}2&1-i\\1+i&3\end{pmatrix}$ , then conjugate gives $\begin{pmatrix}2&1+i\\1-i&3\end{pmatrix}=H$ . So $H$ is Hermitian. Its eigenvalues: trace 5, determinant $6-(1+i)(1-i)=6-2=4$ , so $\lambda^2-5\lambda+4=0$ , giving 1 and 4: real, as promised.
$U^*=\dfrac1{\sqrt2}\begin{pmatrix}1&-i\\-i&1\end{pmatrix}$ . Then $U^*U=\frac12\begin{pmatrix}1+1&i-i\\-i+i&1+1\end{pmatrix}=I$ . So $U$ is unitary. But $U^*\ne U$ , so it is not Hermitian: a "unitary but not Hermitian" matrix, a favourite exam object.

Your turn

Your turn: 1. Spectral theorem

Which are guaranteed for a real symmetric matrix? (a) real eigenvalues (b) orthogonal eigenvectors for different eigenvalues (c) at least one eigenvalue is positive (d) diagonalisable.

Need a hint?

Think about the three clauses of the spectral theorem.

Show the full solution

(a), (b), (d) are guaranteed. (c) is not: $\begin{pmatrix}-1&0\\0&-2\end{pmatrix}$ is symmetric with only negative eigenvalues.

Your turn: 2. Definiteness

Is $\begin{pmatrix}3&1\\1&1\end{pmatrix}$ positive definite? And $\begin{pmatrix}1&3\\3&1\end{pmatrix}$ ?

Need a hint?

Check $a>0$ and $\det>0$ .

Show the full solution

First: $a=3>0$ , $\det=3-1=2>0$ : PD. Second: $\det=1-9=-8<0$ : indefinite.

Your turn: 3. Orthogonal facts

$Q$ is orthogonal and $\mathbf x=(3,4)$ . What is $\|Q\mathbf x\|$ ? If $\det Q=-1$ , what kind of transformation could $Q$ be?

Need a hint?

Orthogonal matrices preserve length.

Show the full solution

$\|Q\mathbf x\|=\|\mathbf x\|=5$ . A determinant of $-1$ means a reflection (it flips orientation). A rotation has determinant $+1$ .

Your turn: 4. Parameter in a quadratic form

For which values of $a$ is $\begin{pmatrix}2&a\\a&2\end{pmatrix}$ positive definite?

Need a hint?

The $2\times2$ test: $2>0$ and $\det>0$ .

Show the full solution

$\det=4-a^2>0\iff|a|<2$ . So PD exactly when $-2<a<2$ . At $a=\pm2$ it is PSD (determinant 0).

Your turn: 5. Hermitian or not?

Is $\begin{pmatrix}1&i\\i&1\end{pmatrix}$ Hermitian?

Need a hint?

Compute $A^*$ : transpose, then conjugate.

Show the full solution

Transpose gives the same matrix (it is symmetric as a table), conjugating turns $i$ into $-i$ : $A^*=\begin{pmatrix}1&-i\\-i&1\end{pmatrix}\neq A$ . Not Hermitian, even though it is symmetric as a table of numbers. Being symmetric is not enough in the complex world.

Exam radar

MLF Quiz 2 summaries (30 of the 53 papers have one) mention quadratic forms in 19 papers, positive definite matrices in 12 + 10, Hermitian matrices in 10, unitary matrices in 9, Schur's theorem in 6. Real symmetric properties and PD properties appear as MSQs in nearly every cycle of the End Term, as does the "unitary but not Hermitian" question.

Typical MSQ: "Which of the following is true for a real symmetric matrix / positive definite matrix / Hermitian matrix?" Memorise the table in Idea 3 and the spectral theorem's three clauses.

Watch out - real exam trap

"Symmetric" is a real-matrix word. For complex matrices the test is $A^*=A$ .
PSD allows zero eigenvalues; PD does not. A PD matrix is invertible, a merely PSD one need not be.
Leading minors prove PD. They do not prove PSD.
Orthogonal $\ne$ projection. $P^2=P$ projections squash lengths; orthogonal matrices preserve them.
"Orthogonal matrix" means perpendicular unit columns, not just perpendicular columns.

Quick quiz

A real symmetric matrix has eigenvalues 2, 5 and 7. Which word describes it?

positive definite
negative definite
indefinite
orthogonal

All eigenvalues are positive. Orthogonal would need all eigenvalues of absolute value 1.

Which is always TRUE for a real symmetric matrix?

It has a zero eigenvalue.
Eigenvectors for different eigenvalues are perpendicular.
It has complex eigenvalues.
It is not invertible.

That is the second clause of the spectral theorem.

The matrix [[4, 2], [2, 1]] is...

positive definite
positive semi-definite but not positive definite
indefinite
negative definite

trace 5, determinant 4 - 4 = 0, so eigenvalues 5 and 0. A zero eigenvalue means semi-definite only.

Select all TRUE statements about an orthogonal matrix Q. (select all that apply)

Q transpose times Q equals the identity.
It preserves lengths of vectors.
Its determinant must be exactly 1.
It can have eigenvalue 3.

The determinant is 1 or -1. Every eigenvalue has absolute value 1, so 3 is impossible.

Recap

Remember this

Symmetric: real eigenvalues, perpendicular eigenvectors, $A=Q\Lambda Q^T$ .
Orthogonal: $Q^TQ=I$ , $Q^{-1}=Q^T$ , preserves lengths, $\det=\pm1$ , $|\lambda|=1$ .
Definiteness is decided by eigenvalue signs. $2\times2$ : PD $\iff a>0$ and $\det>0$ . Leading minors all positive $\iff$ PD.
Hermitian $=$ complex symmetric ( $A^*=A$ , real eigenvalues). Unitary $=$ complex orthogonal ( $U^*U=I$ , $|\lambda|=1$ ).

Formulas to keep

Spectral theorem: $A=A^T\Rightarrow A=Q\Lambda Q^T$ , $Q^TQ=I$ , $\lambda_i\in\mathbb R$
Quadratic form $q(\mathbf x)=\mathbf x^TA\mathbf x=\sum\lambda_iy_i^2$
PD $\iff$ all $\lambda>0$ $\iff$ all leading minors $>0$ ; PSD $\iff$ all $\lambda\ge0$
Orthogonal: $Q^TQ=I$ , $\|Q\mathbf x\|=\|\mathbf x\|$ , $\det Q=\pm1$
Hermitian: $A^*=A$ ; unitary: $U^*U=I$ ; Schur: $A=UTU^*$

Connected weeks (learn them together)

MLT week 1 - PCA: finding the best view of your data: Covariance matrices are symmetric, so the spectral theorem promises real eigenvalues and perpendicular principal components.
MLT week 2 - Kernel PCA: curved views of data: A function is a valid kernel exactly when its matrix is positive semi-definite.

Practise with real exam questions

These come straight from past papers (with the real figures and the official answer key). Open 50 real questions on this week All Quiz 2 papers