IIT M IMPROVEMENT AN EXAM QIV3 03 Aug - 03 Aug 2025

Concepts this paper tests

Auto-generated summary from the source site. Handy as a checklist; the lessons above explain each idea from scratch.

Summary of Core Topics and Concepts

1. Optimization and Critical Points

• Gradient and Hessian Matrix:

  • Gradient ($\nabla f$): A vector of partial derivatives.
  • Hessian Matrix ($\nabla^2 f$): A matrix of second-order partial derivatives.
  • Key Concept: If $\nabla f(x^*) = 0$ and $\nabla^2 f(x^*) > 0$, then $x^*$ is a local minimum.

• Quadratic Forms:

  • Definition: $Q(x, y, z) = x^2 + 2xy - 2xz$
  • Definiteness: Positive definite, negative definite, etc.
  • Key Concept: Understanding the definiteness of quadratic forms is crucial for optimization.

2. Matrix Properties

• Positive Definite Matrices:

  • Definition: A matrix $A$ is positive definite if $x^T A x > 0$ for all non-zero vectors $x$.
  • Example: $A = \begin{pmatrix} 2 & -1 & 0 \\ -1 & 2 & -1 \\ 0 & -1 & 2 \end{pmatrix}$ is positive definite.

• Eigenvalues and Eigenvectors:

  • Definition: For a matrix $A$, $\lambda$ is an eigenvalue and $v$ is an eigenvector if $Av = \lambda v$.
  • Key Concept: Eigenvalues and eigenvectors are essential for understanding matrix properties and transformations.

3. Singular Value Decomposition (SVD)

• Definition: $A = Q_1 \Sigma Q_2^T$

  • Key Concept: SVD is used for dimensionality reduction, data compression, and noise reduction.
  • Example: For matrix $A = \begin{pmatrix} 3 & 2 & -2 \\ 2 & 3 & 2 \end{pmatrix}$, find $Q_1$, $\Sigma$, and $Q_2^T$.

4. Principal Component Analysis (PCA)

• Definition: A technique to reduce the dimensionality of a dataset while preserving its variance.

  • Key Concept: PCA involves finding the eigenvalues and eigenvectors of the covariance matrix.
  • Example: Projecting a dataset onto a 1-dimensional space using PCA.

5. Gradient Descent

• Definition: An optimization algorithm used to minimize some function by iteratively moving in the direction of steepest descent.

  • Key Concept: Learning rate ($\eta_t$) and initial guess are crucial for the algorithm's performance.
  • Example: Using gradient descent to find the minimum of $f(x, y) = (x - 1)^2 + (y + 2)^2$.

6. Hermitian and Unitary Matrices

• Hermitian Matrix:

  • Definition: A matrix $A$ is Hermitian if $A = A^H$ (conjugate transpose).
  • Key Concept: Eigenvalues of a Hermitian matrix are real, and eigenvectors corresponding to different eigenvalues are orthogonal.

• Unitary Matrix:

  • Definition: A matrix $A$ is unitary if $A^H A = I$.
  • Key Concept: Unitary matrices preserve the norm of vectors.

Equations

Optimization

1. Gradient: $\nabla f(x, y) = \left( \frac{\partial f}{\partial x}, \frac{\partial f}{\partial y} \right)$
2. Hessian Matrix: $\nabla^2 f(x, y) = \begin{pmatrix} \frac{\partial^2 f}{\partial x^2} & \frac{\partial^2 f}{\partial x \partial y} \\ \frac{\partial^2 f}{\partial y \partial x} & \frac{\partial^2 f}{\partial y^2} \end{pmatrix}$

Quadratic Forms

1. Quadratic Form: $Q(x, y, z) = x^2 + 2xy - 2xz$

Matrix Properties

1. Positive Definite Matrix: $x^T A x > 0$ for all non-zero $x$
2. Eigenvalue Equation: $Av = \lambda v$

SVD

1. SVD: $A = Q_1 \Sigma Q_2^T$

PCA

1. Covariance Matrix: $C = \frac{1}{n-1} X^T X$
2. Eigenvalue Decomposition: $C = V \Lambda V^T$

Gradient Descent

1. Update Rule: $x_{t+1} = x_t - \eta_t \nabla f(x_t)$

Hermitian and Unitary Matrices

1. Hermitian Condition: $A = A^H$
2. Unitary Condition: $A^H A = I$