Unit 08

Matrix Decompositions for Machine Learning

breaking matrices into fundamental components for analysis and computation

Core Concepts Covered

Eigenvalues, eigenvectors, and spectral decomposition
Singular Value Decomposition (SVD) and low-rank approximation
Principal Component Analysis (PCA) for dimensionality reduction

Local Setup Recommendation

To execute and experiment with the code cells below on your local machine, ensure you have set up your isolated virtual environments and scientific libraries by following the detailed protocols in Unit 03: Environment Setup or run them in Google Colab.

1. Eigenvalues, Eigenvectors, and Spectral Decomposition

Just as integers can be factored into prime numbers (e.g., $15 = 3 \times 5$ ), matrices can be decomposed into fundamental vector and scaling components. This allows us to analyze the structural properties of complex linear transformations.

An Eigenvector of a square matrix $A$ is a special non-zero vector $v$ such that multiplying $A$ by $v$ only scales the vector, without changing its geometric direction. This scaling factor is called the Eigenvalue $\lambda$ . Mathematically, this is written as:

$Av = \lambda v$

The Stretching Sheet Analogy: Imagine taking a square rubber sheet and stretching it diagonally. Most points on the sheet move in new directions. However, points lying along the exact diagonal line of the stretch continue to point in the exact same direction—they have only been pulled further apart. This diagonal line represents the Eigenvector of the stretching transformation, and the amount of stretch represents its Eigenvalue.

If a matrix $A$ is symmetric ( $A = A^T$ ), we can perform Spectral Decomposition. This factors the matrix into an orthogonal basis of eigenvectors and a diagonal matrix of eigenvalues:

$A = Q \Lambda Q^T$

where $Q$ is an orthogonal matrix whose columns are the eigenvectors of $A$ , and $\Lambda$ (Lambda) is a diagonal matrix containing the corresponding eigenvalues. We use eigenvalues to measure the scale of variance along specific vector directions.

Jupyter Code Notebook Cell

Python 3 (ipykernel)

In [1]:

import numpy as np

# Create a symmetric matrix A
A = np.array([[4, 2], [2, 4]])

# Compute eigenvalues and eigenvectors
eigenvalues, eigenvectors = np.linalg.eig(A)

print("Eigenvalues:", eigenvalues)
# Columns of eigenvectors correspond to eigenvectors
print("Eigenvector Matrix:\n", eigenvectors)

# Verify Av = lambda v for the first eigenvector
v0 = eigenvectors[:, 0]
lambda0 = eigenvalues[0]
print("Av0:", A @ v0)
print("lambda0 * v0:", lambda0 * v0)
print("Av0 equals lambda0 * v0:", np.allclose(A @ v0, lambda0 * v0))

Out [1]:

Eigenvalues: [6. 2.]
Eigenvector Matrix:
 [[ 0.70710678 -0.70710678]
 [ 0.70710678  0.70710678]]
Av0: [4.24264069 4.24264069]
lambda0 * v0: [4.24264069 4.24264069]
Av0 equals lambda0 * v0: True

2. Singular Value Decomposition (SVD)

While eigensystem decompositions are mathematically restricted strictly to square matrices, Singular Value Decomposition (SVD) is a highly robust factorization that works on *any* rectangular matrix $A \in \mathbb{R}^{m \times n}$ (such as a table with $m$ samples and $n$ features).

SVD decomposes $A$ into three matrices:

$A = U \Sigma V^T$

• $U \in \mathbb{R}^{m \times m}$ : An orthogonal matrix containing the left-singular vectors of $A$ (which represent eigenvectors of $A A^T$ ).

• $\Sigma \in \mathbb{R}^{m \times n}$ : A diagonal matrix containing positive singular values $\sigma_i$ sorted in descending order of magnitude.

• $V^T \in \mathbb{R}^{n \times n}$ : The transpose of an orthogonal matrix containing the right-singular vectors of $A$ (which represent eigenvectors of $A^T A$ ).

The Movie Recommender Example: Imagine a matrix $A$ where rows represent users and columns represent movies. The cell $A_{ij}$ is the rating user $i$ gave to movie $j$ . SVD factorizes this rating grid into: User-concept profiles ( $U$ ), mapping users to latent concepts (like *Action* or *Romance*); concept importance scales ( $\Sigma$ ); and Movie-concept profiles ( $V^T$ ), mapping movies to those same concepts. By keeping only the top $k$ largest singular values in $\Sigma$ (a low-rank approximation), we compress the massive rating matrix, filtering out minor noise and predicting missing user reviews smoothly.

3. Dimensionality Reduction & Principal Component Analysis (PCA)

In high-dimensional datasets, features are often highly redundant (for instance, house area in square feet vs. square meters). We use Principal Component Analysis (PCA) to reduce dataset dimensions while retaining as much variance (information) as possible.

To perform PCA on a dataset matrix $X$ :

1. Zero-center the data by subtracting the mean of each feature: $X_0 = X - \mu$ .

2. Compute the Covariance Matrix: $\Sigma_0 = \frac{1}{n} X_0^T X_0$ . This matrix measures how all features vary together.

3. Compute the Eigensystem: Find the eigenvectors and eigenvalues of the covariance matrix $\Sigma_0$ . The eigenvector corresponding to the largest eigenvalue is the First Principal Component—the axis in coordinate space along which the data is most spread out (maximum variance).

4. Project: Project our original high-dimensional coordinates onto these principal eigenvectors, compressing the coordinates onto fewer dimensions while minimizing information loss.

Interactive Practice Quiz

Test your understanding with instant feedback

QUESTION 01

Which of the following describes the definition of an Eigenvector $v$ and Eigenvalue $\lambda$ for a matrix $A$ ?

QUESTION 02

Why is Singular Value Decomposition (SVD) preferred over Eigensystem decomposition for general-purpose dataset matrices?

QUESTION 03

In SVD decomposition $A = U \Sigma V^T$ , what does the diagonal matrix $\Sigma$ contain?

QUESTION 04

In Principal Component Analysis (PCA), the principal components are calculated as the eigenvectors of which matrix?

QUESTION 05

How do we compress data using SVD low-rank approximations?

QUESTION 06

If a matrix $A \in \mathbb{R}^{d \times d}$ is symmetric ( $A = A^T$ ), which of the following properties must its eigenvalues and eigenvectors satisfy?

QUESTION 07

In Spectral Decomposition $A = Q \Lambda Q^T$ , what does the diagonal matrix $\Lambda$ represent?

QUESTION 08

For a dataset with $n$ samples and $d$ features, the dimension of the right-singular vector matrix $V^T$ in SVD is:

QUESTION 09

Which of the following describes the first step of Principal Component Analysis (PCA)?

QUESTION 10

Matrix Decompositions for Machine Learning

Core Concepts Covered

Local Setup Recommendation

1. Eigenvalues, Eigenvectors, and Spectral Decomposition

2. Singular Value Decomposition (SVD)

3. Dimensionality Reduction & Principal Component Analysis (PCA)

Interactive Practice Quiz

Which of the following describes the definition of an Eigenvector $v$ and Eigenvalue $\lambda$ for a matrix $A$ ?

Why is Singular Value Decomposition (SVD) preferred over Eigensystem decomposition for general-purpose dataset matrices?

In SVD decomposition $A = U \Sigma V^T$ , what does the diagonal matrix $\Sigma$ contain?

In Principal Component Analysis (PCA), the principal components are calculated as the eigenvectors of which matrix?

How do we compress data using SVD low-rank approximations?

If a matrix $A \in \mathbb{R}^{d \times d}$ is symmetric ( $A = A^T$ ), which of the following properties must its eigenvalues and eigenvectors satisfy?

In Spectral Decomposition $A = Q \Lambda Q^T$ , what does the diagonal matrix $\Lambda$ represent?

For a dataset with $n$ samples and $d$ features, the dimension of the right-singular vector matrix $V^T$ in SVD is:

Which of the following describes the first step of Principal Component Analysis (PCA)?

The first principal component axis extracted during PCA represents:

Further Readings

Matrix Decompositions for Machine Learning

Core Concepts Covered

Local Setup Recommendation

1. Eigenvalues, Eigenvectors, and Spectral Decomposition

2. Singular Value Decomposition (SVD)

3. Dimensionality Reduction & Principal Component Analysis (PCA)

Interactive Practice Quiz

Which of the following describes the definition of an Eigenvector vvv and Eigenvalue λ\lambdaλ for a matrix AAA?

Why is Singular Value Decomposition (SVD) preferred over Eigensystem decomposition for general-purpose dataset matrices?

In SVD decomposition A=UΣVTA = U \Sigma V^TA=UΣVT, what does the diagonal matrix Σ\SigmaΣ contain?

In Principal Component Analysis (PCA), the principal components are calculated as the eigenvectors of which matrix?

How do we compress data using SVD low-rank approximations?

If a matrix A∈Rd×dA \in \mathbb{R}^{d \times d}A∈Rd×d is symmetric (A=ATA = A^TA=AT), which of the following properties must its eigenvalues and eigenvectors satisfy?

In Spectral Decomposition A=QΛQTA = Q \Lambda Q^TA=QΛQT, what does the diagonal matrix Λ\LambdaΛ represent?

For a dataset with nnn samples and ddd features, the dimension of the right-singular vector matrix VTV^TVT in SVD is:

Which of the following describes the first step of Principal Component Analysis (PCA)?

The first principal component axis extracted during PCA represents:

Further Readings

Which of the following describes the definition of an Eigenvector $v$ and Eigenvalue $\lambda$ for a matrix $A$ ?

In SVD decomposition $A = U \Sigma V^T$ , what does the diagonal matrix $\Sigma$ contain?

If a matrix $A \in \mathbb{R}^{d \times d}$ is symmetric ( $A = A^T$ ), which of the following properties must its eigenvalues and eigenvectors satisfy?

In Spectral Decomposition $A = Q \Lambda Q^T$ , what does the diagonal matrix $\Lambda$ represent?

For a dataset with $n$ samples and $d$ features, the dimension of the right-singular vector matrix $V^T$ in SVD is: