Unit 07

Linear Algebra and Matrix Mechanics for Machine Learning

vector spaces, matrix operations, and mathematical representations of data

Core Concepts Covered

Vectors, operations, dot products, and angles
Matrix operations, systems of linear equations, and inverses
Vector spaces, linear independence, basis, and transformations

Local Setup Recommendation

To execute and experiment with the code cells below on your local machine, ensure you have set up your isolated virtual environments and scientific libraries by following the detailed protocols in Unit 03: Environment Setup or run them in Google Colab.

1. Vectors, Dot Products, and Semantic Similarities

In machine learning, data is represented as vectors. A single sample is a vector of features, and an entire dataset is structured as a matrix of these vectors. Therefore, linear algebra is the mathematical language used to represent, project, and manipulate high-dimensional data.

A vector $x \in \mathbb{R}^d$ is an ordered sequence of $d$ real numbers. We can add vectors element-wise and multiply them by scalar values.

The Dot Product (or inner product) of two vectors $x, y \in \mathbb{R}^d$ multiplies corresponding elements and sums the results, producing a single scalar. The formula is:

$x \cdot y = x^T y = \sum_{i=1}^d x_i y_i$

Semantic Similarity Example: In Natural Language Processing, words are represented as numeric vectors ('word embeddings'). Suppose the word *king* is mapped to $u = [1, 2]^T$ and the word *queen* to $v = [2, 1.5]^T$ . Their dot product is $u \cdot v = (1)(2) + (2)(1.5) = 2 + 3 = 5$ . Mathematically, the dot product is related to the angle $\theta$ between the vectors by $u \cdot v = \|u\| \|v\| \cos(\theta)$ . By calculating the cosine similarity $\cos(\theta) = \frac{u \cdot v}{\|u\| \|v\|}$ , we measure how semantically similar the words are. Here: $\cos(\theta) = \frac{5}{\sqrt{1^2+2^2} \cdot \sqrt{2^2+1.5^2}} = \frac{5}{\sqrt{5} \cdot \sqrt{6.25}} = \frac{5}{2.236 \cdot 2.5} \approx 0.89$ A cosine similarity of $0.89$ (close to $1.0$ ) shows that the model successfully understands these words are highly related!

A set of vectors spans a vector space if any coordinate in that space can be reached through a linear combination of those vectors. If no vector in a set can be written as a linear combination of the others, the vectors are linearly independent. A set of linearly independent vectors that spans a space forms a Basis for that space, serving as the coordinates axis.

2. Matrix Mechanics and Linear Transformations

A matrix is a 2D grid of numbers representing a linear transformation. When we multiply a matrix $A$ by a vector $x$ (written as $Ax$ ), we are geometrically transforming the vector $x$ (scaling, rotating, or projecting it) into a new coordinate space.

• Matrix Multiplication ( $C = AB$ ): To compute elements of $C$ , we take the dot product of the $i$ -th row of $A$ with the $j$ -th column of $B$ : $C_{ij} = \sum_{k} A_{ik} B_{kj}$

• Determinant ( $\det(A)$ ): A scalar measuring how much the linear transformation scales the area or volume of space. If $\det(A) = 0$ , the matrix squashes space into a lower dimension (e.g. flatting a 2D plane into a 1D line), meaning the matrix is singular and cannot be inverted.

• Matrix Inverse ( $A^{-1}$ ): Bypasses standard division for matrices. If a matrix has an inverse, then multiplying $A^{-1} A = I$ (the Identity Matrix, which acts as the number 1 for matrices). We use inverses to solve systems of linear equations: $Ax = b \implies x = A^{-1} b$ .

Jupyter Code Notebook Cell

Python 3 (ipykernel)

In [1]:

import numpy as np

# Define matrices A and B
A = np.array([[2, 1], [1, 3]])
B = np.array([[1, 0], [2, 1]])

# 1. Matrix Multiplication (AB)
C = np.dot(A, B)  # Or: A @ B
print("Matrix C = A @ B:\n", C)

# 2. Compute Determinant of A
det_A = np.linalg.det(A)
print("Determinant of A:", round(det_A, 2))

# 3. Compute Matrix Inverse of A
A_inv = np.linalg.inv(A)
print("Inverse of A:\n", A_inv)
print("A_inv @ A (Identity):\n", np.round(A_inv @ A))

Out [1]:

Matrix C = A @ B:
 [[4 1]
 [7 3]]
Determinant of A: 5.0
Inverse of A:
 [[ 0.6 -0.2]
 [-0.2  0.4]]
A_inv @ A (Identity):
 [[1. 0.]
 [0. 1.]]

3. Vector and Matrix Norms

In machine learning, we need to measure the size, length, or magnitude of vectors and matrices (especially inside regularization terms to prevent overfitting). We calculate magnitude using mathematical functions called norms.

• $L_1$ Norm (Taxicab Norm): Sums the absolute values of the vector coordinates. It promotes sparsity (forcing weights to exactly zero, helpful for feature selection): $\|x\|_1 = \sum_{i=1}^d |x_i|$

• $L_2$ Norm (Euclidean Norm): Measures the standard straight-line distance from the origin. It is the most common norm used in weight decay and Ridge regressions: $\|x\|_2 = \sqrt{\sum_{i=1}^d x_i^2}$

• Frobenius Norm (for matrices): Measures the overall scale of a matrix $A \in \mathbb{R}^{m \times n}$ by taking the square root of the sum of all squared entries: $\|A\|_F = \sqrt{\sum_{i=1}^m \sum_{j=1}^n A_{ij}^2}$

Interactive Practice Quiz

Test your understanding with instant feedback

QUESTION 01

Calculate the dot product of vectors $x = [1, 3, -2]^T$ and $y = [4, 2, 5]^T$ :

QUESTION 02

Under what condition does a square matrix $A$ NOT have an inverse matrix $A^{-1}$ ?

QUESTION 03

Which vector norm promotes weight 'sparsity' (forcing parameter weights to be exactly zero) during regularization optimization?

QUESTION 04

Calculate the $L_2$ Norm of the coordinate vector $x = [3, -4]^T$ :

QUESTION 05

If a set of vectors spans a space and is completely linearly independent, what is this set called?

QUESTION 06

Compute the matrix-vector product $A v$ where $A = \begin{bmatrix} 2 & 1 \\ 3 & 4 \end{bmatrix}$ and $v = \begin{bmatrix} 1 \\ 2 \end{bmatrix}$ :

QUESTION 07

Compute the determinant of the matrix $A = \begin{bmatrix} 3 & 2 \\ 1 & 4 \end{bmatrix}$ :

QUESTION 08

Compute the Frobenius Norm of the matrix $A = \begin{bmatrix} 1 & 2 \\ 3 & 0 \end{bmatrix}$ :

QUESTION 09

If two vectors $u$ and $v$ have a cosine similarity of $1.0$ , what does this mathematically represent?

QUESTION 10

Linear Algebra and Matrix Mechanics for Machine Learning

Core Concepts Covered

Local Setup Recommendation

1. Vectors, Dot Products, and Semantic Similarities

2. Matrix Mechanics and Linear Transformations

3. Vector and Matrix Norms

Interactive Practice Quiz

Calculate the dot product of vectors $x = [1, 3, -2]^T$ and $y = [4, 2, 5]^T$ :

Under what condition does a square matrix $A$ NOT have an inverse matrix $A^{-1}$ ?

Which vector norm promotes weight 'sparsity' (forcing parameter weights to be exactly zero) during regularization optimization?

Calculate the $L_2$ Norm of the coordinate vector $x = [3, -4]^T$ :

If a set of vectors spans a space and is completely linearly independent, what is this set called?

Compute the matrix-vector product $A v$ where $A = \begin{bmatrix} 2 & 1 \\ 3 & 4 \end{bmatrix}$ and $v = \begin{bmatrix} 1 \\ 2 \end{bmatrix}$ :

Compute the determinant of the matrix $A = \begin{bmatrix} 3 & 2 \\ 1 & 4 \end{bmatrix}$ :

Compute the Frobenius Norm of the matrix $A = \begin{bmatrix} 1 & 2 \\ 3 & 0 \end{bmatrix}$ :

If two vectors $u$ and $v$ have a cosine similarity of $1.0$ , what does this mathematically represent?

What is the key characteristic of an Identity Matrix $I$ ?

Further Readings

Linear Algebra and Matrix Mechanics for Machine Learning

Core Concepts Covered

Local Setup Recommendation

1. Vectors, Dot Products, and Semantic Similarities

2. Matrix Mechanics and Linear Transformations

3. Vector and Matrix Norms

Interactive Practice Quiz

Calculate the dot product of vectors x=[1,3,−2]Tx = [1, 3, -2]^Tx=[1,3,−2]T and y=[4,2,5]Ty = [4, 2, 5]^Ty=[4,2,5]T:

Under what condition does a square matrix AAA NOT have an inverse matrix A−1A^{-1}A−1?

Which vector norm promotes weight 'sparsity' (forcing parameter weights to be exactly zero) during regularization optimization?

Calculate the L2L_2L2​ Norm of the coordinate vector x=[3,−4]Tx = [3, -4]^Tx=[3,−4]T:

If a set of vectors spans a space and is completely linearly independent, what is this set called?

Compute the matrix-vector product AvA vAv where A=[2134]A = \begin{bmatrix} 2 & 1 \\ 3 & 4 \end{bmatrix}A=[23​14​] and v=[12]v = \begin{bmatrix} 1 \\ 2 \end{bmatrix}v=[12​]:

Compute the determinant of the matrix A=[3214]A = \begin{bmatrix} 3 & 2 \\ 1 & 4 \end{bmatrix}A=[31​24​]:

Compute the Frobenius Norm of the matrix A=[1230]A = \begin{bmatrix} 1 & 2 \\ 3 & 0 \end{bmatrix}A=[13​20​]:

If two vectors uuu and vvv have a cosine similarity of 1.01.01.0, what does this mathematically represent?

What is the key characteristic of an Identity Matrix III?

Further Readings

Calculate the dot product of vectors $x = [1, 3, -2]^T$ and $y = [4, 2, 5]^T$ :

Under what condition does a square matrix $A$ NOT have an inverse matrix $A^{-1}$ ?

Calculate the $L_2$ Norm of the coordinate vector $x = [3, -4]^T$ :

Compute the matrix-vector product $A v$ where $A = \begin{bmatrix} 2 & 1 \\ 3 & 4 \end{bmatrix}$ and $v = \begin{bmatrix} 1 \\ 2 \end{bmatrix}$ :

Compute the determinant of the matrix $A = \begin{bmatrix} 3 & 2 \\ 1 & 4 \end{bmatrix}$ :

Compute the Frobenius Norm of the matrix $A = \begin{bmatrix} 1 & 2 \\ 3 & 0 \end{bmatrix}$ :

If two vectors $u$ and $v$ have a cosine similarity of $1.0$ , what does this mathematically represent?

What is the key characteristic of an Identity Matrix $I$ ?