Unit 07

Linear Algebra and Matrix Mechanics for Machine Learning

vector spaces, matrix operations, and mathematical representations of data

Core Concepts Covered

Vectors, operations, dot products, and angles
Matrix operations, systems of linear equations, and inverses
Vector spaces, linear independence, basis, and transformations

Local Setup Recommendation

To execute and experiment with the code cells below on your local machine, ensure you have set up your isolated virtual environments and scientific libraries by following the detailed protocols in Unit 03: Environment Setup or run them in Google Colab.

1. Vectors, Dot Products, and Semantic Similarities

Machine learning turns data into vectors. One vector can describe a single sample, and a matrix can hold many samples at once. Linear algebra tells us how to compare, transform, and combine those representations, even when they have thousands of dimensions.

A vector $x \in \mathbb{R}^d$ is an ordered sequence of $d$ real numbers. We can add vectors element-wise and multiply them by scalar values.

The Dot Product (or inner product) of two vectors $x, y \in \mathbb{R}^d$ multiplies corresponding elements and sums the results, producing a single scalar. The formula is:

$x \cdot y = x^T y = \sum_{i=1}^d x_i y_i$

Semantic Similarity Example: In Natural Language Processing, words are represented as numeric vectors ('word embeddings'). Suppose the word king is mapped to $u = [1, 2]^T$ and the word queen to $v = [2, 1.5]^T$ . Their dot product is $u \cdot v = (1)(2) + (2)(1.5) = 2 + 3 = 5$ . Mathematically, the dot product is related to the angle $\theta$ between the vectors by $u \cdot v = \|u\| \|v\| \cos(\theta)$ . By calculating the cosine similarity $\cos(\theta) = \frac{u \cdot v}{\|u\| \|v\|}$ , we measure how semantically similar the words are. Here: $\cos(\theta) = \frac{5}{\sqrt{1^2+2^2} \cdot \sqrt{2^2+1.5^2}} = \frac{5}{\sqrt{5} \cdot \sqrt{6.25}} = \frac{5}{2.236 \cdot 2.5} \approx 0.89$ A cosine similarity of $0.89$ (close to $1.0$ ) shows that the model successfully understands these words are highly related!

A set of vectors spans a vector space if any coordinate in that space can be reached through a linear combination of those vectors. If no vector in a set can be written as a linear combination of the others, the vectors are linearly independent. A set of linearly independent vectors that spans a space forms a basis.

A Matrix Transforms Basis Vectors

Read diagram labels

original basis
columns of A
A →

Worked Example 1

Independence and Coordinates

Problem

Do $v_1=(1,2)^T$ and $v_2=(2,1)^T$ form a basis for the two-dimensional real coordinate space, $\mathbb{R}^2$ ? Express $b=(5,4)^T$ in that basis.

Step-by-step solution

1.Place the vectors in $A=\begin{bmatrix}1&2\\2&1\end{bmatrix}$ . Since $\det(A)=1-4=-3\neq0$ , its columns are independent and form a basis.

2.Solve $c_1+2c_2=5$ and $2c_1+c_2=4$ . This gives $c_2=2$ and $c_1=1$ .

Final answer and interpretation

Therefore $b=1v_1+2v_2$ .

A nonzero determinant certifies that two vectors span the plane without redundancy.

2. Matrix Mechanics and Linear Transformations

A matrix is a 2D grid of numbers representing a linear transformation. When we multiply a matrix $A$ by a vector $x$ (written as $Ax$ ), we are geometrically transforming the vector $x$ (scaling, rotating, or projecting it) into a new coordinate space.

Shape compatibility is non-negotiable: $A\in\mathbb{R}^{m\times n}$ can multiply $B\in\mathbb{R}^{n\times p}$ because the inner dimensions match, and the result has the outer dimensions: $(m\times n)(n\times p)\rightarrow(m\times p).$

• Matrix Multiplication ( $C = AB$ ): To compute elements of $C$ , we take the dot product of the $i$ -th row of $A$ with the $j$ -th column of $B$ : $C_{ij} = \sum_{k} A_{ik} B_{kj}$

• Determinant ( $\det(A)$ ): A scalar measuring how much the linear transformation scales the area or volume of space. If $\det(A) = 0$ , the matrix squashes space into a lower dimension (e.g. flatting a 2D plane into a 1D line), meaning the matrix is singular and cannot be inverted.

• Matrix Inverse ( $A^{-1}$ ): If an inverse exists, $A^{-1}A=I$ . It can express $Ax=b$ as $x=A^{-1}b$ , although numerical software normally solves the system directly.

For a machine-learning batch $X\in\mathbb{R}^{32\times10}$ and weights $W\in\mathbb{R}^{10\times4}$ , the score matrix $XW$ has shape $32\times4$ : one row per sample and one column per output. A bias $b\in\mathbb{R}^4$ is broadcast across all 32 rows.

Matrix Multiplication Shape Flow

Read diagram labels

A: 3 × 4
×
B: 4 × 2
=
C: 3 × 2
inner dimensions 4 must match

Jupyter Code Notebook Cell

Python 3 (ipykernel)

In [1]:

import numpy as np

# Define matrices A and B
A = np.array([[2, 1], [1, 3]])
B = np.array([[1, 0], [2, 1]])

# 1. Matrix Multiplication (AB)
C = np.dot(A, B)  # Or: A @ B
print("Matrix C = A @ B:\n", C)

# 2. Compute Determinant of A
det_A = np.linalg.det(A)
print("Determinant of A:", round(det_A, 2))

# 3. Compute Matrix Inverse of A
A_inv = np.linalg.inv(A)
print("Inverse of A:\n", A_inv)
print("A_inv @ A (Identity):\n", np.round(A_inv @ A))

Out [1]:

Matrix C = A @ B:
 [[4 1]
 [7 3]]
Determinant of A: 5.0
Inverse of A:
 [[ 0.6 -0.2]
 [-0.2  0.4]]
A_inv @ A (Identity):
 [[1. 0.]
 [0. 1.]]

Worked Example 1

Matrix–Vector Multiplication Entry by Entry

Problem

Compute $Ax$ for $A=\begin{bmatrix}2&-1\\3&4\end{bmatrix}$ and $x=(5,2)^T$ .

Step-by-step solution

1.The first output is the first row dotted with $x$ : $2(5)+(-1)(2)=8$ .

2.The second output is $3(5)+4(2)=23$ .

Final answer and interpretation

Thus $Ax=(8,23)^T$ ; the shape is $(2\times2)(2\times1)=(2\times1)$ .

Worked Example 2

Matrix–Matrix Multiplication and Shape

Problem

Multiply $A=\begin{bmatrix}1&2&0\\-1&3&1\end{bmatrix}$ by $B=\begin{bmatrix}2&1\\0&4\\5&-2\end{bmatrix}$ .

Step-by-step solution

1. $C_{11}=1(2)+2(0)+0(5)=2$ and $C_{12}=1(1)+2(4)+0(-2)=9$ .

2. $C_{21}=-1(2)+3(0)+1(5)=3$ and $C_{22}=-1(1)+3(4)+1(-2)=9$ .

Final answer and interpretation

$C=\begin{bmatrix}2&9\\3&9\end{bmatrix}$ . The shape trace is $(2\times3)(3\times2)=(2\times2)$ .

3. Vector and Matrix Norms

In machine learning, we need to measure the size, length, or magnitude of vectors and matrices (especially inside regularization terms to prevent overfitting). We calculate magnitude using mathematical functions called norms.

• $L_1$ Norm (Taxicab Norm): Sums the absolute values of the vector coordinates. It promotes sparsity (forcing weights to exactly zero, helpful for feature selection): $\|x\|_1 = \sum_{i=1}^d |x_i|$

• $L_2$ Norm (Euclidean Norm): Measures the standard straight-line distance from the origin. It is the most common norm used in weight decay and Ridge regressions: $\|x\|_2 = \sqrt{\sum_{i=1}^d x_i^2}$

• Frobenius Norm (for matrices): Measures the overall scale of a matrix $A \in \mathbb{R}^{m \times n}$ by taking the square root of the sum of all squared entries: $\|A\|_F = \sqrt{\sum_{i=1}^m \sum_{j=1}^n A_{ij}^2}$

Worked Example 1

Norms and Regularization

Problem

For $x=(-3,4,0)^T$ , calculate $\|x\|_1$ and $\|x\|_2$ .

Step-by-step solution

1. $\|x\|_1=|-3|+|4|+0=7$ .

Final answer and interpretation

$\|x\|_2=\sqrt{(-3)^2+4^2}=5$ .

$L_1$ and $L_2$ measure size differently, so their regularizers encourage different weight patterns.

Worked Example 2

Cumulative Linear-System Check

Problem

For $A=\begin{bmatrix}2&1\\1&3\end{bmatrix}$ and $b=(5,7)^T$ , solve $Ax=b$ and verify the result.

Step-by-step solution

1.Elimination gives $x_2=9/5$ and $x_1=8/5$ .

Final answer and interpretation

$A(8/5,9/5)^T=(16/5+9/5,8/5+27/5)^T=(5,7)^T$ .

Interactive Practice Quiz

Test your understanding with instant feedback

QUESTION 01

Calculate the dot product of vectors $x = [1, 3, -2]^T$ and $y = [4, 2, 5]^T$ :

QUESTION 02

Under what condition does a square matrix $A$ NOT have an inverse matrix $A^{-1}$ ?

QUESTION 03

Which vector norm promotes weight 'sparsity' (forcing parameter weights to be exactly zero) during regularization optimization?

QUESTION 04

Calculate the $L_2$ Norm of the coordinate vector $x = [3, -4]^T$ :

QUESTION 05

If a set of vectors spans a space and is completely linearly independent, what is this set called?

QUESTION 06

Compute the matrix-vector product $A v$ where $A = \begin{bmatrix} 2 & 1 \\ 3 & 4 \end{bmatrix}$ and $v = \begin{bmatrix} 1 \\ 2 \end{bmatrix}$ :

QUESTION 07

Compute the determinant of the matrix $A = \begin{bmatrix} 3 & 2 \\ 1 & 4 \end{bmatrix}$ :

QUESTION 08

Compute the Frobenius Norm of the matrix $A = \begin{bmatrix} 1 & 2 \\ 3 & 0 \end{bmatrix}$ :

QUESTION 09

If two vectors $u$ and $v$ have a cosine similarity of $1.0$ , what does this mathematically represent?

QUESTION 10

Linear Algebra and Matrix Mechanics for Machine Learning

Core Concepts Covered

Local Setup Recommendation

1. Vectors, Dot Products, and Semantic Similarities

Independence and Coordinates

2. Matrix Mechanics and Linear Transformations

Matrix–Vector Multiplication Entry by Entry

Matrix–Matrix Multiplication and Shape

3. Vector and Matrix Norms

Norms and Regularization

Cumulative Linear-System Check

Interactive Practice Quiz

Calculate the dot product of vectors $x = [1, 3, -2]^T$ and $y = [4, 2, 5]^T$ :

Under what condition does a square matrix $A$ NOT have an inverse matrix $A^{-1}$ ?

Which vector norm promotes weight 'sparsity' (forcing parameter weights to be exactly zero) during regularization optimization?

Calculate the $L_2$ Norm of the coordinate vector $x = [3, -4]^T$ :

If a set of vectors spans a space and is completely linearly independent, what is this set called?

Compute the matrix-vector product $A v$ where $A = \begin{bmatrix} 2 & 1 \\ 3 & 4 \end{bmatrix}$ and $v = \begin{bmatrix} 1 \\ 2 \end{bmatrix}$ :

Compute the determinant of the matrix $A = \begin{bmatrix} 3 & 2 \\ 1 & 4 \end{bmatrix}$ :

Compute the Frobenius Norm of the matrix $A = \begin{bmatrix} 1 & 2 \\ 3 & 0 \end{bmatrix}$ :

If two vectors $u$ and $v$ have a cosine similarity of $1.0$ , what does this mathematically represent?

What is the key characteristic of an Identity Matrix $I$ ?

Further Readings

Linear Algebra and Matrix Mechanics for Machine Learning

Core Concepts Covered

Local Setup Recommendation

1. Vectors, Dot Products, and Semantic Similarities

Independence and Coordinates

2. Matrix Mechanics and Linear Transformations

Matrix–Vector Multiplication Entry by Entry

Matrix–Matrix Multiplication and Shape

3. Vector and Matrix Norms

Norms and Regularization

Cumulative Linear-System Check

Interactive Practice Quiz

Calculate the dot product of vectors x=[1,3,−2]Tx = [1, 3, -2]^Tx=[1,3,−2]T and y=[4,2,5]Ty = [4, 2, 5]^Ty=[4,2,5]T:

Under what condition does a square matrix AAA NOT have an inverse matrix A−1A^{-1}A−1?

Which vector norm promotes weight 'sparsity' (forcing parameter weights to be exactly zero) during regularization optimization?

Calculate the L2L_2L2​ Norm of the coordinate vector x=[3,−4]Tx = [3, -4]^Tx=[3,−4]T:

If a set of vectors spans a space and is completely linearly independent, what is this set called?

Compute the matrix-vector product AvA vAv where A=[2134]A = \begin{bmatrix} 2 & 1 \\ 3 & 4 \end{bmatrix}A=[23​14​] and v=[12]v = \begin{bmatrix} 1 \\ 2 \end{bmatrix}v=[12​]:

Compute the determinant of the matrix A=[3214]A = \begin{bmatrix} 3 & 2 \\ 1 & 4 \end{bmatrix}A=[31​24​]:

Compute the Frobenius Norm of the matrix A=[1230]A = \begin{bmatrix} 1 & 2 \\ 3 & 0 \end{bmatrix}A=[13​20​]:

If two vectors uuu and vvv have a cosine similarity of 1.01.01.0, what does this mathematically represent?

What is the key characteristic of an Identity Matrix III?

Further Readings

Calculate the dot product of vectors $x = [1, 3, -2]^T$ and $y = [4, 2, 5]^T$ :

Under what condition does a square matrix $A$ NOT have an inverse matrix $A^{-1}$ ?

Calculate the $L_2$ Norm of the coordinate vector $x = [3, -4]^T$ :

Compute the matrix-vector product $A v$ where $A = \begin{bmatrix} 2 & 1 \\ 3 & 4 \end{bmatrix}$ and $v = \begin{bmatrix} 1 \\ 2 \end{bmatrix}$ :

Compute the determinant of the matrix $A = \begin{bmatrix} 3 & 2 \\ 1 & 4 \end{bmatrix}$ :

Compute the Frobenius Norm of the matrix $A = \begin{bmatrix} 1 & 2 \\ 3 & 0 \end{bmatrix}$ :

If two vectors $u$ and $v$ have a cosine similarity of $1.0$ , what does this mathematically represent?

What is the key characteristic of an Identity Matrix $I$ ?