Matrix Calculus for Machine Learning

In deep learning, we optimize functions that operate on vectors, matrices, and tensors. Single-variable calculus is too tedious for tracking these complex high-dimensional parameter systems. We use Matrix Calculus to write derivatives compactly and compute thousands of partial gradients simultaneously.

When we take the derivative of a scalar function $f(x)$ with respect to a column vector $x \in \mathbb{R}^d$, the result is a gradient vector of partial derivatives:

$$\nabla_x f(x) = \frac{\partial f}{\partial x} = \begin{bmatrix} \frac{\partial f}{\partial x_1} & \frac{\partial f}{\partial x_2} & \dots & \frac{\partial f}{\partial x_d} \end{bmatrix}$$

Let's trace a mathematical proof for a symmetric quadratic form $f(x) = x^T A x$ where $A \in \mathbb{R}^{d \times d}$ is symmetric ($A = A^T$). Expanding this expression using summation notations yields: $$f(x) = \sum_{i=1}^d \sum_{j=1}^d x_i A_{ij} x_j$$ If we take the partial derivative with respect to a single component $x_k$: $$\frac{\partial f}{\partial x_k} = \frac{\partial}{\partial x_k} \left( x_k^2 A_{kk} + x_k \sum_{j \neq k} A_{kj} x_j + x_k \sum_{i \neq k} x_i A_{ik} \right)$$ $$\frac{\partial f}{\partial x_k} = 2 x_k A_{kk} + \sum_{j \neq k} A_{kj} x_j + \sum_{i \neq k} x_i A_{ik}$$ Since $A$ is symmetric ($A_{ik} = A_{ki}$), the two summation terms are identical, merging into a single clean dot product: $$\frac{\partial f}{\partial x_k} = 2 \sum_{j=1}^d A_{kj} x_j = 2 (A x)_k$$ When we stack these partial results back into a full row vector, we derive our compact gradient identity: $$\nabla_x(x^T A x) = 2 x^T A$$

This identity is used repeatedly when deriving the analytical solutions of Least Squares and Ridge regressions!

When dealing with vector-valued functions (where the output of a layer is a vector, such as $y = f(x) \in \mathbb{R}^m$), we represent all possible first-order partial derivatives inside the Jacobian Matrix.

The Jacobian $J \in \mathbb{R}^{m \times d}$ of a function $f: \mathbb{R}^d \to \mathbb{R}^m$ contains the partial derivative of every output component with respect to every input parameter:

$$J = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \dots & \frac{\partial f_1}{\partial x_d} \\ \vdots & \ddots & \vdots \\ \frac{\partial f_m}{\partial x_1} & \dots & \frac{\partial f_m}{\partial x_d} \end{bmatrix}$$

The Hessian Matrix ($H$): A square matrix containing all second-order partial derivatives of a scalar function. While the gradient vector measures the slope (first-order change), the Hessian measures the local curvature (second-order change) of the loss surface:

$$H_{ij} = \frac{\partial^2 f}{\partial x_i \partial x_j}$$

If the Hessian is positive definite ($H \succ 0$, all eigenvalues are positive), the loss landscape curves upward like a bowl, indicating a local minimum. If eigenvalues are mixed (some positive, some negative), the landscape represents a saddle point, which stalls vanilla gradient descent.

A neural network is essentially an enormous composite function. To optimize its parameters, we must compute the partial derivative of the final Loss ($L$) with respect to every single internal weight weight ($w$). We do this systematically using computational graphs and the Backpropagation Algorithm.

• Computational Graphs: Represent mathematical formulas as directed graphs where nodes are mathematical operators and edges are tensors flowing forward.

• The Backpropagation Trace: Let's trace a concrete example. Suppose we have a node computing $z = wx$ and a loss $L = f(z)$. During the forward pass, we calculate $z$ and $L$. During the backward pass, our goal is to find the gradient $\frac{\partial L}{\partial w}$. We apply the multivariable Chain Rule:

$$\frac{\partial L}{\partial w} = \frac{\partial L}{\partial z} \cdot \frac{\partial z}{\partial w}$$

Since $z = wx$, the local derivative is $\frac{\partial z}{\partial w} = x$. Therefore, the gradient passed back upstream is: $$\frac{\partial L}{\partial w} = \frac{\partial L}{\partial z} \cdot x$$ This shows that backpropagation is simply the recursive execution of local node derivative multiplications flowing backward through our graph!