Matrix Calculus for Machine Learning

A deep learning model may contain thousands or millions of parameters arranged in vectors, matrices, and tensors. Writing a separate derivative for each one would quickly become unmanageable. Matrix Calculus lets us describe all of those related derivatives compactly and compute them together.

When we take the derivative of a scalar function $f(x)$ with respect to a column vector $x \in \mathbb{R}^d$, the result is a gradient vector of partial derivatives:

This lesson uses the column-gradient convention, so $$\nabla_x f(x)=\begin{bmatrix}\frac{\partial f}{\partial x_1}\\\frac{\partial f}{\partial x_2}\\\vdots\\\frac{\partial f}{\partial x_d}\end{bmatrix}\in\mathbb{R}^{d\times1}.$$ Some references transpose scalar-by-vector derivatives; consistency matters more than the choice.

Expand $f(x)=x^TAx=\sum_i\sum_jx_iA_{ij}x_j$. A coordinate $x_k$ appears once as the left factor and once as the right factor, giving $\frac{\partial f}{\partial x_k}=\sum_jA_{kj}x_j+\sum_i x_iA_{ik}$. Stacking the coordinates yields the general identity below.

For the column-gradient convention used here, the general identity is $$\nabla_x(x^TAx)=(A+A^T)x.$$ It reduces to $2Ax$ when $A$ is symmetric. The gradient has the same $d\times1$ shape as $x$.

When dealing with vector-valued functions (where the output of a layer is a vector, such as $y = f(x) \in \mathbb{R}^m$), we represent all possible first-order partial derivatives inside the Jacobian Matrix.

The Jacobian $J \in \mathbb{R}^{m \times d}$ of a function $f: \mathbb{R}^d \to \mathbb{R}^m$ contains the partial derivative of every output component with respect to every input parameter:

$$J = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \dots & \frac{\partial f_1}{\partial x_d} \\ \vdots & \ddots & \vdots \\ \frac{\partial f_m}{\partial x_1} & \dots & \frac{\partial f_m}{\partial x_d} \end{bmatrix}$$

The Hessian Matrix ($H$): A square matrix containing all second-order partial derivatives of a scalar function. While the gradient vector measures the slope (first-order change), the Hessian measures the local curvature (second-order change) of the loss surface:

$$H_{ij} = \frac{\partial^2 f}{\partial x_i \partial x_j}$$

If the Hessian is positive definite ($H \succ 0$, all eigenvalues are positive), the loss landscape curves upward like a bowl, indicating a local minimum. If eigenvalues are mixed (some positive, some negative), the landscape represents a saddle point, which stalls vanilla gradient descent.

A neural network is an enormous composite function. To optimize it, we compute the partial derivative of the final loss $L$ with respect to every internal weight $w$ using computational graphs and backpropagation.

• Computational Graphs: Represent mathematical formulas as directed graphs where nodes are mathematical operators and edges are tensors flowing forward.

• The Backpropagation Trace: Let's trace a concrete example. Suppose we have a node computing $z = wx$ and a loss $L = f(z)$. During the forward pass, we calculate $z$ and $L$. During the backward pass, our goal is to find the gradient $\frac{\partial L}{\partial w}$. We apply the multivariable Chain Rule:

$$\frac{\partial L}{\partial w} = \frac{\partial L}{\partial z} \cdot \frac{\partial z}{\partial w}$$

Since $z = wx$, the local derivative is $\frac{\partial z}{\partial w} = x$. Therefore, the gradient passed upstream is $\frac{\partial L}{\partial w}=\frac{\partial L}{\partial z}x$. Reverse-mode automatic differentiation reuses these local products instead of constructing a full Jacobian at every node.