Calculus for Machine Learning

Calculus is the mathematical engine that powers modern machine learning optimization. To understand how complex deep learning networks iteratively minimize error, we must start with the building blocks of limits, continuity, and single-variable derivatives.

A limit describes the behavior of a function as its input approaches a specific target value. Formally, we write:

$$\lim_{x \to c} f(x) = L$$

A function is continuous if its limit as $x$ approaches $c$ is equal to its actual value $f(c)$ at that point. Differentiability is a stronger condition: a function is differentiable if its derivative exists at every point in its domain. Loss functions must be continuous and differentiable so that gradient descent can compute smooth mathematical updates without sudden breaks.

The derivative of a function measures its instantaneous rate of change. Geometrically, it is the slope of the tangent line to the curve at a specific point. Mathematically, it is defined as:

$$f'(x) = \lim_{h \to 0} \frac{f(x+h) - f(x)}{h}$$

To find derivatives of complex, nested networks, we use four standard rules:

• Power Rule: $\frac{d}{dx}(x^n) = n x^{n-1}$.

• Product Rule: $\frac{d}{dx}(f(x)g(x)) = f'(x)g(x) + f(x)g'(x)$.

• Quotient Rule: $\frac{d}{dx}\left(\frac{f(x)}{g(x)}\right) = \frac{f'(x)g(x) - f(x)g'(x)}{(g(x))^2}$.

• Chain Rule (for composite layers): If $y = f(g(x))$, let $u = g(x)$, then: $$\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx} = f'(g(x)) \cdot g'(x)$$,

Chain Rule Example: Suppose we have a nested function $y = (3x^2 + 1)^4$. Let the inner function be $u = g(x) = 3x^2 + 1$ (with derivative $g'(x) = 6x$), and the outer function be $y = f(u) = u^4$ (with derivative $f'(u) = 4u^3$). Applying the Chain Rule: $$\frac{dy}{dx} = 4(3x^2 + 1)^3 \cdot (6x) = 24x(3x^2 + 1)^3$$

In machine learning, models have millions of input parameters (weights and biases). To measure how changing *each* parameter affects the overall loss, we use partial derivatives.

A partial derivative, written as $\frac{\partial f}{\partial x_i}$, measures the rate of change of a multivariable function $f(x_1, x_2, ..., x_n)$ with respect to one variable $x_i$, while holding all other variables constant.

The Gradient of a multivariable function, denoted by $\nabla f(x)$ (read as 'nabla f'), is a vector of all its partial derivatives: $$\nabla f(x) = \begin{bmatrix} \frac{\partial f}{\partial x_1} \\ \frac{\partial f}{\partial x_2} \\ \vdots \\ \frac{\partial f}{\partial x_n} \end{bmatrix}$$

Concrete Gradient Example: Let $f(x, y) = 3x^2 y + e^y$. Let's compute its gradient vector $\nabla f(x, y)$: - The partial derivative with respect to $x$ (treating $y$ as a constant) is: $\frac{\partial f}{\partial x} = 6xy$. - The partial derivative with respect to $y$ (treating $x$ as a constant) is: $\frac{\partial f}{\partial y} = 3x^2 + e^y$. - Therefore, the gradient vector is: $$\nabla f(x, y) = \begin{bmatrix} 6xy \\ 3x^2 + e^y \end{bmatrix}$$ At the specific coordinate point $(1, 0)$, the gradient is evaluated as: $\nabla f(1, 0) = \begin{bmatrix} 6(1)(0) \\ 3(1)^2 + e^0 \end{bmatrix} = \begin{bmatrix} 0 \\ 4 \end{bmatrix}$.

Geometrically, the gradient vector points in the direction of the steepest ascent (fastest increase). By stepping in the exact *opposite* direction of the gradient ($-\nabla f(x)$), gradient descent moves down the steepest path to find the local minimum.

Unconstrained optimization is the process of locating a function's maximum or minimum points by finding where its gradient is equal to the zero vector: $\nabla f(x) = 0$.

In many machine learning algorithms, however, our parameters are subject to strict constraints. For instance, Support Vector Machines (SVMs) must maximize decision margins subject to classification boundaries. We solve these constrained systems using the Method of Lagrange Multipliers.

To find the local minimum of a function $f(x, y)$ subject to an equality constraint $g(x, y) = 0$, we introduce a new variable $\lambda$ (the Lagrange Multiplier) and construct the Lagrangian function: $$\mathcal{L}(x, y, \lambda) = f(x, y) - \lambda g(x, y)$$,

We find critical points of the Lagrangian by setting its gradient with respect to all variables ($x, y,$ and $\lambda$) to zero: $$\nabla_x \mathcal{L} = 0, \quad \nabla_y \mathcal{L} = 0, \quad \frac{\partial \mathcal{L}}{\partial \lambda} = 0$$,

Constraint Example: Suppose we want to maximize our model's feature coverage area $f(x, y) = xy$ subject to a perimeter boundary constraint $g(x, y) = 2x + 2y - 20 = 0$. We construct the Lagrangian: $$\mathcal{L}(x, y, \lambda) = xy - \lambda(2x + 2y - 20)$$ Taking partial derivatives and setting them to zero: $$\frac{\partial \mathcal{L}}{\partial x} = y - 2\lambda = 0 \implies y = 2\lambda$$ $$\frac{\partial \mathcal{L}}{\partial y} = x - 2\lambda = 0 \implies x = 2\lambda$$ Since $x = 2\lambda$ and $y = 2\lambda$, we find $x = y$. Substituting into the constraint $2(2\lambda) + 2(2\lambda) = 20 \implies 8\lambda = 20 \implies \lambda = 2.5$. This yields the optimal dimensions $x = 5, y = 5$, proving mathematically that a square maximizes area subject to a perimeter constraint!