Calculus for Machine Learning

Training a model means changing its parameters in a direction that reduces error. Calculus gives us the language for describing that change, beginning with limits, continuity, and derivatives of one variable.

A limit describes the behavior of a function as its input approaches a specific target value. Formally, we write:

$$\lim_{x \to c} f(x) = L$$

A function is continuous if its limit as $x$ approaches $c$ equals $f(c)$. Differentiability is stronger: a derivative must exist locally. Smooth losses give ordinary gradient descent a well-defined direction, but useful objectives such as MAE and $L_1$ penalties have isolated kinks and can still be optimized with subgradients or related methods.

The derivative of a function measures its instantaneous rate of change. Geometrically, it is the slope of the tangent line to the curve at a specific point. Mathematically, it is defined as:

$$f'(x) = \lim_{h \to 0} \frac{f(x+h) - f(x)}{h}$$

To find derivatives of complex, nested networks, we use four standard rules:

• Power Rule: $\frac{d}{dx}(x^n) = n x^{n-1}$.

• Product Rule: $\frac{d}{dx}(f(x)g(x)) = f'(x)g(x) + f(x)g'(x)$.

• Quotient Rule: $\frac{d}{dx}\left(\frac{f(x)}{g(x)}\right) = \frac{f'(x)g(x) - f(x)g'(x)}{(g(x))^2}$.

• Chain Rule (for composite layers): If $y = f(g(x))$, let $u = g(x)$, then: $$\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx} = f'(g(x)) \cdot g'(x)$$,

Chain Rule Example: Suppose we have a nested function $y = (3x^2 + 1)^4$. Let the inner function be $u = g(x) = 3x^2 + 1$ (with derivative $g'(x) = 6x$), and the outer function be $y = f(u) = u^4$ (with derivative $f'(u) = 4u^3$). Applying the Chain Rule: $$\frac{dy}{dx} = 4(3x^2 + 1)^3 \cdot (6x) = 24x(3x^2 + 1)^3$$

In machine learning, models have millions of input parameters (weights and biases). To measure how changing each parameter affects the overall loss, we use partial derivatives.

A partial derivative, written as $\frac{\partial f}{\partial x_i}$, measures the rate of change of a multivariable function $f(x_1, x_2, ..., x_n)$ with respect to one variable $x_i$, while holding all other variables constant.

The Gradient of a multivariable function, denoted by $\nabla f(x)$ (read as 'nabla f'), is a vector of all its partial derivatives: $$\nabla f(x) = \begin{bmatrix} \frac{\partial f}{\partial x_1} \\ \frac{\partial f}{\partial x_2} \\ \vdots \\ \frac{\partial f}{\partial x_n} \end{bmatrix}$$

Concrete Gradient Example: Let $f(x, y) = 3x^2 y + e^y$. Let's compute its gradient vector $\nabla f(x, y)$: - The partial derivative with respect to $x$ (treating $y$ as a constant) is: $\frac{\partial f}{\partial x} = 6xy$. - The partial derivative with respect to $y$ (treating $x$ as a constant) is: $\frac{\partial f}{\partial y} = 3x^2 + e^y$. - Therefore, the gradient vector is: $$\nabla f(x, y) = \begin{bmatrix} 6xy \\ 3x^2 + e^y \end{bmatrix}$$ At the specific coordinate point $(1, 0)$, the gradient is evaluated as: $\nabla f(1, 0) = \begin{bmatrix} 6(1)(0) \\ 3(1)^2 + e^0 \end{bmatrix} = \begin{bmatrix} 0 \\ 4 \end{bmatrix}$.

A unit vector $u$ defines a direction. The directional derivative is $D_u f=\nabla f^T u$; it is largest when $u$ points along the gradient and most negative in the opposite direction.

Geometrically, the gradient vector points in the direction of the steepest ascent (fastest increase). By stepping in the exact opposite direction of the gradient ($-\nabla f(x)$), gradient descent moves down the steepest path to find the local minimum.

Unconstrained optimization is the process of locating a function's maximum or minimum points by finding where its gradient is equal to the zero vector: $\nabla f(x) = 0$.

In many machine learning algorithms, however, our parameters are subject to strict constraints. For instance, Support Vector Machines (SVMs) must maximize decision margins subject to classification boundaries. We solve these constrained systems using the Method of Lagrange Multipliers.

To find the local minimum of a function $f(x, y)$ subject to an equality constraint $g(x, y) = 0$, we introduce a new variable $\lambda$ (the Lagrange Multiplier) and construct the Lagrangian function: $$\mathcal{L}(x, y, \lambda) = f(x, y) - \lambda g(x, y)$$,

We find critical points of the Lagrangian by setting its gradient with respect to all variables ($x, y,$ and $\lambda$) to zero: $$\nabla_x \mathcal{L} = 0, \quad \nabla_y \mathcal{L} = 0, \quad \frac{\partial \mathcal{L}}{\partial \lambda} = 0$$,

The Hessian distinguishes curvature at a stationary point. A positive-definite Hessian indicates a strict local minimum, a negative-definite Hessian a strict local maximum, and mixed-sign curvature a saddle point.

Constraint Example: Suppose we want to maximize our model's feature coverage area $f(x, y) = xy$ subject to a perimeter boundary constraint $g(x, y) = 2x + 2y - 20 = 0$. We construct the Lagrangian: $$\mathcal{L}(x, y, \lambda) = xy - \lambda(2x + 2y - 20)$$ Taking partial derivatives and setting them to zero: $$\frac{\partial \mathcal{L}}{\partial x} = y - 2\lambda = 0 \implies y = 2\lambda$$ $$\frac{\partial \mathcal{L}}{\partial y} = x - 2\lambda = 0 \implies x = 2\lambda$$ Since $x = 2\lambda$ and $y = 2\lambda$, we find $x = y$. Substituting into the constraint $2(2\lambda) + 2(2\lambda) = 20 \implies 8\lambda = 20 \implies \lambda = 2.5$. This yields the optimal dimensions $x = 5, y = 5$.