Optimization for Machine Learning

In machine learning, optimization is the mathematical process of adjusting a model's internal parameters (weights and biases) to minimize its overall error (Loss Function). The shape of this loss surface determines how easily our optimization algorithms can locate the absolute optimal parameter set.

• Convex Optimization: A function $f(x)$ is convex if a line segment drawn between any two points on its graph lies entirely above or on the graph. Mathematically, a function is convex if its second-order Hessian matrix is positive semi-definite ($H \succeq 0$) everywhere. Convex landscapes are highly desirable because they have a unique global minimum and zero trapping local minima, guaranteeing that standard gradient descent will always converge to the absolute optimum.

• Non-Convex Optimization: Most modern deep learning loss landscapes are non-convex. They represent highly complex, non-linear terrains with millions of local minima (sub-optimal basins), flat plateaus (which slow down updates), and high-dimensional Saddle Points (where the slope/gradient is zero, but the curvature is mixed—curving up in some directions and down in others). Adaptive optimizers are required to navigate and bypass these hurdles successfully.

The standard way to navigate optimization landscapes is Gradient Descent. The algorithm calculates the gradient of the loss with respect to parameters, and takes step offsets in the downhill direction governed by a learning rate hyperparameter ($\eta$):

$$w \leftarrow w - \eta \nabla_w \text{Loss}$$

There are three main variants based on how much data is parsed before performing a weight update:

• Batch Gradient Descent: Computes the exact gradient of the loss using the *entire* dataset. It is computationally expensive and slow for massive data, but yields stable, deterministic steps.

• Stochastic Gradient Descent (SGD): Computes the gradient and performs an update using a single random sample at each step. It is extremely fast but highly noisy and erratic, which helps bounce out of shallow local minima.

• Mini-Batch Gradient Descent: The industry standard. Computes the gradient over a moderate subset of data (a 'batch', e.g., 32, 64, or 128 samples). It combines the speed of parallelized hardware matrix operations with the stable convergence of batch updates.

To accelerate convergence and prevent models from getting stuck in saddle points, we use adaptive optimizers:

• Momentum: Introduces a velocity term that accumulates previous gradient steps, helping accelerate down steep ravines and dampening erratic oscillations.

• RMSprop: Restricts extreme updates by dividing the learning rate by a running average of the squared gradients, adapting step sizes feature-by-feature.

• Adam (Adaptive Moment Estimation): The industry gold standard. It combines momentum and RMSprop, tracking both the first moment (average velocity) and second moment (uncentered variance) of gradients, dynamically scaling learning rates for each weight parameter.

• Regularization Math (L1 vs. L2): To prevent overfitting, we add mathematical penalty terms to our loss function:

- $L_2$ Regularization (Ridge / Weight Decay): Adds a squared magnitude penalty to prevent weights from growing excessively large: $$\text{Loss}_{L2} = \text{Loss}_0 + \frac{\lambda}{2} \|w\|_2^2 = \text{Loss}_0 + \frac{\lambda}{2} \sum w_j^2$$ *The Elastic Band Analogy:* $L_2$ acts like a set of elastic bands pulling all weights toward zero smoothly. It keeps weights small and uniform but rarely forces them to be exactly zero.

- $L_1$ Regularization (Lasso): Adds an absolute magnitude penalty: $$\text{Loss}_{L1} = \text{Loss}_0 + \lambda \|w\|_1 = \text{Loss}_0 + \lambda \sum |w_j|$$ *The Feature Scissor:* Because the absolute value function has sharp corners at zero, $L_1$ forces redundant weights to be exactly zero, acting as an automated feature selection tool that strips out uninformative features.