Optimization for Machine Learning

Training chooses parameters $\theta$ to minimize an objective $J(\theta)$. A per-example loss measures prediction error, a regularizer expresses a preference such as small weights, and the objective combines them: $$J(\theta)=\frac{1}{n}\sum_{i=1}^n\ell(y_i,f_\theta(x_i))+\lambda R(\theta).$$ A metric reports behavior we care about, such as accuracy, and need not be differentiable or identical to the optimized loss.

A function is convex when every chord lies above its graph. For differentiable convex objectives, every local minimum is global; strict convexity gives at most one minimizer. A positive-semidefinite Hessian is a useful twice-differentiable test. Deep networks are generally non-convex and include saddles and flat directions.

Constraints restrict feasible parameters, for example $\|w\|_2\leq c$. Unit 6 introduced Lagrange multipliers; this unit focuses on iterative algorithms and diagnostics.

For a regression residual $e=\hat y-y$, MSE uses $e^2$ and strongly penalizes outliers; MAE uses $|e|$ and is more robust but has a kink at zero. Huber loss is quadratic for $|e|\leq\delta$ and linear outside: $$\ell_\delta(e)=\begin{cases}\tfrac{1}{2}e^2,&|e|\leq\delta,\\\delta(|e|-\tfrac{1}{2}\delta),&|e|>\delta.\end{cases}$$

Binary cross-entropy is $-[y\log p+(1-y)\log(1-p)]$. For mutually exclusive classes, softmax gives $p_j=e^{z_j}/\sum_ke^{z_k}$, and cross-entropy for true class $c$ is $-\log p_c$. Unit 12 explains its information-theoretic meaning.

Hinge loss for $y\in\{-1,+1\}$ and score $s$ is $\max(0,1-ys)$. It becomes zero once the correct-class score clears the unit margin, matching Unit 13's SVM geometry.

Gradient descent updates $w_{t+1}=w_t-\eta\nabla J(w_t)$. Batch descent uses every example for a stable but costly step. SGD uses one random example for a cheap, noisy estimate. Mini-batches average a moderate group, exploiting vectorized hardware while retaining useful gradient noise.

The learning rate $\eta$ controls step size. Too small wastes computation; too large can oscillate or diverge. Shuffle data and report whether a logged loss is per example, per batch, or an epoch average.

Momentum accumulates velocity, $v_t=\beta v_{t-1}+(1-\beta)g_t$. RMSprop tracks $s_t=\rho s_{t-1}+(1-\rho)g_t^2$ and scales the step by $1/(\sqrt{s_t}+\epsilon)$, damping coordinates with persistently large gradients.

Adam combines both: $m_t=\beta_1m_{t-1}+(1-\beta_1)g_t$, $v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2$, and $$w_t=w_{t-1}-\eta\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}.$$ Warmup, step decay, cosine schedules, or plateau reductions can improve stability.

Convergence may mean small gradients, small objective changes, exhausted compute, or best validation performance. State the criterion and inspect gradient norms, validation curves, and sensitivity to random seeds.

$L_2$ regularization adds $\frac\lambda2\|w\|_2^2$, producing gradient $\lambda w$ and discouraging large weights. $L_1$ adds $\lambda\|w\|_1$ and promotes sparsity. Decoupled weight decay is not always identical to an $L_2$ penalty with adaptive optimizers.

Early stopping saves the checkpoint with the best validation score and halts after a patience window. Both losses staying high suggests underfitting; training loss falling while validation loss rises suggests overfitting. Exploding gradients suggest clipping or a lower rate; flat training may indicate poor scaling, saturation, or a rate that is too small.