Classical Machine Learning Foundations

Machine Learning algorithms are broadly categorized based on the nature of the training data and feedback loops they receive:

• Supervised Learning: The model is trained on labeled historical examples containing both features and explicit targets. If the target is a continuous real number (like predicting house prices), it is a Regression problem. If the target is a discrete label (like classifying an email as Spam or Inbox), it is a Classification problem.

• Unsupervised Learning: The dataset contains no target labels, so the goal is to discover structure rather than predict a known target. Unit 14 develops this branch in depth through clustering, density estimation, dimensionality reduction, and pattern evaluation.

• Reinforcement Learning: An autonomous agent learns optimal decision-making policies by taking actions within an environment to maximize a scalar feedback reward over time (such as training an autopilot or a game-playing engine).

• Linear Regression: For features $x$, weights $w$, and intercept $b$, the model predicts a continuous target with $\hat{y}=w^Tx+b$. A residual is the signed error $e=y-\hat{y}$. Training commonly minimizes the Sum of Squared Errors (SSE), which makes large misses especially costly:

$$\text{SSE} = \sum_{i=1}^n (y_i - (w^T x_i + b))^2$$

For a one-feature house-price model $\hat{y}=2x+10$, a home with $x=3$ produces $\hat{y}=16$. If its observed price is $y=18$, the residual is $18-16=2$ and this example contributes $2^2=4$ to SSE. Linear regression is appropriate when the target is continuous and an approximately additive relationship is a useful baseline.

• Logistic Regression: Binary logistic regression first computes a logit $z=w^Tx+b$, then converts it into a class probability with the sigmoid function:

$$p(y=1\mid x)=\sigma(z)=\frac{1}{1+e^{-z}}$$

A threshold turns that probability into a label. With the common threshold $0.5$, $z=1.2$ gives $\sigma(1.2)\approx0.769$, so the predicted class is $1$. Thresholds are decisions rather than fixed laws: a medical screener may lower its threshold to catch more possible cases, accepting more false alarms.

Training minimizes Binary Cross-Entropy, $\ell=-[y\log(p)+(1-y)\log(1-p)]$. For the example above with $y=1$ and $p=0.769$, the loss is $-\log(0.769)\approx0.263$. A confidently wrong probability receives a much larger penalty. Logistic regression is therefore a useful interpretable baseline for tasks such as spam detection or disease-risk classification.

Supervised learning is not limited to one straight regression line. SVMs can build maximum-margin boundaries, while trees and forests learn conditional rules and feature interactions.

• Support Vector Machines (SVMs): A linear SVM predicts from the sign of $w^Tx+b$ and chooses the separating hyperplane that maximizes the margin. The closest training examples are the support vectors; they anchor the solution, while moving a point far from the boundary usually has no effect. In the canonical scaling, the distance between the two supporting hyperplanes is $2/\lVert w\rVert$, so minimizing $\lVert w\rVert$ produces a wider margin.

The perpendicular distance from a point $x$ to a boundary $w^Tx+b=0$ is $|w^Tx+b|/\lVert w\rVert$. For $2x_1-x_2+1=0$ and $x=(1,1)$, the distance is $|2(1)-1+1|/\sqrt{2^2+(-1)^2}=2/\sqrt{5}\approx0.894$. This geometry is why unscaled features can distort an SVM boundary.

• Hard and Soft Margins: A hard-margin SVM requires every training point to be correctly separated and works only when the classes are linearly separable. A soft-margin SVM introduces slack variables that permit points to enter the margin or be misclassified. The $C$ hyperparameter prices those violations: large $C$ favors fewer training errors and a potentially narrower boundary; small $C$ accepts more violations to obtain stronger regularization and a wider margin.

• The Kernel Trick (Intuitive Analogy): Imagine drawing a circle of red dots on a sheet of paper, surrounded by a larger ring of blue dots. You cannot draw a straight line on that 2D paper to separate them. However, what if you 'pop' the center of the paper upward, pushing the red dots into 3D space? You can now pass a flat 2D sheet of glass (a linear hyperplane) horizontally between the floating red dots and the flat blue dots. The Kernel Trick measures similarities as if points had been mapped into that richer space, without explicitly calculating every high-dimensional coordinate. The radial-basis-function (RBF) kernel is a common choice; its $\gamma$ setting controls how local each point's influence is.

With an RBF kernel, a small $\gamma$ gives each example broad influence and tends to create a smoother boundary; a large $\gamma$ makes influence highly local and can produce a complicated boundary that overfits. Tune $C$ and $\gamma$ together using validation data. Put scaling inside a pipeline so it learns only from training data. SVMs are especially useful for small-to-medium, high-dimensional datasets such as text or biological measurements, but kernel training can become costly on very large datasets.

• Decision Trees: A tree recursively asks threshold questions such as age <= 30 and chooses splits that make the child nodes purer. For a parent with $6$ positive and $4$ negative examples, the Gini impurity is $1-0.6^2-0.4^2=0.48$. Suppose a candidate split creates a pure four-example left node and a right node with $2$ positive and $4$ negative examples. The right-node impurity is $1-(2/6)^2-(4/6)^2=4/9$, so the weighted child impurity is $(4/10)(0)+(6/10)(4/9)\approx0.267$. The Gini reduction is $0.48-0.267=0.213$, which the tree compares with other candidate splits.

Trees naturally model non-linear interactions and can classify loan risk or regress a home's price, but an unrestricted tree can memorize noise. Depth limits, minimum leaf sizes, and pruning control this variance. Entropy and information gain provide an alternative split criterion; Unit 12 develops their information-theory foundations.

• Random Forests: A forest trains many trees on different bootstrap samples, created by sampling the training rows with replacement. Each split also considers only a random subset of features, preventing the same dominant predictor from making every tree alike. For classification, trees vote; for regression, their numeric predictions are averaged. If five fraud-classification trees vote $[1,1,0,1,0]$, the forest predicts class $1$ by a $3$-to-$2$ majority. Combining diverse, imperfect trees cancels some of their individual fluctuations, reducing variance and usually generalizing better than one deep tree.

To evaluate our models, we construct a Confusion Matrix tracking True Positives ($TP$), False Positives ($FP$), True Negatives ($TN$), and False Negatives ($FN$). From these, we calculate crucial evaluation metrics:

• Accuracy: Overall proportion of correct predictions: $\frac{TP+TN}{TP+FP+TN+FN}$.

• Precision: Out of all positive predictions, how many were actually positive? Highly critical when false positives are expensive (e.g., spam filters): $$\text{Precision} = \frac{TP}{TP + FP}$$

• Recall (Sensitivity): Out of all actual positive samples, how many did the model find? Highly critical when false negatives are dangerous (e.g., medical diagnoses): $$\text{Recall} = \frac{TP}{TP + FN}$$

• Specificity: Out of all actual negative samples, how many did the model correctly reject? $\text{Specificity}=\frac{TN}{TN+FP}$.

• $F_1$-Score: The harmonic mean of Precision and Recall, providing a robust evaluation metric for highly imbalanced datasets: $$F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$$

For $TP=40$, $FP=10$, $TN=45$, and $FN=5$, there are $100$ cases. Accuracy is $(40+45)/100=85\%$, precision is $40/50=80\%$, recall is $40/45\approx88.9\%$, specificity is $45/55\approx81.8\%$, and $F_1=2TP/(2TP+FP+FN)=80/95\approx84.2\%$. One confusion matrix therefore reveals several different views of the same predictions.

A ROC curve varies the classification threshold and plots the true-positive rate against the false-positive rate. ROC-AUC summarizes how well a model ranks a randomly chosen positive above a randomly chosen negative across all thresholds. It is useful when comparing ranking performance before selecting an operating threshold, but it does not encode the application's actual error costs; for rare positive classes, also inspect precision-recall behavior.

Choose the final threshold and metric based on the cost of errors. In a screening task, missing a true positive can be dangerous, so recall may matter more than raw accuracy. Always calculate final metrics on a held-out test set that was not used to fit the model, tune hyperparameters, or select the threshold.