Unit 13

Classical Machine Learning Foundations

learning the core algorithms and concepts behind predictive modeling

SVM Maximum Margin Decision Boundary

Core Concepts Covered

Supervised vs. unsupervised vs. reinforcement learning
Linear & logistic regression, decision trees, and SVMs
Clustering and model evaluation metrics

Local Setup Recommendation

To execute and experiment with the code cells below on your local machine, ensure you have set up your isolated virtual environments and scientific libraries by following the detailed protocols in Unit 03: Environment Setup or run them in Google Colab.

1. Taxonomy of Machine Learning

Machine Learning algorithms are broadly categorized based on the nature of the training data and feedback loops they receive:

• Supervised Learning: The model is trained on labeled historical examples containing both features and explicit targets. If the target is a continuous real number (like predicting house prices), it is a Regression problem. If the target is a discrete label (like classifying an email as *Spam* or *Inbox*), it is a Classification problem.

• Unsupervised Learning: The dataset contains no target labels. The model's objective is to autonomously discover hidden structures, correlations, or natural groupings inside the input features (such as customer segmentation clustering).

• Reinforcement Learning: An autonomous agent learns optimal decision-making policies by taking actions within an environment to maximize a scalar feedback reward over time (such as training an autopilot or a game-playing engine).

2. Core Linear Models: Regression and Classification

• Linear Regression: Models a linear relationship between features $x$ and a continuous target $y$ . We define the prediction as $y = w^T x + b$ and optimize parameters by minimizing the Sum of Squared Errors (SSE) loss function:

$\text{Loss} = \sum_{i=1}^n (y_i - (w^T x_i + b))^2$

• Logistic Regression: Used for binary classification. Instead of outputting unbounded continuous values, it passes the linear combination through the Sigmoid (logistic) Function $\sigma(z)$ to map any real number to a probability value strictly between 0 and 1:

$\sigma(z) = \frac{1}{1 + e^{-z}}$ ,

The model is optimized by minimizing the Binary Cross-Entropy loss function, which penalizes predictions that diverge from actual class labels.

Jupyter Code Notebook Cell

Python 3 (ipykernel)

In [1]:

# Practical implementation of Linear and Logistic regressions using Scikit-Learn
from sklearn.linear_model import LinearRegression, LogisticRegression
import numpy as np

# Linear Regression: Predict continuous outputs
X_reg = np.array([[1], [2], [3], [4]])
y_reg = np.array([2.1, 3.9, 6.1, 8.0])

reg_model = LinearRegression().fit(X_reg, y_reg)
print("Linear Regression Slope (w):", reg_model.coef_[0].round(2))
print("Linear Regression Intercept (b):", reg_model.intercept_.round(2))

# Logistic Regression: Classify binary categories
X_clf = np.array([[1], [2], [5], [6]])
y_clf = np.array([0, 0, 1, 1])

clf_model = LogisticRegression().fit(X_clf, y_clf)
print("Logistic Prediction for x=1.5 (Class):", clf_model.predict([[1.5]])[0])
print("Logistic Probabilities for x=3.5 (P(0), P(1)):", clf_model.predict_proba([[3.5]])[0].round(2))

Out [1]:

Linear Regression Slope (w): 1.98
Linear Regression Intercept (b): 0.08
Logistic Prediction for x=1.5 (Class): 0
Logistic Probabilities for x=3.5 (P(0), P(1)): [0.5 0.5]

3. Non-Linear Models: SVMs, Decision Trees, and K-Means

When datasets are complex and cannot be separated by simple straight lines, we use non-linear models:

• Support Vector Machines (SVMs): Finds an optimal decision hyperplane that maximizes the Margin (the perpendicular distance) between different classes. The closest boundary coordinates of each class are the Support Vectors.

• The Kernel Trick (Intuitive Analogy): Imagine drawing a circle of red dots on a sheet of paper, surrounded by a larger ring of blue dots. You cannot draw a straight line on that 2D paper to separate them. However, what if you 'pop' the center of the paper upward, pushing the red dots into 3D space? You can now pass a flat 2D sheet of glass (a linear hyperplane) horizontally between the floating red dots and the flat blue dots. The Kernel Trick is a mathematical shortcut that allows SVMs to project low-dimensional data into higher-dimensional spaces where they become linearly separable, without ever needing to compute the expensive high-dimensional coordinates explicitly!

• Decision Trees & Random Forests: Decision Trees recursively partition the feature space by making sequential splits that maximize Information Gain (reducing Gini Impurity or Shannon Entropy). To prevent a single tree from overfitting, a Random Forest trains hundreds of independent trees on random subsets of samples and features (bagging), averaging their votes.

• K-Means Clustering: An unsupervised algorithm that partitions unlabeled data into $K$ distinct clusters. It starts with $K$ random centroids, assigns points to the nearest centroid, updates centroids, and repeats until convergence. We evaluate clustering quality using the Silhouette Score (measuring how close a point is to its own cluster compared to others, on a scale from $-1$ to $+1$ ).

4. Classification Metrics and Model Evaluation

To evaluate our models, we construct a Confusion Matrix tracking True Positives ( $TP$ ), False Positives ( $FP$ ), True Negatives ( $TN$ ), and False Negatives ( $FN$ ). From these, we calculate crucial evaluation metrics:

• Accuracy: Overall proportion of correct predictions: $\frac{TP+TN}{TP+FP+TN+FN}$ .

• Precision: Out of all positive predictions, how many were actually positive? Highly critical when false positives are expensive (e.g., spam filters): $\text{Precision} = \frac{TP}{TP + FP}$

• Recall (Sensitivity): Out of all actual positive samples, how many did the model find? Highly critical when false negatives are dangerous (e.g., medical diagnoses): $\text{Recall} = \frac{TP}{TP + FN}$

• $F_1$ -Score: The harmonic mean of Precision and Recall, providing a robust evaluation metric for highly imbalanced datasets: $F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$

Hands-On Kaggle Challenge #1

Titanic: Machine Learning from Disaster

The most famous starter classification challenge on Kaggle! Predict survival outcomes on the Titanic using classification models (like Support Vector Machines, Logistic Regression, or Decision Trees) and submit your predictions to get ranked on the live leaderboard.

Refer back to Unit 03 for environment and Colab setup!

Join Kaggle Challenge

Hands-On Kaggle Challenge #2

House Prices: Advanced Regression Techniques

The classic starter regression challenge! Predict sales prices of residential houses using advanced regressions and tree-based ensembles (like Random Forests or Gradient Boosting) and submit your predictions.

Refer back to Unit 03 for environment and Colab setup!

Join Kaggle Challenge

Interactive Practice Quiz

Test your understanding with instant feedback

QUESTION 01

Which of the following describes the mathematical output range of the Sigmoid activation function $\sigma(z) = \frac{1}{1 + e^{-z}}$ ?

QUESTION 02

If a cancer diagnostic model must minimize the danger of missing actual positive cancer cases (False Negatives), which metric should we prioritize maximizing?

QUESTION 03

Under which task category does 'K-Means Clustering' fall, given that the input dataset has no target label columns?

QUESTION 04

What is the primary role of the Support Vector Machine (SVM) algorithm?

QUESTION 05

Compute the $F_1$ -Score for a classification model that achieves a Precision of $0.80$ and a Recall of $0.80$ :

QUESTION 06

What is the core mathematical purpose of the 'Kernel Trick' inside Support Vector Machines?

QUESTION 07

How do Decision Trees choose split points during training?

QUESTION 08

What is the core mechanism of the Random Forest ensemble model?

QUESTION 09

Which of the following describes the meaning of a Silhouette Score of $+1.0$ in K-Means clustering?

QUESTION 10

Classical Machine Learning Foundations

Core Concepts Covered

Local Setup Recommendation

1. Taxonomy of Machine Learning

2. Core Linear Models: Regression and Classification

3. Non-Linear Models: SVMs, Decision Trees, and K-Means

4. Classification Metrics and Model Evaluation

Hands-On Kaggle Challenge #1

Hands-On Kaggle Challenge #2

Interactive Practice Quiz

Which of the following describes the mathematical output range of the Sigmoid activation function $\sigma(z) = \frac{1}{1 + e^{-z}}$ ?

If a cancer diagnostic model must minimize the danger of missing actual positive cancer cases (False Negatives), which metric should we prioritize maximizing?

Under which task category does 'K-Means Clustering' fall, given that the input dataset has no target label columns?

What is the primary role of the Support Vector Machine (SVM) algorithm?

Compute the $F_1$ -Score for a classification model that achieves a Precision of $0.80$ and a Recall of $0.80$ :

What is the core mathematical purpose of the 'Kernel Trick' inside Support Vector Machines?

How do Decision Trees choose split points during training?

What is the core mechanism of the Random Forest ensemble model?

Which of the following describes the meaning of a Silhouette Score of $+1.0$ in K-Means clustering?

The loss function minimized by standard Linear Regression is called:

Further Readings

Classical Machine Learning Foundations

Core Concepts Covered

Local Setup Recommendation

1. Taxonomy of Machine Learning

2. Core Linear Models: Regression and Classification

3. Non-Linear Models: SVMs, Decision Trees, and K-Means

4. Classification Metrics and Model Evaluation

Hands-On Kaggle Challenge #1

Hands-On Kaggle Challenge #2

Interactive Practice Quiz

Which of the following describes the mathematical output range of the Sigmoid activation function σ(z)=11+e−z\sigma(z) = \frac{1}{1 + e^{-z}}σ(z)=1+e−z1​?

If a cancer diagnostic model must minimize the danger of missing actual positive cancer cases (False Negatives), which metric should we prioritize maximizing?

Under which task category does 'K-Means Clustering' fall, given that the input dataset has no target label columns?

What is the primary role of the Support Vector Machine (SVM) algorithm?

Compute the F1F_1F1​-Score for a classification model that achieves a Precision of 0.800.800.80 and a Recall of 0.800.800.80:

What is the core mathematical purpose of the 'Kernel Trick' inside Support Vector Machines?

How do Decision Trees choose split points during training?

What is the core mechanism of the Random Forest ensemble model?

Which of the following describes the meaning of a Silhouette Score of +1.0+1.0+1.0 in K-Means clustering?

The loss function minimized by standard Linear Regression is called:

Further Readings

Which of the following describes the mathematical output range of the Sigmoid activation function $\sigma(z) = \frac{1}{1 + e^{-z}}$ ?

Compute the $F_1$ -Score for a classification model that achieves a Precision of $0.80$ and a Recall of $0.80$ :

Which of the following describes the meaning of a Silhouette Score of $+1.0$ in K-Means clustering?