Data Fundamentals for Machine Learning

Data is the absolute fuel of any machine learning system. Before writing training loops, we must understand how data is formatted and represented. At their core, machine learning models are mathematical systems that can only calculate on numbers, vectors, and matrices. Therefore, all real-world data must be mapped into structured numerical spaces.

• Structured Data: Organized in highly formatted tables, schemas, or relational databases (e.g., CSV files containing age, income, and house price columns). Each sample is a row, and each feature is a column, which can be loaded directly into a standard tabular frame.

• Unstructured Data: Has no pre-defined organizational layout (e.g., raw text articles, JPEG image pixel grids, audio wave recordings, or MP4 video frames). To feed unstructured data into an ML model, we use deep representation layers to automatically extract features. For instance, a Convolutional Neural Network (CNN) transforms a 2D JPEG pixel grid into a flat vector of feature scores, representing the visual contents mathematically in a structured vector space.

In raw tabular datasets, features often span completely different mathematical scales. For instance, in a real estate dataset, the number of bedrooms spans $1-5$, while annual household income spans $$$$20,000-$200,000$.

Why Scaling Matters: If we attempt to run Gradient Descent on unscaled features, the loss function's contour landscape becomes extremely skewed and elongated along the larger-scale feature axes. This causes gradient updates to oscillate erratically, slowing down convergence or preventing learning entirely. Scaling the features squashes the contour landscape into a symmetric bowl, enabling smooth, direct steps toward the local minimum.

We solve this using two primary scaling techniques:

• Normalization (Min-Max Scaling): Linearly shifts and scales all values so they lie strictly within a bounded interval, typically $[0, 1]$. It is ideal when you know your data boundaries have no extreme outliers. The formula is:

$$x_{\text{scaled}} = \frac{x - x_{\text{min}}}{x_{\text{max}} - x_{\text{min}}}$$

• Standardization (Standard Scaling): Centers the feature values to have a mean ($\mu$) of $0$ and a standard deviation ($\sigma$) of $1$. Unlike normalization, it does not restrict values to a hard range, making it highly robust to extreme outliers. The formula is:

$$x_{\text{standardized}} = \frac{x - \mu}{\sigma}$$

To perform strict scientific evaluation and ensure our models can generalize to new real-world environments, we must split our dataset into three mutually exclusive partitions:

1. Training Set ($70\%$-$80\%$): The data passed directly to our optimization algorithms to fit the model parameters (weights and biases).

2. Validation Set ($10\%$-$15\%$): Used during development to monitor model generalization, tune hyperparameters (like learning rate or model architecture size), and trigger early stopping if the model begins to overfit.

3. Testing Set ($10\%$-$15\%$): Kept completely isolated in a vault until the end of development. It serves as an uncorrupted final benchmark to measure the model's true real-world generalization performance on novel inputs.

• The Threat of Data Leakage: Data leakage occurs when information from the validation or testing set is unintentionally exposed to the model during training. A common source of leakage is scaling: if you compute the global mean ($\mu$) and variance ($\sigma$) using the *entire* dataset before splitting, the model 'leaks' properties of the test set into its training phase, yielding artificially optimistic testing scores that collapse when deployed to actual production environments.