Unsupervised Learning and Pattern Discovery

In supervised learning, each training example has a target such as a house price or a spam label. Unsupervised learning instead starts with feature vectors $x_1, x_2, ..., x_n$ and no target column. Its purpose is exploratory: reveal groups, compact representations, dense regions, or unusual observations that may be useful to people or later models.

A discovered pattern is not automatically a real-world category. The result depends on how examples are represented, which features are included, and how distance or similarity is measured. For example, customer groups based on annual spending may change dramatically if age and income are measured on much larger numeric scales and are not standardized.

Common goals include clustering similar observations, reducing dimensions for visualization, estimating latent distributions, and identifying possible outliers. Unit 8 introduced the mathematics of PCA; here, PCA becomes one practical tool for inspecting an unlabeled dataset before or alongside clustering.

K-Means divides observations into a chosen number of clusters $K$. Each cluster is represented by a centroid $\mu_k$. The algorithm repeatedly assigns every point to its nearest centroid, then replaces each centroid with the mean of its assigned points until the assignments no longer change substantially.

Its objective is to make points within each cluster close to their centroid by minimizing the within-cluster sum of squares, often called inertia: $$J = \sum_{i=1}^{n} \min_{k \in \{1, ..., K\}} \lVert x_i - \mu_k \rVert^2$$

Because K-Means uses distances, scale numeric features first when their units differ. It also requires a value of $K$, is sensitive to initialization, and works best when groups are reasonably compact and similar in spread. Run multiple initializations, inspect the result, and use domain knowledge rather than treating a cluster ID as a ground-truth label.

To compare candidate values of $K$, plot inertia against $K$ and look for an elbow where additional clusters bring diminishing improvement. A silhouette score can provide another view of separation, but neither method discovers a uniquely correct $K$. Prefer a solution that is stable under small data changes and useful in the problem domain.

Hierarchical clustering builds nested groups. Agglomerative methods begin with each point separate and merge groups. Single linkage uses the nearest pair and can recover long shapes but may chain through noise; complete linkage uses the farthest pair and favors compact groups; average linkage compromises between them; Ward linkage merges groups that cause the smallest rise in within-cluster variance. A dendrogram records merge distance, and a horizontal cut chooses a clustering.

DBSCAN uses a radius $\varepsilon$ (eps) and neighbor threshold min_samples. A core point has enough points in its neighborhood, a border point is reachable from a core but lacks enough neighbors itself, and a noise point is not density-reachable. It can recover irregular shapes and needs no preset $K$.

Scale features before interpreting $\varepsilon$. DBSCAN struggles when clusters have sharply different densities: one radius can fragment a sparse cluster or merge dense nearby clusters. HDBSCAN varies the density scale and can be more robust, but it still requires domain validation.

A Gaussian Mixture Model (GMM) assumes the data was generated by a weighted combination of $K$ Gaussian distributions. Each component has a mixing weight $\pi_k$, a mean vector $\mu_k$, and a covariance matrix $\Sigma_k$. The model assigns a probability density to any observation $x$: $$p(x) = \sum_{k=1}^{K} \pi_k \mathcal{N}(x \mid \mu_k, \Sigma_k)$$ The weights are non-negative and sum to one: $\pi_k \geq 0$ and $\sum_{k=1}^{K} \pi_k = 1$.

Unlike K-Means, which gives every point one hard cluster ID, a GMM calculates a responsibility for each component: the posterior probability that a component generated the point. For an observation near an overlap between two groups, the model might assign probabilities of $0.55$ and $0.45$ instead of pretending the boundary is certain. This also lets a GMM estimate density and identify points that are unlikely under every component.

The covariance setting determines the shapes that components can learn. full gives each component its own complete covariance matrix and can represent tilted ellipses. tied shares one full covariance matrix across components. diag allows different variance per feature but no feature correlation, while spherical uses one variance in every direction. More flexible covariance types need more data and can overfit, so start with the simplest option that matches the geometry of the problem.

GMMs are usually fit with Expectation-Maximization (EM). In the E-step, calculate responsibilities using the current parameters. In the M-step, update the weights, means, and covariances using those responsibilities as soft weights. Repeating these steps increases the data likelihood until the parameters stabilize. Unit 16 develops the full EM derivation and its latent-variable interpretation.

Without known labels, classification accuracy is unavailable. The silhouette score compares how close a point is to its own cluster with how close it is to the nearest other cluster. Scores near $+1$ suggest well-separated compact clusters, values near $0$ suggest overlap, and negative values can indicate that points may be assigned to the wrong cluster.

Use internal scores as diagnostics, not as proof. Compare several reasonable hyperparameter choices, check whether clusters remain stable under small data changes, visualize the data when possible, and ask whether the groups make sense in the domain. A visually appealing cluster can still reflect a data-collection artifact or an irrelevant feature.

Pattern discovery can also surface rare cases. A point that DBSCAN marks as noise, or that lies far from every centroid, may be an error, a fraud case, a novel subgroup, or simply a valid but uncommon observation. Investigate it before deciding what it means.

Clustering is only one possibility. Anomaly discovery ranks unusual transactions or sensor readings; segmentation groups customers, images, or documents for inspection. Spectral clustering builds a similarity graph and can separate non-convex structures, but its affinity choice and computational cost matter.

PCA supplies a linear, variance-preserving projection whose derivation remains in Unit 8. t-SNE is valuable for local exploratory visualization but distorts global distances and should not be treated as a clustering algorithm. UMAP constructs a neighborhood graph and often preserves more global organization, though its embedding also changes with hyperparameters and random seeds. HDBSCAN can cluster variable-density structures in an embedding, but validate results in the original feature space.