Probabilistic Models for Machine Learning

In classical statistics, we use Maximum Likelihood Estimation (MLE) to find parameters $\theta$ that maximize the likelihood of the observed data: $P(D | \theta)$. However, MLE is highly prone to severe overfitting when data is scarce.

The Coin-Flip Example: Suppose you find a coin on the street, flip it twice, and get heads twice. An MLE model calculates the probability of heads as exactly $\theta_{\text{MLE}} = 1.0$ ($100\%$). This is highly unrealistic—it claims the coin is mathematically impossible to land on tails!

We solve this using Maximum A Posteriori (MAP) estimation. MAP is a Bayesian framework that incorporates a Prior Distribution $P(\theta)$—representing our prior beliefs about likely parameter values before observing data (e.g., 'most coins are generally fair and land on heads $\approx 50\%$ of the time'). Using Bayes' Theorem, the posterior probability of $\theta$ given data $D$ is:

$$P(\theta | D) = \frac{P(D | \theta) P(\theta)}{P(D)}$$

To find the MAP estimate, we maximize the posterior. Since the evidence denominator $P(D)$ is constant with respect to $\theta$, this is equivalent to:

$$\theta_{\text{MAP}} = \arg\max_{\theta} P(D | \theta) P(\theta)$$

Taking the natural log gives:

$$\theta_{\text{MAP}} = \arg\max_{\theta} [ \ln P(D | \theta) + \ln P(\theta) ]$$

This formula demonstrates beautifully that MAP acts as a regularized version of MLE. The data likelihood term $\ln P(D | \theta)$ pushes the parameters to fit the observed data, while the prior term $\ln P(\theta)$ acts as a mathematical penalty, preventing the parameters from taking extreme or unrealistic values based on our prior knowledge. In our coin-flip example, the prior pulls the final MAP estimate back down from a ridiculous $100\%$ to a highly realistic $53\%$!

Representing high-dimensional joint probability distributions is computationally impossible because the number of parameters grows exponentially with the number of variables. To solve this, we use Bayesian Networks (Probabilistic Graphical Models).

A Bayesian Network is a directed acyclic graph (DAG) where nodes represent random variables, and edges represent conditional dependencies. By asserting conditional independencies (that a variable is independent of non-descendants given its parents), we can factorize the joint probability compactly into a product of local conditional distributions: $$P(x_1, x_2, ..., x_n) = \prod_{i=1}^n P(x_i | \text{Parents}(x_i))$$

In many real-world datasets, we have variables that are never directly observed in our data. These are called Latent (hidden) Variables (for instance, a user's hidden interest category, or a patient's hidden disease state). Modeling these requires specialized latent variable frameworks.

• Variational Inference (VI): In complex Bayesian models, calculating the exact posterior probability $P(z|x)$ of latent variables given our observations is mathematically intractable because the denominator (evidence integration) requires integrating over all possible hidden states. Variational Inference solves this by framing inference as an Optimization Problem. We choose a family of simpler distributions $q(z)$ (like independent Gaussians) and find the specific parameters that minimize the KL divergence from our approximation to the true posterior. Minimizing this divergence is mathematically achieved by maximizing the Evidence Lower Bound (ELBO).

A Gaussian Mixture Model (GMM) is a powerful probabilistic clustering model that assumes all data points are generated from a mixture of $K$ different Gaussian distributions with unknown parameters.

Because we do not observe which Gaussian generated which data point, the assignment is a latent variable. We cannot solve GMM parameters using standard analytical derivatives. Instead, we use the Expectation-Maximization (EM) Algorithm:

1. Initialization: Start with random guesses for the means, variances, and mixing weights of the $K$ Gaussians.

2. Expectation (E-step): Calculate the probability (responsibility) that each data point belongs to each of the $K$ Gaussians, based on our current parameter guesses: $$\gamma_{ik} = P(\text{Gaussian } k | x_i, \theta)$$

3. Maximization (M-step): Update the Gaussian parameters (means, variances, and weights) to maximize the expected log-likelihood of the data, using the responsibilities calculated in the E-step as weights.

4. Convergence: Repeat the E and M steps until the parameters stabilize.