Probabilistic Models for Machine Learning

A probabilistic model assigns a likelihood $p(D\mid\theta)$ to observed data $D$ under parameters $\theta$. For independent observations, the likelihood is a product, and the log-likelihood turns that product into a sum: $$\ell(\theta)=\log p(D\mid\theta)=\sum_{i=1}^{n}\log p(x_i\mid\theta).$$

Maximum Likelihood Estimation (MLE) chooses the parameter that best explains the observed data: $$\hat\theta_{\mathrm{MLE}}=\arg\max_{\theta}p(D\mid\theta).$$ For $h$ heads and $t$ tails from a Bernoulli coin, $p(D\mid\theta)=\theta^h(1-\theta)^t$ and $\hat\theta_{\mathrm{MLE}}=h/(h+t)$.

Maximum A Posteriori (MAP) combines the likelihood with a prior $p(\theta)$. Bayes' rule gives $$p(\theta\mid D)=\frac{p(D\mid\theta)p(\theta)}{p(D)},$$ and the evidence $p(D)$ can be ignored when optimizing with respect to $\theta$: $$\hat\theta_{\mathrm{MAP}}=\arg\max_{\theta}\left[\log p(D\mid\theta)+\log p(\theta)\right].$$

A Beta prior is conjugate to a Bernoulli likelihood. If $\theta\sim\operatorname{Beta}(\alpha,\beta)$, then observing $h$ heads and $t$ tails gives $$\theta\mid D\sim\operatorname{Beta}(\alpha+h,\beta+t).$$ The posterior mean is $\alpha'/(\alpha'+\beta')$, while the interior posterior mode is $(\alpha'-1)/(\alpha'+\beta'-2)$.

A Gaussian prior centered at zero adds a quadratic penalty to the negative log-posterior. This is the probabilistic reason MAP estimation often looks like $L_2$ regularization: the data term pulls toward a good fit, while the prior shrinks implausibly large parameters.

A Bayesian Network is a directed acyclic graph whose nodes are random variables and whose arrows encode direct conditional dependence. The graph factorizes a large joint distribution into smaller local pieces: $$p(x_1,\ldots,x_n)=\prod_{i=1}^{n}p\!\left(x_i\mid\operatorname{Parents}(x_i)\right).$$

The missing arrows carry information too. Once a node's parents are known, the node is conditionally independent of its non-descendants. These independencies reduce how many probabilities the model must store and which terms an inference calculation must inspect.

Inference reverses or fills in parts of the graph. For a hypothesis $H$ and evidence $E$, normalize the joint probability over all competing hypotheses: $$p(H\mid E)=\frac{p(E\mid H)p(H)}{\sum_h p(E\mid h)p(h)}.$$

A latent variable $z$ is not observed directly but helps explain an observed variable $x$. To obtain the probability of $x$, sum over a discrete latent state or integrate over a continuous one: $$p(x)=\sum_z p(x,z)\qquad\text{or}\qquad p(x)=\int p(x,z)\,dz.$$

Exact Bayesian inference asks for $p(z\mid x)=p(x,z)/p(x)$. The denominator can require an enormous sum or integral, so Variational Inference (VI) chooses a tractable approximation $q_\phi(z\mid x)$ and fits it to the true posterior.

Unit 12 introduced KL divergence as a measure of how one distribution differs from another. Here it becomes an optimization target: $$D_{\mathrm{KL}}\!\left(q_\phi(z\mid x)\parallel p(z\mid x)\right)=\mathbb{E}_{q_\phi}\!\left[\log q_\phi(z\mid x)-\log p(z\mid x)\right].$$

Because the true posterior contains the difficult evidence $p(x)$, VI maximizes the Evidence Lower Bound (ELBO) instead: $$\mathcal{L}(q)=\mathbb{E}_{q(z)}[\log p(x,z)]-\mathbb{E}_{q(z)}[\log q(z)].$$ The identity $$\log p(x)=\mathcal{L}(q)+D_{\mathrm{KL}}(q(z)\parallel p(z\mid x))$$ shows that maximizing the ELBO closes the non-negative KL gap.

Unit 21 applies this machinery to Variational Autoencoders, where an encoder supplies $q_\phi(z\mid x)$ and a decoder supplies $p_\theta(x\mid z)$. This unit focuses on the inference calculation underneath that architecture.

Unit 14 introduced Gaussian Mixture Models (GMMs) as an unsupervised-learning method. K-Means gives each point one hard cluster label. A GMM instead models a probability density and assigns soft membership across $K$ Gaussian components: $$p(x_i)=\sum_{k=1}^{K}\pi_k\mathcal{N}(x_i\mid\mu_k,\Sigma_k),\qquad \sum_{k=1}^{K}\pi_k=1.$$

The unobserved component label $z_i$ is a latent variable. Given current parameters, the E-step computes its posterior probability, called a responsibility: $$\gamma_{ik}=p(z_i=k\mid x_i)=\frac{\pi_k\mathcal{N}(x_i\mid\mu_k,\Sigma_k)}{\sum_{j=1}^{K}\pi_j\mathcal{N}(x_i\mid\mu_j,\Sigma_j)}.$$

The M-step treats responsibilities as fractional counts. With $N_k=\sum_i\gamma_{ik}$, the updates are $$\pi_k^{\mathrm{new}}=\frac{N_k}{n},\qquad \mu_k^{\mathrm{new}}=\frac{1}{N_k}\sum_i\gamma_{ik}x_i,$$ $$\Sigma_k^{\mathrm{new}}=\frac{1}{N_k}\sum_i\gamma_{ik}(x_i-\mu_k^{\mathrm{new}})(x_i-\mu_k^{\mathrm{new}})^T.$$

The covariance structure controls each component's geometry. full learns a complete covariance matrix per component, tied shares one complete matrix, diag learns one variance per feature without correlations, and spherical learns one variance per component. Flexible choices need more data and can produce nearly singular covariance matrices, so use regularization and inspect component counts.

Each EM iteration does not decrease the observed-data log-likelihood $$\ell(\theta)=\sum_{i=1}^{n}\log\left[\sum_{k=1}^{K}\pi_k\mathcal{N}(x_i\mid\mu_k,\Sigma_k)\right],$$ but EM can settle at a local optimum. Multiple initializations, covariance regularization, and checks for nearly empty components reduce common failures.

More components almost always improve training likelihood, so model selection must penalize complexity. The Bayesian Information Criterion is $$\operatorname{BIC}=p\log n-2\ell(\hat\theta),$$ where $p$ is the number of fitted parameters. Lower BIC is preferred among models fitted to the same dataset.