Probability for Machine Learning

In machine learning, we deal with noisy, incomplete, or chaotic environments. We use probability to represent and reason about this uncertainty. Instead of asserting absolute binary facts, probability allows models to express predictions as confidence percentages (for example: 'this image is $98\%$ likely to be a cat').

• Conditional Probability ($P(A | B)$): The probability of event $A$ occurring, given that event $B$ has already occurred: $$P(A | B) = \frac{P(A \cap B)}{P(B)}$$

• Statistical Independence: Events $A$ and $B$ are independent if the occurrence of $B$ does not affect the probability of $A$, meaning $P(A | B) = P(A)$ or equivalently $P(A \cap B) = P(A)P(B)$.

• Bayes' Theorem (Bayesian Inference): The mathematical bedrock of probabilistic machine learning. It allows us to update our prior belief about an event ($P(A)$) after observing new evidence ($B$). It maps the relationship between prior, likelihood, evidence, and posterior: $$P(A | B) = \frac{P(B | A) P(A)}{P(B)}$$

Spam Filter Trace: Let's calculate whether an incoming email containing the word free ($F$) is actually spam ($S$): - Prior belief: $30\%$ of all emails are spam: $P(S) = 0.30$. - Likelihood: $80\%$ of spam emails contain the word free: $P(F | S) = 0.80$. - General Evidence: $40\%$ of all emails contain the word free: $P(F) = 0.40$. - Applying Bayes' Theorem: $$P(S | F) = \frac{P(F | S) P(S)}{P(F)} = \frac{0.80 \times 0.30}{0.40} = \frac{0.24}{0.40} = 0.60$$ This calculation proves that observing the word free updates our belief from $30\%$ prior to a $60\%$ posterior probability that the email is spam!

A random variable is a variable whose possible values are numerical outcomes of a random process. They are categorized into discrete (taking countable values like a dice roll) or continuous (taking any real value in an interval like temperature).

• PMF vs. PDF: Discrete variables are defined by a Probability Mass Function (PMF), while continuous variables are defined by a continuous Probability Density Function (PDF) integrated to find probabilities over ranges.

• Expectation ($E[X]$): The theoretical mean or average value of a random variable over infinite trials. For discrete variables, it is: $$E[X] = \sum_{x} x P(X = x)$$,

• Variance ($\text{Var}(X)$): Measures how spread out the values are from their mean value: $$\text{Var}(X) = E[(X - E[X])^2] = E[X^2] - (E[X])^2$$,

Discrete Dice Trace: Let's calculate the expectation and variance of rolling a fair 4-sided die ($X \in \{1, 2, 3, 4\}$, each with probability $0.25$): - Expectation $E[X]$: $$E[X] = (1)(0.25) + (2)(0.25) + (3)(0.25) + (4)(0.25) = 2.5$$ - Expected Squared Value $E[X^2]$: $$E[X^2] = (1^2)(0.25) + (2^2)(0.25) + (3^2)(0.25) + (4^2)(0.25) = 0.25 + 1.0 + 2.25 + 4.0 = 7.5$$ - Variance $\text{Var}(X)$: $$\text{Var}(X) = E[X^2] - (E[X])^2 = 7.5 - 2.5^2 = 7.5 - 6.25 = 1.25$$ This gives us a precise mathematical measure of the spread of the die's outcomes!

• Covariance ($\text{Cov}(X,Y)$): Measures joint linear movement. Correlation standardizes it: $\rho_{XY}=\operatorname{Cov}(X,Y)/(\sigma_X\sigma_Y)$. Independence implies zero covariance when moments exist, but zero covariance does not generally imply independence.

A probability distribution is a mathematical profile mapping the likelihood of all possible values for a random variable. The most important distribution in machine learning is the Gaussian (Normal) Distribution due to the Central Limit Theorem, which states that sums of independent random variables tend to converge to a normal distribution.

The standard single-variable normal PDF with mean $\mu$ and variance $\sigma^2$ is: $$f(x) = \frac{1}{\sigma \sqrt{2\pi}} e^{-\frac{(x - \mu)^2}{2\sigma^2}}$$

In high-dimensional spaces, the multivariate Gaussian for $x\in\mathbb{R}^d$ has mean $\mu$ and covariance $\Sigma$: $$f(x)=\frac{1}{(2\pi)^{d/2}|\Sigma|^{1/2}}\exp\left[-\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)\right].$$

where $|\Sigma|$ is the determinant of the covariance matrix. Geometrically, the mean vector $\mu$ translates the center of the distribution in space, while the covariance matrix $\Sigma$ scales and rotates the probability density ellipses. The eigenvalues of $\Sigma$ represent the variance scales along those rotated principal axes!