Statistics for Machine Learning

Statistics provides the mathematical framework to summarize datasets, detect hidden patterns, and draw conclusions under uncertainty. It is broadly divided into two domains:

• Descriptive Statistics: Summarizes the numerical properties of an observed sample (such as the sample mean $\bar{x} = \frac{1}{n} \sum x_i$ and sample variance $s^2$).

• Why Bessel's Correction ($n-1$) is Crucial: When computing sample variance, we divide by $n-1$ rather than $n$:

$$s^2 = \frac{1}{n-1} \sum_{i=1}^n (x_i - \bar{x})^2$$

Dividing by $n$ underestimates the true population variance because the sample data points are naturally closer to their own sample mean $\bar{x}$ than they are to the true, unobserved population mean $\mu$. Dividing by $n-1$ (Bessel's Correction) mathematically corrects this bias, providing an unbiased estimator of the population variance.

• Inferential Statistics: Uses sample data to make generalized predictions, test claims, or draw conclusions about a larger, unobserved population.

In machine learning, we use hypothesis testing to determine if our model improvements are statistically genuine or merely the result of random noise.

The A/B Test Example: Suppose your baseline model (A) has an accuracy of $80\%$. You develop a new model (B) and test it on a sample of $100$ users, achieving an accuracy of $83\%$. Is this $3\%$ improvement genuine, or did you just get lucky with a favorable sample of users? We resolve this using hypothesis testing:

• Null Hypothesis ($H_0$): The default assumption that there is no real difference or effect (e.g., 'Model B has the same performance as Model A; the $3\%$ difference is random noise').

• Alternative Hypothesis ($H_a$): The claim we want to prove (e.g., 'Model B is genuinely superior to Model A').

• Significance Level ($\alpha$): The threshold of risk we are willing to accept (typically set to $0.05$ or $5\%$). It is the probability of rejecting $H_0$ when it is actually true (Type I error).

• The $p$-value: The probability of observing an accuracy difference of $3\%$ or greater if the null hypothesis $H_0$ is true. If the calculated $p$-value is below our significance level $\alpha$ (e.g., $p$-value $= 0.01 < 0.05$), we reject the null hypothesis in favor of the alternative, concluding that Model B is statistically superior!

• Confidence Intervals (CI): A range of values likely to contain the true population parameter with a specified confidence level (e.g., $95\%$ confidence interval: $\bar{x} \pm z^* \frac{s}{\sqrt{n}}$).

Machine learning is essentially the search for optimal parameters. We find these parameters using parameter estimation frameworks, most notably Maximum Likelihood Estimation (MLE).

The goal of MLE is to find the parameter values $\theta$ that maximize the likelihood of observing our actual dataset $D$. Assuming samples are independent and identically distributed (i.i.d.), the joint likelihood is the product of individual sample probabilities: $$L(\theta) = \prod_{i=1}^n P(x_i | \theta)$$

Because multiplying probabilities causes numerical underflow, we maximize the log-likelihood function instead: $$\ln L(\theta) = \sum_{i=1}^n \ln P(x_i | \theta)$$

The Coin-Flip Derivation: If we flip a coin $n$ times and observe $k$ heads (successes), we model this using a Bernoulli distribution with success parameter $\theta$. The likelihood function is: $$L(\theta) = \theta^k (1 - \theta)^{n-k}$$ Taking the natural log converts the exponent multiplication into addition: $$\ln L(\theta) = k \ln(\theta) + (n-k) \ln(1 - \theta)$$ To find the parameter value that maximizes this log-likelihood, we take the derivative with respect to $\theta$, set it to 0, and solve: $$\frac{d}{d\theta} \ln L(\theta) = \frac{k}{\theta} - \frac{n-k}{1-\theta} = 0$$ $$\frac{k}{\theta} = \frac{n-k}{1-\theta} \implies k(1-\theta) = \theta(n-k)$$ $$k - k\theta = n\theta - k\theta \implies \theta_{\text{MLE}} = \frac{k}{n}$$ This proves mathematically that our intuitive estimate—the proportion of observed heads—is the mathematically optimal parameter maximizing the likelihood of our observations!