Stochastic Dynamics

In many machine learning applications (such as sequence generation, reinforcement learning, or generative diffusion models), physical states evolve over time with an element of randomness. We model these time-dependent noisy systems using Stochastic Processes.

• The Markov Property: A stochastic process has the Markov property (or is 'memoryless') if its future state depends strictly on its *present* state, and is independent of its entire past history. Formally:

$$P(X_{t+1} = x_{t+1} | X_t = x_t, X_{t-1} = x_{t-1}, ..., X_0 = x_0) = P(X_{t+1} = x_{t+1} | X_t = x_t)$$

• Markov Chains: Discrete-time Markov processes operating on discrete state spaces. We represent transition probabilities from any state $i$ to $j$ inside a square Transition Matrix $P$ where rows sum to exactly $1.0$.

Weather Chain Example: Suppose we have a 2-state weather system: Sunny ($S$) and Rainy ($R$). Let the transition matrix be:

$$P = \begin{bmatrix} 0.8 & 0.2 \\ 0.4 & 0.6 \end{bmatrix}$$

This means if today is Sunny, there is an $80\%$ chance tomorrow is Sunny, and a $20\%$ chance it rains. If today is Rainy, there is a $40\%$ chance tomorrow is Sunny. Over infinite days, the weather probabilities settle into a stable Stationary Distribution $\pi = [\pi_S, \pi_R]$ satisfying $\pi P = \pi$:

$$\begin{bmatrix} \pi_S & \pi_R \end{bmatrix} \begin{bmatrix} 0.8 & 0.2 \\ 0.4 & 0.6 \end{bmatrix} = \begin{bmatrix} \pi_S & \pi_R \end{bmatrix}$$ $$0.8\pi_S + 0.4\pi_R = \pi_S \implies 0.4\pi_R = 0.2\pi_S \implies \pi_S = 2\pi_R$$ Since probabilities must sum to 1 ($\pi_S + \pi_R = 1$), we solve: $2\pi_R + \pi_R = 1 \implies \pi_R = \frac{1}{3}, \pi_S = \frac{2}{3}$. Thus, over infinite time, the climate is guaranteed to be Sunny $66.7\%$ of the time, regardless of today's weather!

While standard calculus models smooth, deterministic curves, physical systems fluctuate randomly. We model continuous-time noisy trajectories using Stochastic Differential Equations (SDEs).

An SDE adds a stochastic noise term (governed by a Brownian motion / Wiener process $W_t$) to a standard deterministic differential equation: $$dX_t = f(X_t, t)dt + g(X_t, t)dW_t$$

where $f(X_t, t)$ represents the deterministic drift (forces pulling the system toward a trend), and $g(X_t, t)$ represents the stochastic diffusion (fluctuations or noise surrounding the trend).

• Langevin Dynamics: A specialized SDE modeling the physical thermal motion of particles suspended in fluid. In machine learning, Langevin dynamics is incredibly powerful because it allows us to draw samples from highly complex, unnormalized probability distributions $p(x)$ using only their gradients (the 'score' function $\nabla_x \ln p(x)$): $$dx_t = \frac{1}{2} \nabla_x \ln p(x_t) dt + dW_t$$ By taking derivative steps towards high-probability densities while injecting continuous Wiener noise $dW_t$, the path explores and samples the entire target distribution safely.

In Generative AI, we often need to generate samples from highly complex, high-dimensional probability distributions (like the distribution of all realistic human faces).

• Monte Carlo Sampling: Simulates random experiments repeatedly to approximate deterministic parameters (for instance, estimating the area of a circle by throwing random darts inside a bounding square).

• Markov Chain Monte Carlo (MCMC): A class of algorithms (like Metropolis-Hastings) that constructs a custom Markov chain whose stationary distribution is exactly our target distribution $p(x)$, allowing us to sample from complex spaces by walking along the chain.

• Denoising Diffusion Probabilistic Models (DDPM): The stochastic foundation behind modern generative image models. A forward diffusion process gradually degrades an input image into pure Gaussian noise by adding noise in tiny SDE steps. The model then learns a reverse process—using neural networks to estimate and subtract noise at each step, stochastically regenerating highly realistic images from pure noise.