Modern AI Systems and Generative AI

Early neural machine translation commonly used an encoder RNN or LSTM to compress a source sentence into a fixed-length context vector, then a decoder RNN to generate the translation. The 2014 sequence-to-sequence result showed that this approach could work well, but asking one vector to preserve every detail created a bottleneck, especially for long sentences. Recurrence also makes the training path serial: hidden state $h_t$ depends on $h_{t-1}$.

Bahdanau additive attention changed the interface between encoder and decoder. At every decoding step, the model scored all encoder states, normalized those scores, and formed a new weighted context. Luong attention explored effective dot-product and multiplicative scoring variants. Attention therefore began as a way for an RNN decoder to retrieve the source information it needed instead of relying on one fixed summary.

The 2017 Transformer removed recurrence from the main sequence path and built the model from attention and position-wise feed-forward layers. In 2018, GPT demonstrated generative pretraining followed by task adaptation, while BERT demonstrated deep bidirectional encoder pretraining. Later systems scaled these architectural families with more data, parameters, compute, and increasingly specialized training stages.

Transformers became dominant in large-scale language systems because tokens can be processed in parallel during training, attention gives short paths between distant positions, retrieval is content-dependent, and the architecture scales efficiently on accelerator hardware. The comparison needs nuance: autoregressive generation is still sequential, standard attention costs $O(n^2)$ in sequence length, position information must be supplied, and RNNs can remain attractive for low-memory streaming or tightly constrained devices.

Suppose $X\in\mathbb{R}^{n\times d_{model}}$ holds $n$ token embeddings. Learned projections produce $Q=XW_Q$, $K=XW_K$, and $V=XW_V$. With key width $d_k$ and value width $d_v$, their shapes are $Q,K\in\mathbb{R}^{n\times d_k}$ and $V\in\mathbb{R}^{n\times d_v}$. Row $i$ of $Q$ asks a query; every row of $K$ describes what that token offers; the matching weights retrieve rows of $V$.

The scores, row-wise attention weights, and contextual outputs are $$S=\frac{QK^T}{\sqrt{d_k}},\qquad A=\operatorname{softmax}_{row}(S),\qquad O=AV.$$ The scale $\sqrt{d_k}$ prevents high-dimensional dot products from routinely pushing softmax into nearly flat-gradient regions. Each row of $A$ sums to $1$, so output row $i$ is a weighted mixture of the value vectors.

For a complete three-token example, let $$Q=K=\begin{bmatrix}1&0\\0&1\\1&1\end{bmatrix},\qquad V=\begin{bmatrix}1&0\\0&2\\3&1\end{bmatrix}.$$ Then $$S\approx\begin{bmatrix}.707&0&.707\\0&.707&.707\\.707&.707&1.414\end{bmatrix},\quad A\approx\begin{bmatrix}.401&.198&.401\\.198&.401&.401\\.248&.248&.503\end{bmatrix},$$ and $$AV\approx\begin{bmatrix}1.604&.797\\1.401&1.203\\1.759&1.000\end{bmatrix}.$$ For example, the first output is $.401[1,0]+.198[0,2]+.401[3,1]=[1.604,.797]$.

A decoder must not look at future tokens. A causal mask replaces entries above the score-matrix diagonal with $-\infty$ before softmax. Their exponentials become zero, while the permitted entries in each row are renormalized. A padding mask similarly prevents attention to batch padding; it is not the same as a causal mask.

One attention head captures one learned similarity space. Multi-head attention runs $h$ projected heads, concatenates their outputs, and applies $W_O$: $\operatorname{MHA}(X)=\operatorname{Concat}(head_1,\ldots,head_h)W_O$. Different heads can specialize in different patterns, although a head is not guaranteed to have a simple human-readable role.

A block combines attention with a position-wise feed-forward network such as $\operatorname{FFN}(x)=W_2\,\sigma(W_1x+b_1)+b_2$. Residual paths preserve an identity route around each sublayer, and layer normalization stabilizes activations. Implementations may place normalization before or after a sublayer. Token embeddings also receive learned or fixed positional information because attention alone is permutation-equivariant.

Encoder-only models use bidirectional self-attention and are natural for representation tasks. Decoder-only models use causal self-attention for next-token generation. Encoder-decoder models let a causal decoder cross-attend to a separately encoded input, which suits conditional generation such as translation. Padding masks hide absent tokens; causal masks hide future tokens.

Training a decoder can score all known next-token targets in parallel, but inference must produce the next token before the following one exists. A KV cache reuses earlier keys and values instead of recomputing the full prefix at each generation step. It reduces repeated computation but consumes memory. Standard dense attention still stores or computes an $n\times n$ interaction pattern, giving the familiar $O(n^2)$ sequence-length tradeoff.

Generative AI is not one architecture. An autoregressive Transformer factorizes a sequence into next-step conditionals, has no required global latent code, and samples one token at a time. It offers stable likelihood-based training and excels at discrete sequences, but long generation is sequential.

A Variational Autoencoder (VAE) learns a smooth probabilistic latent space with a reconstruction-plus-regularization objective. Sampling is usually one decoder pass and latent interpolation is meaningful, though image outputs can look blurred. A Generative Adversarial Network (GAN) learns through a generator-discriminator game. It can produce sharp samples in one pass but training balance and mode coverage are difficult.

A diffusion model learns to reverse progressive corruption, usually beginning from noise. Its denoising objective is stable and its coverage and visual quality are strong, but conventional sampling requires many iterative network evaluations. Unit 16 provides the variational-inference foundation for VAEs, and Unit 19 develops diffusion and score-based generation in depth. Model choice depends on the data, controllability, latency, likelihood needs, and failure costs, not on a universal winner.

An ordinary autoencoder maps an input to one code and reconstructs it. A Variational Autoencoder (VAE) instead uses an encoder $q_\phi(z\mid x)$ to predict a distribution over latent codes, usually diagonal Gaussian parameters $\mu(x)$ and $\log\sigma^2(x)$. A decoder $p_\theta(x\mid z)$ maps a sampled code back to a distribution over observations.

Sampling directly would obstruct backpropagation through the encoder. The reparameterization trick writes $$z=\mu+\sigma\odot\epsilon,\qquad \epsilon\sim\mathcal{N}(0,I),\qquad \sigma=\exp\left(\tfrac12\log\sigma^2\right).$$ Randomness now enters through $\epsilon$, while $\mu$ and $\sigma$ remain differentiable transformations.

Training minimizes a reconstruction term plus divergence from a simple prior: $$\mathcal{L}_{VAE}=\mathcal{L}_{recon}+D_{KL}\!\left(q_\phi(z\mid x)\,\|\,p(z)\right),\qquad p(z)=\mathcal{N}(0,I).$$ For a diagonal Gaussian, $D_{KL}=-\tfrac12\sum_j(1+\log\sigma_j^2-\mu_j^2-\sigma_j^2)$. The reconstruction term preserves information about $x$; KL regularization makes nearby latent samples decode coherently.

A structured latent space supports interpolation and conditional generation. Important failure modes include posterior collapse, where the decoder ignores $z$ and the approximate posterior matches the prior, and blurred outputs when a simple pixel-wise likelihood averages multiple plausible reconstructions. Capacity, the KL weight, decoder strength, and likelihood choice all influence this balance.

A Generative Adversarial Network (GAN) contains a generator $G$ that maps random noise $z\sim p(z)$ to synthetic samples and a discriminator $D$ that estimates whether an input came from the data distribution. The original minimax game is $$\min_G\max_D \mathbb{E}_{x\sim p_{data}}\log D(x)+\mathbb{E}_{z\sim p(z)}\log\!\left(1-D(G(z))\right).$$

Training alternates the players. During a discriminator update, real examples should approach label $1$, generated examples should approach label $0$, and fake samples are detached so that step does not update $G$. During a generator update, gradients flow through $D(G(z))$ into $G$. In practice, the non-saturating generator loss $\mathcal{L}_G=-\mathbb{E}_z\log D(G(z))$ supplies stronger early gradients than minimizing $\log(1-D(G(z)))$.

GANs can generate sharp samples in one forward pass, but their game is sensitive to learning rates, architecture, and the relative strength of the two players. Mode collapse occurs when many noise inputs map to a narrow set of outputs. A low generator loss alone does not prove diversity or faithful coverage, so sample quality and distributional metrics must be examined together.

Pretraining teaches a decoder-only language model to predict the next token from preceding tokens. This broad objective learns reusable linguistic and factual patterns, but it does not by itself teach the model to follow user instructions. Encoder models such as BERT instead use masked-token objectives, so not every Transformer is pretrained with next-token prediction.

Supervised fine-tuning (SFT) trains on curated prompt-response examples. LoRA and related parameter-efficient methods freeze most base weights and learn small adapters; they often reduce trainable parameters and memory, but the exact cost reduction depends on model, optimizer, hardware, and deployment choices.

Preference tuning targets behavior beyond imitation. RLHF can fit a reward model to human comparisons and optimize a policy against it, commonly with a reference-model constraint. Direct Preference Optimization (DPO) uses chosen/rejected pairs to optimize a closed-form classification-style objective relative to a reference policy. Neither method guarantees truthfulness or safety; data quality, evaluation, system design, and deployment controls still matter.

Retrieval-Augmented Generation (RAG) retrieves candidate passages, places selected evidence in the model context, and asks the model to answer from that evidence. It can improve grounding and update knowledge without changing model weights, but it does not guarantee factuality: retrieval can miss, rank irrelevant material, expose conflicting sources, or be ignored or misinterpreted by the generator. Production systems therefore evaluate retrieval and answer attribution separately.

Multimodal systems map text, images, audio, video, or other signals into representations a shared model can process. Designs may use modality-specific encoders, cross-attention, a unified token space, or combinations of these. The durable idea is learned communication between modalities, not any particular product version.

A tool-using agent lets a model choose structured actions such as search, calculation, code execution, or database queries, observe results, and continue. The surrounding application owns permissions, validation, stopping conditions, and audit trails. More autonomous loops create more opportunities for compounding error, so constrained tools, explicit budgets, and human approval for consequential actions are architectural requirements rather than optional polish.