Modern AI Systems and Generative AI

Modern Generative AI is built almost entirely on the Transformer Architecture (introduced by Vaswani et al. in 2017). The core breakthrough of the Transformer is the Self-Attention Mechanism, which allows models to process sequences in parallel and dynamically calculate relationships between any two tokens in a sequence, completely replacing sequential recurrence.

To calculate self-attention, we project our input embeddings into three vectors for each token: Queries ($Q$), Keys ($K$), and Values ($V$) using learned linear weights. The attention weights are computed using the Scaled Dot-Product formula: $$\text{Attention}(Q, K, V) = \text{softmax}\left( \frac{Q K^T}{\sqrt{d_k}} \right) V$$

Self-Attention Calculation Trace: Let's calculate self-attention step-by-step for a simple sequence of two words: *I* ($x_1 = [1, 0]^T$) and *learn* ($x_2 = [0, 1]^T$). For simplicity, let our projection matrices for Query ($W_q$), Key ($W_k$), and Value ($W_v$) be identity matrices, meaning $Q = X, K = X, V = X$:

1. Compute Dot Products ($Q K^T$): Measures pairwise raw similarities: $$Q K^T = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}$$ This shows that each word is highly similar to itself ($1.0$) and unrelated to the other ($0.0$).

2. Scale by $\sqrt{d_k}$: If our key dimension is $d_k = 4$, then $\sqrt{d_k} = 2$. We divide the similarity matrix by $2$: $$\frac{Q K^T}{\sqrt{d_k}} = \begin{bmatrix} 0.5 & 0.0 \\ 0.0 & 0.5 \end{bmatrix}$$

3. Apply Softmax: Centers and converts rows into probabilities: $$\text{softmax}\left(\begin{bmatrix} 0.5 & 0.0 \\ 0.0 & 0.5 \end{bmatrix}\right) = \begin{bmatrix} 0.62 & 0.38 \\ 0.38 & 0.62 \end{bmatrix}$$ These represent our Attention Weights. Notice how *I* now pays $38\%$ of its attention to *learn*!

4. Multiply by $V$: Computes the final attention-weighted representations: $$\text{Attention} = \begin{bmatrix} 0.62 & 0.38 \\ 0.38 & 0.62 \end{bmatrix} \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} = \begin{bmatrix} 0.62 & 0.38 \\ 0.38 & 0.62 \end{bmatrix}$$ This proves mathematically how self-attention integrates contextual representations of neighboring words!

• Multi-Head Attention: Repeats this process in parallel across different subspaces, allowing the model to simultaneously focus on different parts of the sentence (such as tracking subject-verb dependencies alongside pronoun targets).

Building a production large language model consists of a highly rigorous three-stage training lifecycle:

1. Pre-training (Self-Supervised Learning): The model is trained on trillions of words from the web to predict the next token in a sequence (causal language modeling). This phase is computationally massive and teaches the model grammar, reasoning, and world facts.

2. Supervised Fine-Tuning (SFT): The pre-trained base model is fine-tuned on curated instruction-response pairs (e.g., 'What is a prime number?' followed by a helpful answer). This transforms the auto-regressive text completer into a helpful chatbot assistant.

• Parameter-Efficient Fine-Tuning (PEFT / LoRA): Since full SFT on billions of parameters is expensive, LoRA (Low-Rank Adaptation) freezes the main model weights and injects small trainable rank decomposition matrices into the attention layers, reducing training costs by $99\%$.

3. Alignment (RLHF & DPO): To ensure the model is helpful, honest, and harmless, we align it with human preferences. Reinforcement Learning from Human Feedback (RLHF) trains a secondary 'reward model' on human preference rankings and uses PPO policy gradients to optimize the LLM. Direct Preference Optimization (DPO) bypasses reinforcement complexities, optimizing the policy directly using a closed-form loss over paired preferences.

To deploy LLMs into real-world production systems, we extend their capabilities using advanced application frameworks:

• Retrieval-Augmented Generation (RAG): LLMs suffer from knowledge cutoffs and hallucinations. RAG bypasses this by converting a user's query into an embedding vector, searching a Vector Database for matching semantic documents, and prepending those documents as reference context inside the LLM prompt. This allows the model to answer questions using private or real-time documents accurately without needing fine-tuning.

• Multimodality: Modern models (like GPT-4o or Claude 3.5 Sonnet) integrate visual, text, and audio tokens into a single unified transformer space, allowing them to reason across images, sound waves, and code files simultaneously.

• Autonomous AI Agents: Systems where an LLM is placed in a closed loop with tools (like web search, python compilers, or APIs). The LLM acts as the central 'brain' planning steps, calling appropriate tools, analyzing results, and adjusting its plan dynamically until the goal is achieved.