Unit 16

Deep Learning Foundations

understanding neural networks and the mathematical principles behind modern AI systems

Fully-Connected Deep Neural Network Nodes

Core Concepts Covered

Perceptrons, activation functions, and feedforward networks
Backpropagation derivations and loss optimization
Spatio-temporal architectures: CNNs, RNNs, and basic Transformers

Local Setup Recommendation

To execute and experiment with the code cells below on your local machine, ensure you have set up your isolated virtual environments and scientific libraries by following the detailed protocols in Unit 03: Environment Setup or run them in Google Colab.

1. Perceptrons, Activations, and Multilayer Networks

Deep Learning is a subset of machine learning based on Artificial Neural Networks. These networks are inspired by the biological structure of the brain, using interconnected computational layers to learn rich hierarchies of features from data.

• The Perceptron: The fundamental building block of neural networks. It takes input features $x$ , multiplies them by weights $w$ , adds a bias $b$ , and passes the result through an activation function:

$z = w^T x + b = \sum_{j=1}^d w_j x_j + b$

• Why We Need Non-Linear Activation Functions: Activation functions inject non-linearity into our network. Without them, stacking multiple hidden layers would collapse mathematically into a simple, single linear transformation ( $f(g(x)) = W_2(W_1 x) = W_{\text{combined}} x$ ). This would prevent the network from learning complex, non-linear relationships like curves, images, or circles:

- Sigmoid: $\sigma(z) = \frac{1}{1 + e^{-z}}$ (maps inputs to the interval $(0, 1)$ ; suffers from severe vanishing gradients in deep networks).

- ReLU (Rectified Linear Unit): $f(z) = \max(0, z)$ (the industry standard; computationally extremely fast to compute and prevents vanishing gradients for positive inputs).

- GELU (Gaussian Error Linear Unit): $f(z) = z \cdot \Phi(z)$ (used in modern architectures like BERT and GPT, smoothing ReLU using probability density factors to improve gradient updates).

Jupyter Code Notebook Cell

Python 3 (ipykernel)

In [1]:

# PyTorch representation of neural activation layers
import torch
import torch.nn as nn

# Sample tensor input
x = torch.tensor([-2.0, -0.5, 0.0, 1.5, 3.0])

# Define standard activation layers
relu = nn.ReLU()
sigmoid = nn.Sigmoid()
gelu = nn.GELU()

print("Original Tensors:   ", x.tolist())
print("ReLU Activation:   ", relu(x).tolist())
print("Sigmoid Activation:", [round(val, 3) for val in sigmoid(x).tolist()])
print("GELU Activation:   ", [round(val, 3) for val in gelu(x).tolist()])

Out [1]:

Original Tensors:    [-2.0, -0.5, 0.0, 1.5, 3.0]
ReLU Activation:    [0.0, 0.0, 0.0, 1.5, 3.0]
Sigmoid Activation: [0.119, 0.378, 0.5, 0.818, 0.953]
GELU Activation:    [-0.045, -0.154, 0.0, 1.399, 2.996]

2. Forward Propagation, Backpropagation, and Training Hygiene

Training a deep network consists of a three-step cycle repeated over thousands of epochs:

1. Forward Propagation: The input passes forward through successive layers (features are mapped to weights, activated, and passed to the next layer), producing a final prediction $\hat{y}$ .

2. Loss Computation: We evaluate prediction error using a loss function (e.g., Mean Squared Error (MSE) for regression, or Cross-Entropy for classification).

3. Backward Propagation: We compute the gradient of the loss with respect to all weights using the Chain Rule, moving backward from output to input. We then adjust weights using a gradient descent optimizer.

• Weight Initialization: Initializing weights to zero kills gradient updates due to symmetry (every neuron in a layer learns the same features). We use random initializations (like Xavier/Glorot for Sigmoid, or He/Kaiming initialization for ReLU) to keep gradients alive and distinct.

• Batch Normalization: Standardizes layer inputs within each mini-batch during training (mean=0, variance=1). This stabilizes training, reduces sensitivity to initialization, and acts as a regularizer.

3. Advanced Architectures: CNNs, RNNs, and Transformers

While Fully-Connected (Dense) layers are versatile, specialized neural architectures are required to handle structural data patterns:

• Convolutional Neural Networks (CNNs): Designed for grid-structured data like images. Instead of flattening images and destroying spatial relationships, CNNs slide small spatial kernels (filters) over the image, sharing weights across pixels. This captures translation-invariant visual features (like edges, textures, and objects).

• Recurrent Neural Networks (RNNs): Designed for sequential data like text or timeseries. They maintain a recurrent hidden state $h_t$ that carries historical information across time steps. Standard RNNs suffer from vanishing gradients over long sequences, which is resolved by gated cell designs like LSTMs (Long Short-Term Memory).

• Transformers: The architecture powering modern large language models. It replaces recurrence entirely with the Self-Attention Mechanism, allowing the network to process entire sequences in parallel and weigh relationships between words regardless of distance.

Hands-On Kaggle Challenge

Practice your model-building skills in a real-world sandbox

Test the concepts you learned in this unit by participating in a classic, beginner-friendly Kaggle competition! Write and execute your script locally or on Google Colab, and submit a "Late Submission" to benchmark your model score on the live public leaderboard.

Refer back to Unit 03 for environment and Colab setup!

Join Kaggle Challenge

Interactive Practice Quiz

Test your understanding with instant feedback

QUESTION 01

Why is a non-linear activation function (like ReLU or Sigmoid) mandatory in deep neural networks?

QUESTION 02

Which of the following describes the 'Vanishing Gradient Problem' commonly seen with Sigmoid activation functions?

QUESTION 03

Why is initializing all neural network parameter weights to exactly zero a major error?

QUESTION 04

Which neural architecture is specifically designed to slide weight-shared kernels over images to capture translation-invariant spatial features?

QUESTION 05

What is the primary benefit of Batch Normalization layers in deep learning architectures?

QUESTION 06

Why do we use Xavier (Glorot) or He (Kaiming) initialization methods instead of standard constant random numbers?

QUESTION 07

Which of the following describes the GELU (Gaussian Error Linear Unit) activation function compared to standard ReLU?

QUESTION 08

What is the key structural breakthrough of Recurrent Neural Networks (RNNs) for temporal datasets?

QUESTION 09

The Transformer architecture replaces sequential recurrence entirely with which of the following mechanisms?

QUESTION 10