Unit 17

Deep Learning Foundations

understanding neural networks and the mathematical principles behind modern AI systems

Fully-Connected Deep Neural Network Nodes

Read diagram labels

Input layer: x₁, x₂, x₃
Hidden layer: h₁, h₂, h₃
Output layer: ŷ

Core Concepts Covered

Multilayer perceptrons, activation functions, tensor shapes, and parameter counts
Backpropagation, cross-entropy, optimization, and training hygiene
Convolutional neural networks, MNIST classification, and error analysis

Local Setup Recommendation

To execute and experiment with the code cells below on your local machine, ensure you have set up your isolated virtual environments and scientific libraries by following the detailed protocols in Unit 03: Environment Setup or run them in Google Colab.

1. MLPs: Layers, Shapes, and Activations

A multilayer perceptron, or MLP, learns by passing a vector through a stack of affine transformations and nonlinear activation functions. For one layer, the calculation is $z=Wx+b$ and $h=\phi(z)$ . The matrix $W$ mixes the input features, the bias $b$ shifts each unit, and $\phi$ gives the network the nonlinearity it needs to represent curved decision boundaries.

Shapes tell you whether a network is wired correctly. If $x\in\mathbb{R}^{d_{in}}$ and a layer has $d_{out}$ units, then $W\in\mathbb{R}^{d_{out}\times d_{in}}$ , $b\in\mathbb{R}^{d_{out}}$ , and the output also belongs to $\mathbb{R}^{d_{out}}$ . A batch adds a leading dimension, so an input shaped (batch, features) remains a matrix as it moves through the MLP.

ReLU uses $\operatorname{ReLU}(z)=\max(0,z)$ and is a strong default for hidden layers. Sigmoid maps values into $(0,1)$ but can produce very small gradients when its input is far from zero. GELU gates inputs smoothly instead of clipping every negative value. Without any nonlinear activation, several affine layers collapse into one affine transformation, no matter how many layers you stack.

Jupyter Code Notebook Cell

Python 3 (ipykernel)

In [1]:

import torch
import torch.nn as nn

x = torch.tensor([-2.0, -0.5, 0.0, 1.5, 3.0])

relu = nn.ReLU()
sigmoid = nn.Sigmoid()
gelu = nn.GELU()

print("Original:", x.tolist())
print("ReLU:    ", relu(x).tolist())
print("Sigmoid: ", [round(value, 3) for value in sigmoid(x).tolist()])
print("GELU:    ", [round(value, 3) for value in gelu(x).tolist()])

Out [1]:

Original: [-2.0, -0.5, 0.0, 1.5, 3.0]
ReLU:     [0.0, 0.0, 0.0, 1.5, 3.0]
Sigmoid:  [0.119, 0.378, 0.5, 0.818, 0.953]
GELU:     [-0.045, -0.154, 0.0, 1.399, 2.996]

Worked Example 1

Trace a Small MLP Forward Pass

Problem

Let $x=[1,2]^T$ , $W_1=\begin{bmatrix}1&0\\-1&1\end{bmatrix}$ , $b_1=[0,0]^T$ , $W_2=[2,-1]$ , and $b_2=0.5$ . Calculate the output when the hidden layer uses ReLU.

Step-by-step solution

1.Compute the hidden pre-activation: $z_1=W_1x+b_1=[1,1]^T$ .

2.Apply ReLU: $h=\operatorname{ReLU}(z_1)=[1,1]^T$ .

3.Compute the output: $\hat y=W_2h+b_2=2(1)-1(1)+0.5=1.5$ .

Final answer and interpretation

The forward pass produces $\hat y=1.5$ . Each shape agrees: a two-value input becomes two hidden values and then one output.

Write the shape beside every tensor before multiplying. Most implementation mistakes become obvious at that point.

Worked Example 2

Count the Parameters in an MNIST MLP

Problem

How many trainable parameters are in an MLP with 784 inputs, 128 hidden units, and 10 output logits? Include every bias.

Step-by-step solution

1.The first layer has $784\times128=100{,}352$ weights and $128$ biases, for $100{,}480$ parameters.

2.The output layer has $128\times10=1{,}280$ weights and $10$ biases, for $1{,}290$ parameters.

Final answer and interpretation

Add the layers: $100{,}480+1{,}290=101{,}770$ trainable parameters.

A fully connected layer pays for a separate weight between every input and every output unit.

2. Learning with Backpropagation

A classifier returns one logit for each class. Softmax converts those logits into probabilities, and cross-entropy penalizes the probability assigned to the correct class: $p_k=\frac{e^{z_k}}{\sum_j e^{z_j}},\qquad \mathcal{L}=-\log p_y.$ During training, PyTorch's CrossEntropyLoss expects raw logits, so you should not apply softmax to the model output first.

Backpropagation applies the chain rule from the loss back through every operation. An optimizer then updates each parameter with a rule such as $w_{new}=w_{old}-\eta\frac{\partial\mathcal{L}}{\partial w}$ . In a PyTorch loop, clear old gradients, run the forward pass, calculate the loss, call loss.backward(), and then call optimizer.step().

Initialization and training mode matter. Xavier or He initialization keeps early signals at a useful scale, while setting every weight to zero fails to break symmetry. Batch normalization can stabilize intermediate activations, and dropout randomly masks activations during training. Call model.train() while updating parameters and model.eval() for validation or testing so dropout and normalization behave correctly.

Jupyter Code Notebook Cell

Python 3 (ipykernel)

In [1]:

import torch
import torch.nn as nn

torch.manual_seed(42)

class TinyMLP(nn.Module):
    def __init__(self):
        super().__init__()
        self.network = nn.Sequential(
            nn.Flatten(),
            nn.Linear(28 * 28, 128),
            nn.ReLU(),
            nn.Linear(128, 10),
        )

    def forward(self, images):
        return self.network(images)

model = TinyMLP()
images = torch.zeros(4, 1, 28, 28)
logits = model(images)
parameter_count = sum(parameter.numel() for parameter in model.parameters())

print("Logit shape:", tuple(logits.shape))
print("Parameters:", parameter_count)

Out [1]:

Logit shape: (4, 10)
Parameters: 101770

Worked Example 1

Calculate Softmax Cross-Entropy

Problem

A three-class model returns logits $[2,1,0]$ , and the correct class is class $0$ . Find the softmax probabilities and cross-entropy loss.

Step-by-step solution

1.Exponentiate the logits: $[e^2,e^1,e^0]\approx[7.389,2.718,1]$ .

2.Divide by their sum $11.107$ : $p\approx[0.665,0.245,0.090]$ .

3.Use the correct-class probability: $\mathcal{L}=-\log(0.665)\approx0.408$ .

Final answer and interpretation

The model assigns the largest probability to the correct class, but the nonzero loss leaves room for greater confidence.

Worked Example 2

Apply One SGD Update

Problem

A weight is $1.2$ , its gradient is $0.5$ , and the learning rate is $0.1$ . What value does one SGD step produce?

Step-by-step solution

1.Start with $w_{new}=w_{old}-\eta\nabla_w\mathcal{L}$ .

2.Substitute the values: $w_{new}=1.2-0.1(0.5)$ .

Final answer and interpretation

The updated weight is $w_{new}=1.15$ . The positive gradient makes gradient descent move the weight downward.

3. CNNs: Local Filters and Feature Maps

An image batch is usually arranged as (batch, channels, height, width). A convolutional neural network keeps that spatial layout and slides small learned kernels over local patches. Each output channel comes from one kernel bank and forms a feature map that records where a learned pattern appears.

The same kernel is reused at every position. This parameter sharing lets a feature detector work anywhere in an image and uses far fewer weights than connecting every pixel to every hidden unit. Stacking layers grows the receptive field, so early layers can respond to edges while later layers combine them into larger shapes.

For dilation $1$ , an input width $W$ , kernel size $K$ , padding $P$ , and stride $S$ produce: $W_{out}=\left\lfloor\frac{W+2P-K}{S}\right\rfloor+1.$ A convolution with $C_{in}$ input channels and $C_{out}$ output channels has $C_{out}(C_{in}K_hK_w+1)$ parameters when each output channel includes a bias. Pooling reduces width and height without learning new weights.

Jupyter Code Notebook Cell

Python 3 (ipykernel)

In [1]:

import torch
import torch.nn as nn

class TinyCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 4, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2),
        )
        self.classifier = nn.Sequential(
            nn.Flatten(),
            nn.Linear(4 * 14 * 14, 10),
        )

    def forward(self, images):
        return self.classifier(self.features(images))

images = torch.zeros(2, 1, 28, 28)
model = TinyCNN()
feature_maps = model.features(images)
logits = model(images)
parameter_count = sum(parameter.numel() for parameter in model.parameters())

print("Input:", tuple(images.shape))
print("Feature maps:", tuple(feature_maps.shape))
print("Logits:", tuple(logits.shape))
print("Parameters:", parameter_count)

Out [1]:

Input: (2, 1, 28, 28)
Feature maps: (2, 4, 14, 14)
Logits: (2, 10)
Parameters: 7890

Worked Example 1

Apply a 3 by 3 Edge Filter

Problem

Use the patch $X=\begin{bmatrix}1&1&0\\1&0&0\\1&1&0\end{bmatrix}$ and kernel $K=\begin{bmatrix}1&0&-1\\1&0&-1\\1&0&-1\end{bmatrix}$ . Calculate their elementwise product sum with zero bias.

Step-by-step solution

1.Multiply matching entries in the first row: $1(1)+1(0)+0(-1)=1$ .

2.The second and third rows also contribute $1$ each.

3.Add the row contributions: $1+1+1=3$ .

Final answer and interpretation

This patch produces activation $3$ , a strong response to the left-heavy vertical pattern.

Worked Example 2

Trace CNN Shapes and Parameters

Problem

The TinyCNN applies four $3\times3$ kernels with padding $1$ to one-channel $28\times28$ images, then uses $2\times2$ pooling and a ten-class linear layer. Find the spatial shapes and total parameters.

Step-by-step solution

1.The convolution preserves the spatial size: $(28+2-3)/1+1=28$ , so its output is $(4,28,28)$ per image.

2.Pooling halves each spatial dimension, producing $(4,14,14)$ and therefore $4\times14\times14=784$ flattened values.

3.The convolution has $4(1\times3\times3+1)=40$ parameters. The classifier has $784\times10+10=7{,}850$ .

Final answer and interpretation

The total is $40+7{,}850=7{,}890$ parameters, compared with $101{,}770$ in the earlier MLP.

This tiny comparison shows the efficiency gained by local connectivity and shared convolution weights.

4. MNIST 0-9 Classification Starter

MNIST contains grayscale images of handwritten digits from $0$ through $9$ . Each image has shape $(1,28,28)$ after ToTensor, and each label is an integer class index. The normalization constants $0.1307$ and $0.3081$ are the canonical mean and standard deviation used in PyTorch's MNIST example.

The original training set has $60{,}000$ images. We reserve $5{,}000$ for validation and use a seeded generator so the split can be reproduced. The training loader shuffles examples each epoch; validation and test loaders do not. The separate test set contains $10{,}000$ images and should remain untouched until model choices are finished.

Paste this starter code into a local Python environment or Google Colab, then write your own MLP or CNN, loss function, optimizer, training loop, and evaluation loop around the loaders it creates. download=True writes MNIST under ./data when you run the cell. The website only displays the starter code; it does not download the dataset, provide a finished model, or train anything in your browser.

Jupyter Code Notebook Cell

Python 3 (ipykernel)

In [1]:

import torch
import torchvision
from torchvision import transforms
from torch.utils.data import DataLoader, random_split

transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.1307,), (0.3081,)),
])

full_train_dataset = torchvision.datasets.MNIST(
    root="./data",
    train=True,
    download=True,
    transform=transform,
)

test_dataset = torchvision.datasets.MNIST(
    root="./data",
    train=False,
    download=True,
    transform=transform,
)

train_dataset, val_dataset = random_split(
    full_train_dataset,
    [55_000, 5_000],
    generator=torch.Generator().manual_seed(42),
)

batch_size = 64
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False)
test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)

print("Train:", len(train_dataset))
print("Validation:", len(val_dataset))
print("Test:", len(test_dataset))

Out [1]:

Train: 55000
Validation: 5000
Test: 10000

Worked Example 1

Count the MNIST Mini-Batches

Problem

With batch size $64$ , how many batches do the $55{,}000$ training images, $5{,}000$ validation images, and $10{,}000$ test images produce?

Step-by-step solution

1.Training uses $\lceil55{,}000/64\rceil=860$ batches.

2.Validation uses $\lceil5{,}000/64\rceil=79$ batches.

3.Testing uses $\lceil10{,}000/64\rceil=157$ batches.

Final answer and interpretation

The loaders produce $860$ , $79$ , and $157$ batches. The final batch in each split may contain fewer than $64$ images.

5. Practice Workflow and Error Analysis

Train the MLP and TinyCNN with the same data split, batch size, loss, optimizer, learning rate, and epoch budget. That controlled comparison makes architecture the main changing factor. Track training and validation loss after each epoch, but use validation results, not test results, to choose hyperparameters or decide when to stop.

Once the design is fixed, evaluate the test set once. Overall accuracy gives a useful summary, while a confusion matrix shows which digits the model mixes up. Per-class recall $\frac{TP_c}{TP_c+FN_c}$ reveals whether a strong average is hiding weak performance on a particular digit. Looking at misclassified images often exposes faint strokes, unusual handwriting, or systematic preprocessing mistakes.

A good practice report includes both model parameter counts, training curves, validation accuracy, final test accuracy, per-class recall, a confusion matrix, and a small grid of errors. Explain what changed between the MLP and CNN rather than reporting only the larger score.

Hands-On Kaggle Challenge

Intel Image Classification: Optional CNN Extension

After you finish the guided MNIST exercise, use this separate image dataset to practice adapting a CNN to a new problem. Inspect its class folders and image shapes, then update the transforms, input channels, output classes, and data loaders instead of reusing the MNIST pipeline unchanged.

Refer back to Unit 03 for environment and Colab setup!

Join Kaggle Challenge

Interactive Practice Quiz

Test your understanding with instant feedback

QUESTION 01

An MLP layer maps 20 input features to 8 hidden units. What shape must its weight matrix have when the layer computes $z=Wx+b$ ?

QUESTION 02

Why do MLPs place nonlinear activations between affine layers?

QUESTION 03

For logits $[2,1,0]$ and correct class $0$ , which pair is closest to the correct-class probability and cross-entropy loss?

QUESTION 04

Why is setting every hidden-layer weight to zero a serious initialization mistake?

QUESTION 05

Which order matches a standard PyTorch parameter update?

QUESTION 06

A $28\times28$ image passes through a $3\times3$ convolution with stride $1$ and padding $1$ . What is the output width?

QUESTION 07

What does parameter sharing mean in a convolutional layer?

QUESTION 08

Why is a validation split kept separate from the training and test splits?

QUESTION 09

Why should validation and testing normally use `model.eval()`?

QUESTION 10

Deep Learning Foundations

Core Concepts Covered

Local Setup Recommendation

1. MLPs: Layers, Shapes, and Activations

Trace a Small MLP Forward Pass

Count the Parameters in an MNIST MLP

2. Learning with Backpropagation

Calculate Softmax Cross-Entropy

Apply One SGD Update

3. CNNs: Local Filters and Feature Maps

Apply a 3 by 3 Edge Filter

Trace CNN Shapes and Parameters

4. MNIST 0-9 Classification Starter

Count the MNIST Mini-Batches

5. Practice Workflow and Error Analysis

Hands-On Kaggle Challenge

Interactive Practice Quiz

An MLP layer maps 20 input features to 8 hidden units. What shape must its weight matrix have when the layer computes $z=Wx+b$ ?

Why do MLPs place nonlinear activations between affine layers?

For logits $[2,1,0]$ and correct class $0$ , which pair is closest to the correct-class probability and cross-entropy loss?

Why is setting every hidden-layer weight to zero a serious initialization mistake?

Which order matches a standard PyTorch parameter update?

A $28\times28$ image passes through a $3\times3$ convolution with stride $1$ and padding $1$ . What is the output width?

What does parameter sharing mean in a convolutional layer?

Why is a validation split kept separate from the training and test splits?

Why should validation and testing normally use `model.eval()`?

Why will a CNN often use fewer parameters than an MLP on raw MNIST pixels?

Further Readings

Deep Learning Foundations

Core Concepts Covered

Local Setup Recommendation

1. MLPs: Layers, Shapes, and Activations

Trace a Small MLP Forward Pass

Count the Parameters in an MNIST MLP

2. Learning with Backpropagation

Calculate Softmax Cross-Entropy

Apply One SGD Update

3. CNNs: Local Filters and Feature Maps

Apply a 3 by 3 Edge Filter

Trace CNN Shapes and Parameters

4. MNIST 0-9 Classification Starter

Count the MNIST Mini-Batches

5. Practice Workflow and Error Analysis

Hands-On Kaggle Challenge

Interactive Practice Quiz

An MLP layer maps 20 input features to 8 hidden units. What shape must its weight matrix have when the layer computes z=Wx+bz=Wx+bz=Wx+b?

Why do MLPs place nonlinear activations between affine layers?

For logits [2,1,0][2,1,0][2,1,0] and correct class 000, which pair is closest to the correct-class probability and cross-entropy loss?

Why is setting every hidden-layer weight to zero a serious initialization mistake?

Which order matches a standard PyTorch parameter update?

A 28×2828\times2828×28 image passes through a 3×33\times33×3 convolution with stride 111 and padding 111. What is the output width?

What does parameter sharing mean in a convolutional layer?

Why is a validation split kept separate from the training and test splits?

Why should validation and testing normally use model.eval()?

Why will a CNN often use fewer parameters than an MLP on raw MNIST pixels?

Further Readings

An MLP layer maps 20 input features to 8 hidden units. What shape must its weight matrix have when the layer computes $z=Wx+b$ ?

For logits $[2,1,0]$ and correct class $0$ , which pair is closest to the correct-class probability and cross-entropy loss?

A $28\times28$ image passes through a $3\times3$ convolution with stride $1$ and padding $1$ . What is the output width?

Why should validation and testing normally use `model.eval()`?