Home / MLT / Week 12

Week 12 · Neural networks

Official syllabus line: Artificial Neural networks: Multiclass classification

About 90 minEnd Term material (see exam radar)BSCS2007 · Arun Rajkumar

Before you start: MLF week 2 (chain rule), MLF week 7 (gradient descent), MLT week 9 (logistic regression).

Watch first: the animated story of this weekAbout 1-2 minutes of real 3D animation with a friendly voice. Pause any time.

Read the words of the video

Imagine a huge room of tiny voters. Each voter checks a few things, decides their opinion, then passes it to the next row of voters. This continues until the last row makes a final decision, like "This photo is a cat!" This is like a neural network.
Each voter is a neuron, and each row is a layer. Stacking layers helps the network build complex ideas from simple ones, like finding edges then shapes. Training means nudging their 'trust levels' by sending blame backward if the answer is wrong. This is how it learns.
Let's see training in action! Imagine a tiny chain of two voters. An input goes forward to get an answer. If it's wrong, we send the 'blame' backward through the chain. Each voter adjusts its trust level just a little, learning from the mistake to improve next time.
For your exams, remember to count all the 'trust levels' including biases. Also, non-linear 'squish' functions are key for powerful networks, and randomly starting weights is important! Keep practicing; you're building amazing machine-learning superpowers, one neuron at a time!

The one-minute story

Think of it like this

Picture a huge room of tiny voters. Each voter looks at a few numbers, decides how much to trust each one, adds up their opinion, and then gives a "how excited am I?" answer between low and high. Their answers are passed to a second row of voters, who weigh those answers in the same way, and so on, until a final row announces "this photo is a cat (92%)".

That room is a neural network. Each voter is a neuron; each row is a layer. A single neuron is just logistic regression from last week. Stacking layers lets the network build complicated features out of simple ones: edges, then shapes, then faces.

Training means starting with random trust levels (weights) and nudging them: show an example, see how wrong the answer was, and send the blame backwards through the network (backpropagation) to see how each weight should change. Repeat many times: that is gradient descent.

By the end of this week you can:

describe a neuron, activation functions, and a layer;
run a forward pass and count parameters;
use softmax and cross-entropy for multiclass classification;
explain backpropagation as the chain rule and do a small example;
list practical issues: overfitting, vanishing gradients, learning rate.

Words you need

Neuron (unit) - computes $a=g(\mathbf w^T\mathbf x+b)$ .
Activation function $g$ - the non-linear "squish".
Layer - a group of neurons that all see the same inputs. Input, hidden and output layers.
Weights / biases - the learnable numbers.
Forward pass - computing the output from the input.
Backpropagation - computing every weight's gradient by running the chain rule backwards.
Epoch - one full pass over the training data.

Idea 1: One neuron, and why we need activations

A neuron takes inputs $\mathbf x$ , forms $z=\mathbf w^T\mathbf x+b$ and outputs $a=g(z)$ .

Activation	Formula	Range	Notes
Sigmoid	$\frac1{1+e^{-z}}$	$(0,1)$	probability-like; saturates (gradient $\to0$ ) for large $\|z\|$
tanh	$\frac{e^z-e^{-z}}{e^z+e^{-z}}$	$(-1,1)$	centred at 0; also saturates
ReLU	$\max(0,z)$	$[0,\infty)$	cheap; no saturation for $z>0$ ; today's default
Softmax (output layer)	$\frac{e^{z_k}}{\sum_je^{z_j}}$	probabilities that sum to 1	for multiclass
Step	$1$ if $z>0$ else $0$	$\{0,1\}$	the perceptron's activation

Why non-linear? If $g$ were just the identity, then two layers compute $W_2(W_1\mathbf x)=(W_2W_1)\mathbf x$ , which is one linear layer. Stacking would add nothing. Non-linear activations are what let depth create richer and richer functions.

Special cases: a neuron with a step activation is the perceptron (week 9); with a sigmoid it is logistic regression. A neural network is a stack of logistic-regression-like units whose outputs feed the next row.

Idea 2: Layers and the forward pass

For a layer with weight matrix $W$ (one row per neuron), bias vector $\mathbf b$ and activation $g$ applied entry by entry:

\mathbf h=g(W\mathbf x+\mathbf b)

and the next layer takes $\mathbf h$ as its input.

Solved example: 1. A forward pass and a parameter count

Question. A network has 2 inputs, one hidden layer of 2 ReLU neurons with $W_1=\begin{pmatrix}1&1\\-1&0\end{pmatrix}$ (biases $0$ ), and one linear output neuron with weights $(2,\,5)$ (bias 0). Compute the output for $\mathbf x=(1,2)$ . Then count the parameters of a $4\to5\to3$ network.

Solution.

Hidden pre-activations: $z=W_1\mathbf x=(1\cdot1+1\cdot2,\;-1\cdot1+0\cdot2)=(3,\,-1)$ .
ReLU: $\mathbf h=(\max(0,3),\,\max(0,-1))=(3,\,0)$ .
Output: $2\cdot3+5\cdot0=\mathbf6$ .

Parameter count for $4\to5\to3$ : layer 1 has $4\times5$ weights $+\,5$ biases $=25$ ; layer 2 has $5\times3+3=18$ . Total $=\mathbf{43}$ . In general a layer from $m$ to $n$ units has $mn+n$ parameters.

Idea 3: Multiclass output with softmax

For $K$ classes, the output layer has $K$ neurons producing scores (logits) $z_1,\dots,z_K$ . Softmax turns them into probabilities:

p_k=\frac{e^{z_k}}{\sum_je^{z_j}}

(All positive, adding to 1; the biggest score gets the biggest share.) The loss is the cross-entropy: $-\ln p_{y}$ where $y$ is the true class (with one-hot labels, $-\sum_ky_k\ln p_k$ ). With $K=2$ this is the logistic loss of week 9.

Solved example: 2. Softmax and loss

Question. Scores $(2,\,1,\,0)$ . Find the softmax probabilities and the cross-entropy loss if the true class is the first.

Solution. $e^2=7.389$ , $e^1=2.718$ , $e^0=1$ ; sum $=11.107$ . Probabilities: $0.665,\;0.245,\;0.090$ . Loss $=-\ln0.665=\mathbf{0.408}$ . If the true class were the third, the loss would be $-\ln0.090=2.41$ : a big penalty for a confident mistake.

Idea 4: Training and backpropagation

Define a loss $L$ (squared error for regression, cross-entropy for classification), then run gradient descent on all weights:

w\leftarrow w-\eta\,\frac{\partial L}{\partial w}

The gradients come from the chain rule (MLF week 2). Backpropagation is simply an efficient bookkeeping of it: compute everything forward and remember the intermediate values, then walk backwards from the loss, multiplying local derivatives and reusing results, so one backward pass yields all the gradients.

Solved example: 3. Backpropagation on a tiny chain

Question. A network is $h=w_1x$ , then $\hat y=w_2h$ , with loss $L=\frac12(\hat y-y)^2$ . Take $x=1$ , $y=1$ , $w_1=0.5$ , $w_2=3$ and learning rate $\eta=0.1$ . Compute the gradients and do one update.

Solution.

Forward: $h=0.5$ , $\hat y=1.5$ , $L=\frac12(0.5)^2=0.125$ .

Backward (chain rule, from the loss to the weights):

$\dfrac{\partial L}{\partial\hat y}=\hat y-y=0.5$ .
$\dfrac{\partial L}{\partial w_2}=\dfrac{\partial L}{\partial\hat y}\cdot\dfrac{\partial\hat y}{\partial w_2}=0.5\cdot h=0.25$ .
$\dfrac{\partial L}{\partial h}=\dfrac{\partial L}{\partial\hat y}\cdot w_2=0.5\cdot3=1.5$ .
$\dfrac{\partial L}{\partial w_1}=\dfrac{\partial L}{\partial h}\cdot x=1.5$ .

Update: $w_2=3-0.1(0.25)=2.975$ , $w_1=0.5-0.1(1.5)=0.35$ .

Check: the new prediction is $2.975\times0.35=1.041$ , much closer to $y=1$ ; the loss fell from $0.125$ to about $0.0009$ . ✓ Notice $\frac{\partial L}{\partial h}$ was computed once and reused: that reuse is the whole trick of backprop.

Idea 5: Practical matters

Non-convex: the loss surface has many local minima and saddles, so results depend on the random start (weights are initialised randomly, not to zero: if all weights start equal, all neurons in a layer stay identical).
Mini-batch SGD with a sensible learning rate (too big diverges, too small crawls).
Vanishing gradients: with sigmoid/tanh in deep nets, gradients multiply many numbers below 1 and shrink to nothing; ReLU helps.
Overfitting: networks have huge numbers of parameters and can memorise the training data. Remedies: more data, weight decay (an $\ell_2$ penalty, as in ridge), early stopping (stop when validation error rises), dropout.
Universal approximation: even one hidden layer with enough neurons can approximate any reasonable function; depth makes it far more efficient.
Scale your inputs before training.

Your turn

Your turn: 1. Forward pass

A single neuron has $\mathbf w=(2,-1)$ , $b=1$ and a sigmoid activation. Compute its output for $\mathbf x=(1,3)$ .

Need a hint?

$z=\mathbf w^T\mathbf x+b$ .

Show the full solution

$z=2-3+1=0$ . $\sigma(0)=\mathbf{0.5}$ .

Your turn: 2. Count parameters

A network has inputs of size 10, hidden layers of size 8 and 6, and 3 outputs. How many weights and biases?

Need a hint?

Layer by layer: $mn+n$ .

Show the full solution

$10\to8$ : $80+8=88$ . $8\to6$ : $48+6=54$ . $6\to3$ : $18+3=21$ . Total $=\mathbf{163}$ .

Your turn: 3. Softmax

Scores $(0,\,0,\,0)$ . What are the probabilities? Scores $(5,\,0)$ ?

Need a hint?

Equal scores give equal shares.

Show the full solution

$(\frac13,\frac13,\frac13)$ . For $(5,0)$ : $\frac{e^5}{e^5+1}=\frac{148.4}{149.4}=0.993$ and $0.007$ .

Your turn: 4. Why activations?

A network has two layers with identity activations: $\hat y=W_2(W_1\mathbf x)$ . Show it is no more powerful than one layer.

Need a hint?

Matrix multiplication is associative.

Show the full solution

$W_2(W_1\mathbf x)=(W_2W_1)\mathbf x=W'\mathbf x$ with $W'=W_2W_1$ . It is a single linear layer. Non-linear activations are what make depth useful.

Your turn: 5. Backprop step

For the chain $h=w_1x$ , $\hat y=w_2h$ , $L=\frac12(\hat y-y)^2$ with $x=2$ , $y=0$ , $w_1=1$ , $w_2=1$ , give $\partial L/\partial w_1$ and $\partial L/\partial w_2$ .

Need a hint?

Forward first: $h=2$ , $\hat y=2$ .

Show the full solution

$\partial L/\partial\hat y=2$ . $\partial L/\partial w_2=2\cdot h=4$ . $\partial L/\partial h=2\cdot w_2=2$ . $\partial L/\partial w_1=2\cdot x=4$ .

Exam radar

In the MLT End Term: Activation Functions in 19 of 49 papers, Neural Networks in 16, Architecture in 18, ReLU in 11, Sigmoid in 10, Perceptron in 13.

Typical: forward pass through a tiny network; parameter counting; softmax values; "which activation is needed to make the network non-linear?"; derivative of sigmoid ( $\sigma(1-\sigma)$ ); backprop gradient for a small chain; "what happens with identity activations?".

Watch out - real exam trap

Count the biases too: $mn+n$ per layer.
Without non-linear activations, depth adds nothing.
Softmax outputs sum to 1; sigmoid outputs do not need to.
Do not initialise all weights to the same value.
Sigmoid derivative is $\sigma(z)(1-\sigma(z))\le0.25$ , which is why deep sigmoid nets suffer vanishing gradients.

Quick quiz

How many parameters (weights and biases) are in a 4 to 5 to 3 network?

(4*5 + 5) + (5*3 + 3) = 25 + 18 = 43.

Why do neural networks need non-linear activation functions?

Otherwise any stack of layers collapses to one linear map.
To make the numbers smaller.
To remove the bias.
To speed up the forward pass.

Composition of linear maps is linear, so depth would add no expressive power.

The softmax of the scores (1, 1) is...

(1, 1)
(0, 1)
(0.5, 0.5)
(0.25, 0.75)

Equal scores give equal probabilities that sum to 1.

Select all TRUE statements. (select all that apply)

Backpropagation applies the chain rule to compute gradients.
A sigmoid neuron is a logistic regression unit.
Setting all initial weights to the same constant is good practice.
The loss surface of a neural network is convex.

Identical initial weights keep neurons identical, and neural network losses are non-convex.

Recap

Remember this

Neuron $a=g(\mathbf w^T\mathbf x+b)$ ; layer $\mathbf h=g(W\mathbf x+\mathbf b)$ . Activations: sigmoid, tanh, ReLU, softmax. Non-linearity is essential.
Parameters of a layer $m\to n$ : $mn+n$ .
Softmax + cross-entropy for multiclass.
Backpropagation = chain rule with reuse; update $w\leftarrow w-\eta\,\partial L/\partial w$ . Non-convex; random initialisation; watch overfitting and vanishing gradients.

Formulas to keep

$\mathbf h=g(W\mathbf x+\mathbf b)$ ; ReLU $\max(0,z)$ ; sigmoid $\frac1{1+e^{-z}}$ with derivative $\sigma(1-\sigma)$
Softmax $p_k=e^{z_k}/\sum_je^{z_j}$ ; loss $-\ln p_{y}$
Parameters per layer: $mn+n$
Chain rule: $\frac{\partial L}{\partial w_1}=\frac{\partial L}{\partial\hat y}\frac{\partial\hat y}{\partial h}\frac{\partial h}{\partial w_1}$

Connected weeks (learn them together)

MLF week 7 - Finding the lowest point: gradient descent: Backpropagation is gradient descent plus the chain rule.
MLP week 11 - Neural networks in sklearn: Neural networks: the theory, then the code.
MLP week 7 - Multiclass classifiers: Multiclass classification.

Practise with real exam questions

These come straight from past papers (with the real figures and the official answer key). Open 18 real questions on this week All End Term papers