Lecture 13: Neural Networks — Inside the Black Box

Babafemi Sorinolu

Learning Goals

By the end of this lecture, you should be able to:

  1. Name the parts of a neural network: nodes, weights, biases, activation functions, layers.
  2. Push an input through a small network by hand to get a prediction.
  3. Explain how a network builds a squiggle by stretching, flipping and adding activation functions.
  4. Measure how good a fit is with the sum of squared residuals (SSR).
  5. Use the chain rule and gradient descent to train one parameter, which is the core idea of backpropagation.
  6. Say when a neural network is (and isn’t) the right tool.

Note

Lecture 10 gave the big picture (PyTorch, CNNs). Today we open the black box and do the math by hand on a tiny network.

Part 1: Why Neural Networks?

The Problem

A drug works at a medium dose, but not at low or high doses.

Logistic regression gives a nearly flat line while a neural network fits low-high-low data
  • Logistic regression has one S-shape, so it can’t go up and come back down.
  • A neural network can fit (almost) any shape: it is a “squiggle-fitting machine”.

Anatomy of a Neural Network

Network with one input, two hidden SoftPlus nodes, a sum node and one output
  • Nodes (boxes) are connected by arrows.
  • The numbers on the arrows are parameters, learned from data with backpropagation.
  • The curves inside the hidden nodes are activation functions.

Layers

Layer What it is In our example
Input layer the features we feed in 1 node: Dose
Hidden layer(s) nodes between input and output 1 hidden layer with 2 nodes
Output layer the prediction(s) 1 node: Effectiveness
  • Real networks have many input nodes, hidden layers and nodes per layer.
  • More layers and nodes means more complicated shapes can be fit, but also more risk of overfitting.
  • Choosing the number of layers and nodes is part of the “art” of neural networks.

Weights and Biases

Weights (×)

Numbers we multiply by:

\(-34.4,\ -2.52,\ -1.30,\ 2.28\)

They stretch or flip a value.

Biases (+)

Numbers we add:

\(+2.14,\ +1.29,\ -0.58\)

They shift a value up or down.

Familiar idea

A straight line \(y = 0.64x + 0.95\) also has a weight (slope 0.64) and a bias (intercept 0.95). A neural network is many of these, glued together with activation functions.

Activation Functions

ReLU, SoftPlus and Sigmoid activation functions
  • An activation function is just a math function: plug in \(x\), get \(y\).
  • Our example uses SoftPlus: \(\text{SoftPlus}(x) = \log(1 + e^x)\) (natural log).
  • Example: \(\text{SoftPlus}(2.14) = \log(1 + e^{2.14}) = \log(9.50) \approx 2.25\)
  • ReLU is the most popular in big networks. Sigmoid is common in teaching, rare in practice.

Check Your Understanding

  1. In “\(\times -34.4\ +\ 2.14\)”, which number is the weight and which is the bias?
  2. What does \(\text{ReLU}(-3)\) equal? And \(\text{ReLU}(5)\)?
  3. Why do we need activation functions at all? (Hint: what do you get if you only multiply and add?)
  1. Weight = −34.4 (multiply), bias = 2.14 (add).
  2. \(\text{ReLU}(-3) = 0\), \(\text{ReLU}(5) = 5\).
  3. Only multiplying and adding gives a straight line, no matter how many layers. Activation functions add the bends that let us fit squiggles.

Part 2: A Neural Network in Action

Forward Pass: The Plan

We’ll plug doses from 0 to 1 into the network and watch it build its output, one piece at a time.

  1. Top node: multiply by −34.4, add 2.14, apply SoftPlus, multiply by −1.30 → the blue curve
  2. Bottom node: multiply by −2.52, add 1.29, apply SoftPlus, multiply by 2.28 → the orange curve
  3. Add the blue and orange curves together
  4. Add the final bias (−0.58) → the prediction

Note

To keep the math simple, doses and effectiveness are scaled to go from 0 to 1. For now, assume the parameters have already been learned. We’ll see how in Part 3.

Step 1: The Top Node, Dose by Dose

Dose \(x = \text{Dose} \times (-34.4) + 2.14\) \(\text{SoftPlus}(x)\) \(\times (-1.30)\)
0.0 2.14 2.25 −2.93
0.2 −4.74 0.01 −0.01
0.4 −11.62 ≈ 0 ≈ 0
1.0 −32.26 ≈ 0 ≈ 0
Top node: linear transform, SoftPlus, then multiply by -1.30 giving the blue curve

Step 2: The Bottom Node

Same recipe, different numbers: multiply by −2.52, add 1.29, SoftPlus, then multiply by 2.28.

Bottom node: linear transform, SoftPlus, then multiply by 2.28 giving the orange curve

At dose 0: \(\;0 \times (-2.52) + 1.29 = 1.29 \;\to\; \text{SoftPlus}(1.29) = 1.53 \;\to\; 1.53 \times 2.28 = 3.50\)

Steps 3 & 4: Add Everything Up

Blue and orange curves added, then shifted by the final bias to fit the three training points

At dose 0: \(\;-2.93 + 3.50 = 0.57\), then \(\;0.57 + (-0.58) \approx 0.0\) ✓ (the drug is not effective)

Your Turn: Dose = 0.5

Compute the network’s prediction for a medium dose of 0.5.

Top node

\(0.5 \times (-34.4) + 2.14 = \;?\)

\(\text{SoftPlus}(\,?\,) \times (-1.30) = \;?\)

Bottom node

\(0.5 \times (-2.52) + 1.29 = \;?\)

\(\text{SoftPlus}(\,?\,) \times 2.28 = \;?\)

  • Top: \(-15.06 \to \text{SoftPlus} \approx 0 \to \times(-1.30) \approx 0\)
  • Bottom: \(0.03 \to \text{SoftPlus}(0.03) = \log(1 + e^{0.03}) \approx 0.71 \to \times 2.28 \approx 1.61\)
  • Output: \(0 + 1.61 - 0.58 = 1.03 \approx 1\), so a medium dose is effective. 🎉

The Big Idea

A neural network builds a complicated shape by taking simple activation functions and:

Operation Done by Effect on the curve
multiply the input weight before the node stretch / squeeze sideways (and flip)
add to the input bias before the node slide left or right
multiply the output weight after the node stretch / flip up and down
add curves together the sum node combine shapes
add the final bias final bias shift up or down

More nodes = more pieces to combine = more flexible squiggles.

Part 3: Backpropagation — How the Network Learns

The Problem: Finding the Parameters

  • So far we were given the weights and biases. Where do they come from?
  • Just like linear regression, we choose the values that make the predictions closest to the data.
  • We use gradient descent, but in a neural network the derivatives are found from the back to the front, so it’s called backpropagation.

Today’s simplified example

Pretend we already know every parameter except the final bias. We’ll use backpropagation to find the final bias, starting from 0. Training data: just 3 doses, 0, 0.5 and 1, with effectiveness 0, 1 and 0.

Step 1: How Bad Is the Current Fit?

Network output with final bias zero and residual lines to the three training points

With final bias = 0, the predictions are 0.57, 1.61, 0.58.

Sum of Squared Residuals:

\[ \text{SSR} = \sum_{i=1}^{n} (\text{Observed}_i - \text{Predicted}_i)^2 \]

\[ \begin{aligned} \text{SSR} &= (0 - 0.57)^2 + (1 - 1.61)^2 + (0 - 0.58)^2 \\ &\approx 0.33 + 0.37 + 0.34 = 1.04 \end{aligned} \]

Lower SSR = better fit. We want the final bias that gives the lowest SSR.

Step 2: The Derivative, via the Chain Rule

The prediction depends on the final bias, and the SSR depends on the prediction:

\[ \frac{d\,\text{SSR}}{d\,\text{Final Bias}} = \frac{d\,\text{SSR}}{d\,\text{Predicted}} \times \frac{d\,\text{Predicted}}{d\,\text{Final Bias}} \]

Part 1: SSR with respect to Predicted

\[ \frac{d\,\text{SSR}}{d\,\text{Predicted}} = \sum_{i=1}^{n} -2\,(\text{Observed}_i - \text{Predicted}_i) \]

Part 2: Predicted with respect to the Final Bias

\(\text{Predicted} = \text{blue} + \text{orange} + \text{Final Bias}\)

The blue and orange curves don’t depend on the final bias, so

\[ \frac{d\,\text{Predicted}}{d\,\text{Final Bias}} = 0 + 0 + 1 = 1 \]

Step 3: One Step of Gradient Descent

Putting the two parts together: \(\displaystyle \frac{d\,\text{SSR}}{d\,\text{Final Bias}} = \sum_{i=1}^{n} -2\,(\text{Observed}_i - \text{Predicted}_i) \times 1\)

Plug in the 3 training points (final bias = 0):

\[ \frac{d\,\text{SSR}}{d\,\text{Final Bias}} = -2(0 - 0.57) + -2(1 - 1.61) + -2(0 - 0.58) \approx 1.14 + 1.22 + 1.16 \approx 3.5 \]

Gradient descent rule With learning rate 0.1
Step size = derivative × learning rate \(3.5 \times 0.1 = 0.35\)
New bias = old bias − step size \(0 - 0.35 = -0.35\)
  • The sign of the derivative tells us which way to go (positive slope → go left).
  • The size of the derivative tells us how big a step to take.

Step 4: Repeat Until the Steps Are Tiny

Step Final bias SSR Derivative
0 0.00 1.04 3.54
1 −0.35 0.17 1.41
2 −0.50 0.03 0.57
3 −0.55 0.005 0.23
4 −0.57 0.002 0.09
5 −0.58 0.001 0.04
… ≈ −0.59 ≈ 0.001 ≈ 0

SSR as a function of the final bias with the gradient descent path

The steps get smaller automatically as the slope flattens near the bottom. We land at ≈ −0.58, the value from the start of the lecture! 🎉

From One Parameter to All of Them

In a real network, every weight and bias is updated at the same time:

  1. Forward pass: push the training data through the network to get predictions.
  2. Loss: measure how wrong they are (SSR here; cross-entropy for classification).
  3. Backward pass: use the chain rule to get the derivative of the loss for every parameter, starting at the output and moving back toward the input.
  4. Update: parameter = parameter − learning_rate × derivative
  5. Repeat for many epochs (passes over the training data).

Note

In PyTorch (Lecture 10), step 3 is loss.backward() and step 4 is optimizer.step(). The library does the chain rule for you, but now you know what it’s computing!

Check Your Understanding

  1. The derivative of the SSR at the current final bias is −2.0. With learning rate 0.1, does the bias go up or down? By how much?
  2. Why is \(\dfrac{d\,\text{Predicted}}{d\,\text{Final Bias}} = 1\)?
  3. What could happen if the learning rate were much too large?
  1. Step size \(= -2.0 \times 0.1 = -0.2\), so new bias \(=\) old bias \(- (-0.2)\): it goes up by 0.2.
  2. The final bias is added at the very end, so changing it by 1 changes the prediction by exactly 1.
  3. The steps overshoot the minimum, bouncing back and forth or even diverging. You’ll try this in the notebook!

Part 4: Neural Networks in Practice

A Word of Caution: The Mystery Bump

Fitted network output with a large bump between doses 0 and 0.5

  • We only trained on doses 0, 0.5 and 1.
  • The network is only judged at those points, so in between it can do anything, including a strange bump!
  • With more training data, the fit between points improves.
  • This is why it’s hard to predict what a network will do in new situations (e.g., self-driving cars).

Neural Networks vs. Logistic Regression

Neural network Logistic regression
Flexibility Fits almost any shape One S-shaped curve
Choices to make # layers, # nodes, activation, learning rate … (an art) Very few (closer to a science)
Data needed Usually a lot Works with less
Interpretability Hard: a black box Easy: each coefficient has a meaning
Best for Complex patterns: images, text, audio Simple, tabular problems; explaining results

Tip

Start simple. Try logistic regression (or a tree) first, and use a neural network if the simple model isn’t good enough.

Neural Networks in scikit-learn

from sklearn.neural_network import MLPClassifier  # multi-layer perceptron
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

nn = make_pipeline(
    StandardScaler(),                     # networks train better on scaled data
    MLPClassifier(hidden_layer_sizes=(10, 10),   # 2 layers x 10 nodes
                  activation="relu",
                  learning_rate_init=0.01,
                  max_iter=2000,
                  random_state=0)
)
nn.fit(X_train, y_train)
print("Test accuracy:", nn.score(X_test, y_test))

Key MLPClassifier Settings

Argument Meaning
hidden_layer_sizes nodes per hidden layer, e.g. (2,) = 1 layer with 2 nodes, like today’s example
activation "relu", "logistic" (sigmoid), "tanh"
learning_rate_init the gradient-descent learning rate
max_iter maximum number of epochs

Use MLPRegressor for numeric targets (like effectiveness).

Wrap-up

Key Takeaways

  • A neural network is a squiggle-fitting machine made of nodes, weights (×), biases (+) and activation functions.
  • A forward pass pushes inputs through the network: transform → activate → scale → add the pieces.
  • Activation functions provide the bends. Without them, a network is just a straight line.
  • Training = choose parameters that minimize a loss such as the SSR.
  • Backpropagation = the chain rule (derivatives from back to front) + gradient descent.
  • Networks are flexible but need data, tuning and care. Start with simpler models.

Class Practice 💻

Open the practice notebook and work through it with a partner:

📓 Download nn_practice.ipynb

You will:

  1. Code ReLU, SoftPlus and Sigmoid yourself.
  2. Write a forward pass for today’s network and recreate the blue, orange and green curves.
  3. Train the final bias with gradient descent, and see what a bad learning rate does.
  4. Challenge: train a second parameter using the chain rule.
  5. Use scikit-learn’s MLPClassifier on curved data, and compare it with logistic regression.