Quick recap: the perceptron

\[ z = w_1x_1 + w_2x_2 + b \qquad\qquad \hat y = \text{step}(z) = \begin{cases} 1, & z \ge 0\\ 0, & z < 0 \end{cases} \]

A perceptron can do AND and OR

Now try XOR

XOR (“exclusive or”) is 1 when exactly one input is 1.

Picture it

The two blue points sit on opposite corners. Any straight line that separates them from one orange X puts the other orange X on the wrong side.

The idea: break XOR into easy pieces

XOR means “at least one is on, but not both”:

\[ \text{XOR}(x_1,x_2) = \text{OR}(x_1,x_2)\ \text{ AND NOT }\ \text{AND}(x_1,x_2) \]

Yes: “\(h_1\) and not \(h_2\)” is linearly separable

Use \(\;\hat y = \text{step}(h_1 - h_2 - 0.5)\)

Our first multilayer perceptron

\[ h_1 = \text{step}(x_1 + x_2 - 0.5) \qquad h_2 = \text{step}(x_1 + x_2 - 1.5) \qquad \hat y = \text{step}(h_1 - h_2 - 0.5) \]

Vocabulary

• Input layer: just the features. No computation happens here.
• Hidden layer(s): perceptrons whose outputs we never see directly.
• Output layer: produces the prediction \(\hat y\).
• Fully connected: every node feeds every node in the next layer.
• Feed-forward: information flows one way, left → right.

All four inputs

Why it works: the hidden layer re-draws the data

The hidden layer moves the points into a new space \((h_1, h_2)\). There, both XOR = 1 inputs land on the same spot, and one line separates the classes.

Why it works: two lines → a band

Try it: MLP playground

Drag the sliders to solve XOR, then switch the target to XNOR, NAND, …

The same network in matrix form

Stack the hidden weights in a matrix, one row per hidden unit:

\[ W = \begin{bmatrix} w_{11} & w_{12} \\ w_{21} & w_{22} \end{bmatrix} = \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}, \quad \mathbf b = \begin{bmatrix} -0.5 \\ -1.5 \end{bmatrix}, \quad \mathbf v = \begin{bmatrix} 1 \\ -1 \end{bmatrix}, \quad c = -0.5 \]

Counting parameters

Each layer has (inputs × outputs) weights + (outputs) biases.

Problem 1: hidden units have no target

The perceptron rule uses the error at that neuron:

\[ w_i \leftarrow w_i + \eta\,(y - \hat y)\,x_i \]

Problem 2: the step function is flat

Why not just drop the activation?

Without an activation, each layer is linear:

\[ \mathbf h = W_1 \mathbf x + \mathbf b_1, \qquad \hat y = W_2 \mathbf h + \mathbf b_2 \]

Worked example 2: the same net with sigmoids

Input \(\mathbf x = (1, 0)\), target \(y = 1\), loss \(L = \tfrac12 (y - \hat y)^2\), learning rate \(\eta = 1\).

Step 5: did it help?

Run the forward pass again on \(\mathbf x = (1,0)\) with the new parameters:

The training loop: perceptron vs MLP

In code: scikit-learn

import numpy as np
from sklearn.linear_model import Perceptron
from sklearn.neural_network import MLPClassifier

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([0, 1, 1, 0])                      # XOR

p = Perceptron(random_state=0).fit(X, y)
print("Perceptron:", p.predict(X), "accuracy", p.score(X, y))

mlp = MLPClassifier(hidden_layer_sizes=(4,), activation="tanh",
                    solver="lbfgs", max_iter=1000, random_state=0).fit(X, y)
print("MLP:       ", mlp.predict(X), "accuracy", mlp.score(X, y))

Perceptron: [0 0 0 0] accuracy 0.5
MLP:        [0 1 1 0] accuracy 1.0

hidden_layer_sizes=(4,) means one hidden layer with 4 units. (8, 8) would mean two hidden layers.

Practice 1: forward pass

\[ h_1 = \text{step}(2x_1 - x_2 - 1.5) \qquad h_2 = \text{step}(-x_1 + 2x_2 - 1.5) \qquad \hat y = \text{step}(h_1 + h_2 - 0.5) \]

Fill in the table. Which logic function is this?

Practice 1: answer

Practice 2: the hidden space

Use the network from Practice 1.

1. Where does each input land in the \((h_1, h_2)\) plane?
2. Give a line in that plane that separates the classes.

Practice 3: design XNOR

XNOR is the opposite of XOR: 1 when the inputs are equal.

Keep the lecture’s hidden layer (\(h_1\) = OR, \(h_2\) = AND). Choose new output weights \(v_1, v_2, c\).

Practice 4: count the parameters

How many weights and biases does each fully connected network have?

1. 4 → 5 → 3
2. 784 → 128 → 10
3. 10 → 20 → 20 → 1

Practice 5: why do we need activations?

A “network” has no activation functions:

\[ h = 2x + 1, \qquad \hat y = 3h - 4 \]

Write \(\hat y\) directly in terms of \(x\). What does this tell you?

Practice 6: sigmoid forward pass

Use the sigmoid network from worked example 2 (lecture weights, \(c = -0.5\)) with input \(\mathbf x = (0, 0)\), target \(y = 0\).

Find \(h_1\), \(h_2\), \(\hat y\) and \(L = \frac12 (y - \hat y)^2\). Use \(\sigma(-0.5) = 0.378\), \(\sigma(-1.5) = 0.182\), \(\sigma(-0.304) = 0.424\).

Practice 7: one backprop update

Continue Practice 6 (\(\hat y = 0.424\), \(y = 0\), \(\eta = 1\)).

1. Compute \(\delta_{\text{out}} = (\hat y - y)\,\hat y\,(1-\hat y)\).
2. Update \(c\) and \(v_1\) (recall \(h_1 = 0.378\)).
3. Will the new \(\hat y\) be bigger or smaller? Is that good?

Practice 8 (challenge): “exactly one”

Design an MLP with three inputs \(x_1, x_2, x_3 \in \{0,1\}\) that outputs 1 when exactly one input is 1.

Hint: reuse the “OR and not AND” idea with \(s = x_1 + x_2 + x_3\).

Key takeaways

• One perceptron = one straight line, so it cannot learn XOR.
• An MLP chains perceptrons into layers: input → hidden → output.
• The hidden layer transforms the data into a space where a line does work.
• Forward pass: compute each layer from the previous one, \(\mathbf h = f(W\mathbf x + \mathbf b)\).
• Non-linear activations are essential. Without them, layers collapse into one.
• Training uses smooth activations + backpropagation (the chain rule) + gradient descent.

\[ \boxed{\text{Forward pass} \rightarrow \text{Loss} \rightarrow \text{Backward pass} \rightarrow \text{Update} \rightarrow \text{Repeat}} \]