Teaching the Perceptron

Learning objectives

By the end of this lesson, you should be able to:

• Explain how a perceptron makes a prediction.
• Calculate a weighted sum and apply a step function.
• Calculate the prediction error.
• Update weights and bias using the perceptron learning rule.
• Train a perceptron on a small dataset.

What is a perceptron?

A perceptron is a simple artificial neuron used for binary classification.

It receives inputs, multiplies them by weights, adds a bias, and applies an activation function.

\[ z=w_1x_1+w_2x_2+b \]

The step activation function is

\[ \hat{y}= \begin{cases} 1, & z\geq 0,\\ 0, & z<0. \end{cases} \]

• ((x_1, x_2)): Input features
• ((w_1, w_2)): Weights
• (b): Bias
• (): Predicted class

The perceptron at a glance

flowchart LR
    X1["Input x₁"] --> S["Weighted sum"]
    X2["Input x₂"] --> S
    W["Weights"] --> S
    B["Bias"] --> S
    S --> Z["z = w₁x₁ + w₂x₂ + b"]
    Z --> A["Step activation"]
    A --> Y["Prediction ŷ"]

The perceptron first computes a score, then converts it into a class.

The perceptron at a glance

Our training dataset

We will learn the logical AND function.

Question: When should the output be 1?

Initialize the model

Start with these parameters:

\[ w_1=0,\qquad w_2=0,\qquad b=0 \]

Choose the learning rate:

\[ \eta=1 \]

The learning rate determines how much the parameters change after an error.

We will process the samples in the order shown in the dataset.

The training algorithm

For each training example:

1. Calculate the weighted sum.
2. Apply the step function.
3. Calculate the error.
4. Update the weights and bias if needed.

The training algorithm

The error is

\[ e=y-\hat{y} \]

The weight update rule is

\[ w_i^{\text{new}}=w_i^{\text{old}}+\eta e x_i \]

The bias update rule is

\[ b^{\text{new}}=b^{\text{old}}+\eta e \]

Epoch 1: Sample 1

Input:

\[ x_1=0,\quad x_2=0,\quad y=0 \]

Step 1: Weighted sum

\[ z=(0)(0)+(0)(0)+0=0 \]

Step 2: Prediction

Since (z),

\[ \hat{y}=1 \]

Step 3: Error

\[ e=y-\hat{y}=0-1=-1 \]

The prediction is incorrect.

Epoch 1: Update after Sample 1

Update the first weight:

\[ w_1=0+(1)(-1)(0)=0 \]

Update the second weight:

\[ w_2=0+(1)(-1)(0)=0 \]

Update the bias:

\[ b=0+(1)(-1)=-1 \]

The new parameters are

\[ \boxed{w_1=0,\quad w_2=0,\quad b=-1} \]

Teaching point: The weights do not change because both input features are zero. Only the bias changes.

Epoch 1: Sample 2

Input:

\[ x_1=0,\quad x_2=1,\quad y=0 \]

Using the updated parameters:

\[ z=(0)(0)+(0)(1)-1=-1 \]

Since (z<0),

\[ \hat{y}=0 \]

The error is

\[ e=0-0=0 \]

The prediction is correct, so no update is required.

Parameters remain

\[ w_1=0,\quad w_2=0,\quad b=-1 \]

Epoch 1: Sample 3

Input:

\[ x_1=1,\quad x_2=0,\quad y=0 \]

Calculate the weighted sum:

\[ z=(0)(1)+(0)(0)-1=-1 \]

The prediction is

\[ \hat{y}=0 \]

The error is

\[ e=0-0=0 \]

The prediction is correct. The parameters remain unchanged.

Epoch 1: Sample 4

Input:

\[ x_1=1,\quad x_2=1,\quad y=1 \]

Calculate the weighted sum:

\[ z=(0)(1)+(0)(1)-1=-1 \]

The prediction is

\[ \hat{y}=0 \]

The error is

\[ e=1-0=1 \]

The prediction is incorrect, so update the parameters.

Epoch 1: Update after Sample 4

Update the first weight:

\[ w_1=0+(1)(1)(1)=1 \]

Update the second weight:

\[ w_2=0+(1)(1)(1)=1 \]

Update the bias:

\[ b=-1+(1)(1)=0 \]

The parameters after Epoch 1 are

\[ \boxed{w_1=1,\quad w_2=1,\quad b=0} \]

The perceptron has changed its decision boundary, but training is not finished.

Epoch 1: Summary

Parameters at the end of Epoch 1:

\[ w_1=1,\quad w_2=1,\quad b=0 \]

There were two incorrect predictions and two parameter updates.

Epoch 2: Sample 1

Start with the parameters from Epoch 1.

Input:

\[ x_1=0,\quad x_2=0,\quad y=0 \]

Weighted sum:

\[ z=(1)(0)+(1)(0)+0=0 \]

Prediction:

\[ \hat{y}=1 \]

Error:

\[ e=0-1=-1 \]

Update the parameters:

\[ w_1=1,\qquad w_2=1,\qquad b=-1 \]

Epoch 2: Sample 2

Input:

\[ x_1=0,\quad x_2=1,\quad y=0 \]

Weighted sum:

\[ z=(1)(0)+(1)(1)-1=0 \]

Prediction:

\[ \hat{y}=1 \]

Error:

\[ e=0-1=-1 \]

Update the parameters:

\[ w_1=1 \]

\[ w_2=1+(1)(-1)(1)=0 \]

\[ b=-1+(1)(-1)=-2 \]

The new parameters are

\[ \boxed{w_1=1,\quad w_2=0,\quad b=-2} \]

Epoch 2: Sample 3

Input:

\[ x_1=1,\quad x_2=0,\quad y=0 \]

Weighted sum:

\[ z=(1)(1)+(0)(0)-2=-1 \]

Prediction:

\[ \hat{y}=0 \]

Error:

\[ e=0-0=0 \]

The prediction is correct. No update is required.

Epoch 2: Sample 4

Input:

\[ x_1=1,\quad x_2=1,\quad y=1 \]

Weighted sum:

\[ z=(1)(1)+(0)(1)-2=-1 \]

Prediction:

\[ \hat{y}=0 \]

Error:

\[ e=1-0=1 \]

Update the parameters:

\[ w_1=1+(1)(1)(1)=2 \]

\[ w_2=0+(1)(1)(1)=1 \]

\[ b=-2+(1)(1)=-1 \]

The parameters after Epoch 2 are

\[ \boxed{w_1=2,\quad w_2=1,\quad b=-1} \]

Epoch 2: Summary

At the end of Epoch 2:

\[ w_1=2,\quad w_2=1,\quad b=-1 \]

The perceptron has not yet converged.

Why does the perceptron need more epochs?

The perceptron predicts class 1 whenever

\[ w_1x_1+w_2x_2+b\geq 0 \]

With the current parameters, the prediction for the input ((1,1)) is still incorrect.

Training must continue until a complete epoch has no classification errors.

The AND dataset is linearly separable, so the classical perceptron algorithm can find a separating solution with continued training under the standard assumptions.

Practice 1: Forward pass

Suppose

\[ w_1=0.5,\quad w_2=-0.2,\quad b=-0.1 \]

and the input is

\[ x_1=1,\quad x_2=2 \]

Calculate:

1. The weighted sum (z).
2. The predicted class ().

Practice 1: Forward pass

Practice 2: Weight update

Suppose

\[ w_1=0.2,\quad w_2=0.3,\quad b=-0.4 \]

The input is ((1,1)), the target is (y=1), and the learning rate is (eta=0.1).

Calculate the prediction, error, and updated parameters.

Practice 2: Weight update

Practice 3: Train on OR

Use this dataset:

Initialize:

\[ w_1=0,\quad w_2=0,\quad b=0,\quad \eta=1 \]

Process the first two samples in order. Calculate the weighted sum, prediction, error, and updated parameters after each sample.

Practice 3: Train on OR

Key takeaways

• A perceptron calculates a weighted sum and applies a step activation.
• The prediction error is (e=y-).
• Weights and bias are updated when the prediction is incorrect.
• One epoch means processing every training example once.
• Training continues until the perceptron classifies all training examples correctly, assuming the dataset is linearly separable.

\[ \boxed{ \text{Forward pass} \rightarrow \text{Error} \rightarrow \text{Update} \rightarrow \text{Repeat} } \]