How to Implement Perceptron and Multi-Layer Perceptron from Scratch

You can implement perceptron and multi-layer perceptron from scratch using only NumPy by defining weight matrices and bias vectors, applying activation functions, and implementing the perceptron learning rule for single-layer networks or back-propagation with gradient descent for multi-layer architectures.

The Microsoft AI-For-Beginners curriculum provides hands-on Jupyter notebooks that teach neural network fundamentals without relying on high-level deep learning libraries. By working through the official lessons on perceptrons and multi-layer perceptrons, you will implement these algorithms using pure NumPy to understand exactly how weights evolve and networks learn from data.

Single-Layer Perceptron Implementation

The single-layer perceptron serves as the foundation of neural networks. According to the AI-For-Beginners source code, the lesson at lessons/3-NeuralNetworks/03-Perceptron/Perceptron.ipynb demonstrates a minimal implementation that defines a weight vector w and bias b, then applies the step activation function y = sign(w·x + b) to classify inputs.

The Perceptron Learning Rule

The training algorithm updates parameters using the classic perceptron learning rule:

• Weight update: w ← w + η·(t – y)·x
• Bias update: b ← b + η·(t – y)

Where η represents the learning rate, t is the target label (mapped to {-1, 1}), and y is the predicted output. This rule only triggers on misclassifications, gradually adjusting the decision boundary until the data is linearly separable.

Binary Classification Code Example

The following snippet extracted from Perceptron.ipynb demonstrates the complete implementation on a synthetic dataset generated with make_classification:

import numpy as np
from sklearn.datasets import make_classification

# Create a toy binary dataset

X, Y = make_classification(n_samples=200, n_features=2,
                           n_informative=2, n_redundant=0,
                           flip_y=0.1, random_state=0)
Y = np.where(Y == 0, -1, 1)           # map {0,1} → {-1,1}

# Initialise weights and bias

w = np.random.randn(2)
b = 0.0
lr = 0.01                             # learning rate

def perceptron_predict(x):
    return np.sign(np.dot(w, x) + b)

# Training loop

for epoch in range(100):
    for xi, ti in zip(X, Y):
        yi = perceptron_predict(xi)
        if yi != ti:                  # mis‑classification → update

            w += lr * ti * xi
            b += lr * ti
    # optional: monitor progress

    if epoch % 20 == 0:
        acc = np.mean([perceptron_predict(xi) == ti for xi, ti in zip(X, Y)])
        print(f'Epoch {epoch:02d} – accuracy {acc:.2f}')

Multi-Layer Perceptron from Scratch

While single-layer perceptrons solve linearly separable problems, multi-layer perceptrons (MLPs) handle non-linear classification by stacking hidden layers. The OwnFramework.ipynb notebook at lessons/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb builds a tiny neural-network framework from the ground up using only NumPy.

Framework Architecture

The custom implementation defines several core components:

1. Layer class – Stores weights and bias matrices, implementing a forward method computing z = X·W + b
2. Activation functions – Simple Python functions for ReLU and sigmoid transformations
3. Loss functions – Mean-squared error (MSE) and cross-entropy with their respective gradients
4. Back-propagation – Propagates error gradients backward through each layer to compute ∂L/∂W

Forward Pass and Activations

In the forward pass, input data flows through each hidden layer where the linear transformation z = X·W + b is followed by a non-linear activation. For hidden layers, ReLU (np.maximum(0, z)) introduces non-linearity, while the output layer typically uses softmax for multi-class classification.

Back-Propagation and SGD Updates

The backward pass calculates gradients using the chain rule. For a network using softmax output with cross-entropy loss, the gradient at the output layer simplifies to predicted_probability - true_label. Parameters update via stochastic gradient descent:

W ← W – η·∂L/∂W
b ← b – η·∂L/∂b

Complete MLP Training Example

This self-contained implementation from OwnFramework.ipynb demonstrates a two-hidden-layer MLP solving a three-class classification problem:

import numpy as np
from sklearn.datasets import make_classification
from sklearn.preprocessing import OneHotEncoder

# -------------------------------------------------

# Utility functions

# -------------------------------------------------

def relu(z): return np.maximum(0, z)
def relu_grad(z): return (z > 0).astype(float)

def softmax(z):
    exp_z = np.exp(z - np.max(z, axis=1, keepdims=True))
    return exp_z / exp_z.sum(axis=1, keepdims=True)

def cross_entropy(pred, target):
    # pred – probability matrix, target – one‑hot

    eps = 1e-12
    return -np.mean(np.sum(target * np.log(pred + eps), axis=1))

# -------------------------------------------------

# Simple MLP class

# -------------------------------------------------

class MLP:
    def __init__(self, input_sz, hidden_sizes, output_sz):
        layer_dims = [input_sz] + hidden_sizes + [output_sz]
        self.weights = [np.random.randn(layer_dims[i], layer_dims[i+1]) * 0.1
                        for i in range(len(layer_dims)-1)]
        self.biases  = [np.zeros((1, dim)) for dim in layer_dims[1:]]

    def forward(self, X):
        self.activations = [X]                     # store for back‑prop

        a = X
        for W, b in zip(self.weights[:-1], self.biases[:-1]):
            z = a @ W + b
            a = relu(z)
            self.activations.append(a)
        # output layer

        z = a @ self.weights[-1] + self.biases[-1]
        self.output = softmax(z)
        return self.output

    def backward(self, Y_true, lr=0.01):
        # gradient of loss w.r.t. output (softmax + CE)

        grad = self.output - Y_true                # shape (N, C)

        for i in reversed(range(len(self.weights))):
            a_prev = self.activations[i]
            W = self.weights[i]
            # compute gradients

            dW = a_prev.T @ grad / a_prev.shape[0]
            db = np.mean(grad, axis=0, keepdims=True)
            # propagate to previous layer (except for input layer)

            if i > 0:
                grad = grad @ W.T * relu_grad(a_prev)
            # SGD update

            self.weights[i] -= lr * dW
            self.biases[i]  -= lr * db

# -------------------------------------------------

# Data preparation

# -------------------------------------------------

X, y = make_classification(n_samples=500, n_features=2,
                           n_informative=2, n_redundant=0,
                           n_classes=3, n_clusters_per_class=1,
                           random_state=0)
enc = OneHotEncoder(sparse=False)
Y_onehot = enc.fit_transform(y.reshape(-1, 1))

# -------------------------------------------------

# Training

# -------------------------------------------------

net = MLP(input_sz=2, hidden_sizes=[8, 8], output_sz=3)
for epoch in range(500):
    probs = net.forward(X)
    loss = cross_entropy(probs, Y_onehot)
    net.backward(Y_onehot, lr=0.05)
    if epoch % 100 == 0:
        pred = np.argmax(probs, axis=1)
        acc = np.mean(pred == y)
        print(f'Epoch {epoch:03d} – loss {loss:.4f} – acc {acc:.2f}')

Key Source Files and Notebooks

The Microsoft AI-For-Beginners repository organizes these implementations into specific lesson directories:

• lessons/3-NeuralNetworks/03-Perceptron/Perceptron.ipynb – Implements the classic perceptron learning rule, visualises decision boundaries, and discusses the XOR limitation.
• lessons/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb – Builds the complete neural-network framework with layers, activations, loss functions, and back-propagation.
• lessons/3-NeuralNetworks/03-Perceptron/lab/PerceptronMultiClass.ipynb – Extends the single-layer perceptron to handle multi-class problems using one-vs-rest strategy.
• lessons/3-NeuralNetworks/03-Perceptron/README.md and 04-OwnFramework/README.md – Provide theoretical background and exercise suggestions.

Summary

• Single-layer perceptrons use the perceptron learning rule (w ← w + η·(t – y)·x) to find linear decision boundaries, implemented in Perceptron.ipynb using only NumPy and scikit-learn for data generation.
• Multi-layer perceptrons require a custom framework with Layer classes, forward passes computing z = X·W + b, and back-propagation to update weights via stochastic gradient descent, as demonstrated in OwnFramework.ipynb.
• Both implementations rely exclusively on NumPy for array operations and matplotlib for visualization, making the underlying mathematics transparent and easy to modify.
• The AI-For-Beginners curriculum specifically maps labels to {-1, 1} for binary perceptron training and uses one-hot encoding with softmax for multi-class MLP classification.

Frequently Asked Questions

What is the difference between a perceptron and a multi-layer perceptron?

A perceptron is a single-layer linear classifier that uses a step function and can only solve linearly separable problems. A multi-layer perceptron (MLP) stacks multiple hidden layers with non-linear activation functions between the input and output, enabling it to learn complex, non-linear decision boundaries required for problems like XOR classification.

Why does the perceptron learning rule only update weights on misclassification?

The perceptron learning rule follows the principle of error-correction learning. When the prediction y matches the target t, the term (t - y) equals zero, resulting in no update. Only misclassified samples produce a non-zero error signal that adjusts the weight vector w and bias b to move the decision boundary toward the misclassified point.

How does back-propagation work in the custom MLP framework?

In the MLP class implemented in OwnFramework.ipynb, back-propagation calculates gradients by starting at the output layer where the gradient is self.output - Y_true for softmax with cross-entropy. The algorithm then propagates this error backward through each layer using the chain rule, multiplying by the derivative of the activation function (relu_grad) and the transpose of the weight matrix, finally updating parameters with W -= lr * dW.

Can these implementations handle the XOR problem?

The single-layer perceptron cannot solve XOR because XOR is not linearly separable, as explicitly discussed in the Perceptron.ipynb notebook. However, the multi-layer perceptron implemented in OwnFramework.ipynb with two hidden layers successfully learns XOR and other non-linear classification tasks by combining multiple linear decision boundaries through hidden layer representations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}