# How to Implement Perceptron and Multi-Layer Perceptron from Scratch > Implement perceptron and multi-layer perceptron from scratch using NumPy. Learn to define weight matrices, apply activation functions, and use learning rules or back-propagation for neural networks. - Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners) - Tags: how-to-guide - Published: 2026-08-06 --- **You can implement perceptron and multi-layer perceptron from scratch using only NumPy by defining weight matrices and bias vectors, applying activation functions, and implementing the perceptron learning rule for single-layer networks or back-propagation with gradient descent for multi-layer architectures.** The Microsoft AI-For-Beginners curriculum provides hands-on Jupyter notebooks that teach neural network fundamentals without relying on high-level deep learning libraries. By working through the official lessons on perceptrons and multi-layer perceptrons, you will implement these algorithms using pure NumPy to understand exactly how weights evolve and networks learn from data. ## Single-Layer Perceptron Implementation The single-layer perceptron serves as the foundation of neural networks. According to the AI-For-Beginners source code, the lesson at `lessons/3-NeuralNetworks/03-Perceptron/Perceptron.ipynb` demonstrates a minimal implementation that defines a **weight vector** `w` and **bias** `b`, then applies the step activation function `y = sign(w·x + b)` to classify inputs. ### The Perceptron Learning Rule The training algorithm updates parameters using the classic perceptron learning rule: - **Weight update**: `w ← w + η·(t – y)·x` - **Bias update**: `b ← b + η·(t – y)` Where `η` represents the learning rate, `t` is the target label (mapped to {-1, 1}), and `y` is the predicted output. This rule only triggers on misclassifications, gradually adjusting the decision boundary until the data is linearly separable. ### Binary Classification Code Example The following snippet extracted from `Perceptron.ipynb` demonstrates the complete implementation on a synthetic dataset generated with `make_classification`: ```python import numpy as np from sklearn.datasets import make_classification # Create a toy binary dataset X, Y = make_classification(n_samples=200, n_features=2, n_informative=2, n_redundant=0, flip_y=0.1, random_state=0) Y = np.where(Y == 0, -1, 1) # map {0,1} → {-1,1} # Initialise weights and bias w = np.random.randn(2) b = 0.0 lr = 0.01 # learning rate def perceptron_predict(x): return np.sign(np.dot(w, x) + b) # Training loop for epoch in range(100): for xi, ti in zip(X, Y): yi = perceptron_predict(xi) if yi != ti: # mis‑classification → update w += lr * ti * xi b += lr * ti # optional: monitor progress if epoch % 20 == 0: acc = np.mean([perceptron_predict(xi) == ti for xi, ti in zip(X, Y)]) print(f'Epoch {epoch:02d} – accuracy {acc:.2f}') ``` ## Multi-Layer Perceptron from Scratch While single-layer perceptrons solve linearly separable problems, **multi-layer perceptrons (MLPs)** handle non-linear classification by stacking hidden layers. The `OwnFramework.ipynb` notebook at `lessons/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb` builds a tiny neural-network framework from the ground up using only NumPy. ### Framework Architecture The custom implementation defines several core components: 1. **Layer class** – Stores weights and bias matrices, implementing a forward method computing `z = X·W + b` 2. **Activation functions** – Simple Python functions for **ReLU** and **sigmoid** transformations 3. **Loss functions** – Mean-squared error (MSE) and cross-entropy with their respective gradients 4. **Back-propagation** – Propagates error gradients backward through each layer to compute `∂L/∂W` ### Forward Pass and Activations In the forward pass, input data flows through each hidden layer where the linear transformation `z = X·W + b` is followed by a non-linear activation. For hidden layers, **ReLU** (`np.maximum(0, z)`) introduces non-linearity, while the output layer typically uses **softmax** for multi-class classification. ### Back-Propagation and SGD Updates The backward pass calculates gradients using the chain rule. For a network using softmax output with cross-entropy loss, the gradient at the output layer simplifies to `predicted_probability - true_label`. Parameters update via stochastic gradient descent: ```python W ← W – η·∂L/∂W b ← b – η·∂L/∂b ``` ### Complete MLP Training Example This self-contained implementation from `OwnFramework.ipynb` demonstrates a two-hidden-layer MLP solving a three-class classification problem: ```python import numpy as np from sklearn.datasets import make_classification from sklearn.preprocessing import OneHotEncoder # ------------------------------------------------- # Utility functions # ------------------------------------------------- def relu(z): return np.maximum(0, z) def relu_grad(z): return (z > 0).astype(float) def softmax(z): exp_z = np.exp(z - np.max(z, axis=1, keepdims=True)) return exp_z / exp_z.sum(axis=1, keepdims=True) def cross_entropy(pred, target): # pred – probability matrix, target – one‑hot eps = 1e-12 return -np.mean(np.sum(target * np.log(pred + eps), axis=1)) # ------------------------------------------------- # Simple MLP class # ------------------------------------------------- class MLP: def __init__(self, input_sz, hidden_sizes, output_sz): layer_dims = [input_sz] + hidden_sizes + [output_sz] self.weights = [np.random.randn(layer_dims[i], layer_dims[i+1]) * 0.1 for i in range(len(layer_dims)-1)] self.biases = [np.zeros((1, dim)) for dim in layer_dims[1:]] def forward(self, X): self.activations = [X] # store for back‑prop a = X for W, b in zip(self.weights[:-1], self.biases[:-1]): z = a @ W + b a = relu(z) self.activations.append(a) # output layer z = a @ self.weights[-1] + self.biases[-1] self.output = softmax(z) return self.output def backward(self, Y_true, lr=0.01): # gradient of loss w.r.t. output (softmax + CE) grad = self.output - Y_true # shape (N, C) for i in reversed(range(len(self.weights))): a_prev = self.activations[i] W = self.weights[i] # compute gradients dW = a_prev.T @ grad / a_prev.shape[0] db = np.mean(grad, axis=0, keepdims=True) # propagate to previous layer (except for input layer) if i > 0: grad = grad @ W.T * relu_grad(a_prev) # SGD update self.weights[i] -= lr * dW self.biases[i] -= lr * db # ------------------------------------------------- # Data preparation # ------------------------------------------------- X, y = make_classification(n_samples=500, n_features=2, n_informative=2, n_redundant=0, n_classes=3, n_clusters_per_class=1, random_state=0) enc = OneHotEncoder(sparse=False) Y_onehot = enc.fit_transform(y.reshape(-1, 1)) # ------------------------------------------------- # Training # ------------------------------------------------- net = MLP(input_sz=2, hidden_sizes=[8, 8], output_sz=3) for epoch in range(500): probs = net.forward(X) loss = cross_entropy(probs, Y_onehot) net.backward(Y_onehot, lr=0.05) if epoch % 100 == 0: pred = np.argmax(probs, axis=1) acc = np.mean(pred == y) print(f'Epoch {epoch:03d} – loss {loss:.4f} – acc {acc:.2f}') ``` ## Key Source Files and Notebooks The Microsoft AI-For-Beginners repository organizes these implementations into specific lesson directories: - **`lessons/3-NeuralNetworks/03-Perceptron/Perceptron.ipynb`** – Implements the classic perceptron learning rule, visualises decision boundaries, and discusses the XOR limitation. - **`lessons/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb`** – Builds the complete neural-network framework with layers, activations, loss functions, and back-propagation. - **`lessons/3-NeuralNetworks/03-Perceptron/lab/PerceptronMultiClass.ipynb`** – Extends the single-layer perceptron to handle multi-class problems using one-vs-rest strategy. - **[`lessons/3-NeuralNetworks/03-Perceptron/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/3-NeuralNetworks/03-Perceptron/README.md)** and **[`04-OwnFramework/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/04-OwnFramework/README.md)** – Provide theoretical background and exercise suggestions. ## Summary - **Single-layer perceptrons** use the perceptron learning rule (`w ← w + η·(t – y)·x`) to find linear decision boundaries, implemented in `Perceptron.ipynb` using only NumPy and scikit-learn for data generation. - **Multi-layer perceptrons** require a custom framework with Layer classes, forward passes computing `z = X·W + b`, and back-propagation to update weights via stochastic gradient descent, as demonstrated in `OwnFramework.ipynb`. - Both implementations rely exclusively on **NumPy** for array operations and **matplotlib** for visualization, making the underlying mathematics transparent and easy to modify. - The AI-For-Beginners curriculum specifically maps labels to {-1, 1} for binary perceptron training and uses one-hot encoding with softmax for multi-class MLP classification. ## Frequently Asked Questions ### What is the difference between a perceptron and a multi-layer perceptron? A **perceptron** is a single-layer linear classifier that uses a step function and can only solve linearly separable problems. A **multi-layer perceptron (MLP)** stacks multiple hidden layers with non-linear activation functions between the input and output, enabling it to learn complex, non-linear decision boundaries required for problems like XOR classification. ### Why does the perceptron learning rule only update weights on misclassification? The perceptron learning rule follows the principle of **error-correction learning**. When the prediction `y` matches the target `t`, the term `(t - y)` equals zero, resulting in no update. Only misclassified samples produce a non-zero error signal that adjusts the weight vector `w` and bias `b` to move the decision boundary toward the misclassified point. ### How does back-propagation work in the custom MLP framework? In the `MLP` class implemented in `OwnFramework.ipynb`, back-propagation calculates gradients by starting at the output layer where the gradient is `self.output - Y_true` for softmax with cross-entropy. The algorithm then propagates this error backward through each layer using the chain rule, multiplying by the derivative of the activation function (`relu_grad`) and the transpose of the weight matrix, finally updating parameters with `W -= lr * dW`. ### Can these implementations handle the XOR problem? The **single-layer perceptron cannot solve XOR** because XOR is not linearly separable, as explicitly discussed in the `Perceptron.ipynb` notebook. However, the **multi-layer perceptron** implemented in `OwnFramework.ipynb` with two hidden layers successfully learns XOR and other non-linear classification tasks by combining multiple linear decision boundaries through hidden layer representations.