# How the Perceptron Algorithm Is Implemented from Scratch in Lesson 3

> Learn how the Perceptron algorithm is implemented from scratch in lesson 3 using pure NumPy with stochastic weight updates on misclassified examples. Understand core neural network concepts.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: how-to-guide
- Published: 2026-08-18

---

**The Perceptron is implemented in `lessons/3-NeuralNetworks/03-Perceptron/Perceptron.ipynb` using pure NumPy with stochastic weight updates on misclassified examples.**

The AI-For-Beginners curriculum by Microsoft provides a hands-on introduction to neural networks through progressive lessons. Lesson 3 focuses on the **Perceptron**, the foundational binary classifier that laid the groundwork for modern deep learning. This implementation strips away framework abstractions to reveal how the algorithm learns through iterative error correction.

## Locating the Implementation

The complete Perceptron implementation resides in the notebook **`lessons/3-NeuralNetworks/03-Perceptron/Perceptron.ipynb`**. The core training logic appears in the `train` function spanning **lines 35–68** of that notebook. This notebook demonstrates how to build a functioning neural network component using only **NumPy** and Python's standard library.

## Core Architecture: The `train` Function

The `train` function implements the classic **Perceptron learning rule** through four distinct stages. Understanding each stage clarifies how the algorithm converges to a decision boundary.

### Weight Initialization with Bias Term

The implementation begins by allocating a zero-initialized weight matrix with shape `(num_dims, 1)`. The `num_dims` equals the number of input features plus one additional dimension for the **bias term**:

```python
weights = np.zeros((num_dims, 1))

```

This bias dimension allows the decision boundary to offset from the origin, critical for learning non-centered classification patterns.

### Stochastic Example Selection

Rather than batch processing, the algorithm uses **stochastic updates**—randomly sampling one positive and one negative example per iteration:

```python
pos = random.choice(positive_examples)
neg = random.choice(negative_examples)

```

This approach introduces beneficial noise that helps escape local minima and mirrors the original 1958 Rosenblatt formulation.

### Prediction and Misclassification Detection

Predictions are computed via the dot product between input vectors and current weights:

```python
z = np.dot(pos, weights)

```

A positive example is **misclassified** when `z < 0` (predicted negative), triggering a weight update to correct the error. The symmetric check applies to negative examples.

### Weight Update Rule

When misclassification occurs, weights are adjusted by adding or subtracting the misclassified example vector—multiplying by the learning rate. This **additive correction** nudges the decision boundary toward proper classification.

## Why This Implementation Matters

Studying this scratch implementation reveals principles obscured by high-level frameworks:

- **Linear separability requirement** — The algorithm only converges when classes are linearly separable
- **Role of the learning rate** — Controls update magnitude and convergence stability
- **Order-dependence** — Stochastic sampling makes training non-deterministic across runs

The notebook includes visualization code showing the decision boundary's evolution, reinforcing theoretical concepts with empirical observation.

## Comparison to Modern Approaches

| Aspect | This Perceptron | Modern Neural Networks |
|--------|---------------|------------------------|
| Activation | Heaviside step function | Differentiable functions (ReLU, sigmoid, softmax) |
| Optimization | Stochastic per-example updates | Mini-batch gradient descent with backpropagation |
| Output | Binary classification | Multi-class, regression, generative tasks |
| Layers | Single layer | Deep architectures with hundreds of layers |

Despite these limitations, the Perceptron's update rule prefigures **gradient descent**: both adjust parameters proportional to prediction error.

## Running the Implementation

Execute the notebook sequentially to observe training dynamics:

```bash
cd lessons/3-NeuralNetworks/03-Perceptron
jupyter notebook Perceptron.ipynb

```

The notebook generates synthetic linearly-separable data, trains the Perceptron, and animates the separating hyperplane's convergence.

## Summary

- **Location**: `lessons/3-NeuralNetworks/03-Perceptron/Perceptron.ipynb` — lines 35–68 contain the `train` function
- **Initialization**: Zero weights with bias dimension via `np.zeros((num_dims, 1))`
- **Training**: Stochastic selection of one positive and one negative example per iteration
- **Update**: Additive correction on misclassification using dot-product predictions
- **Dependencies**: Pure NumPy implementation with no deep learning frameworks required

## Frequently Asked Questions

### What makes this Perceptron implementation "from scratch"?

The implementation uses **only NumPy arrays and Python standard library functions**—no TensorFlow, PyTorch, or scikit-learn abstractions. Every operation from matrix multiplication to random sampling is explicitly coded, making the learning mechanics transparent.

### How does this differ from scikit-learn's Perceptron?

Scikit-learn's `Perceptron` class (in `sklearn.linear_model`) adds production features: partial fit support, regularization, early stopping, and optimized Cython backends. The Microsoft implementation prioritizes **educational clarity** over computational efficiency.

### Why does the algorithm randomly select examples instead of using full batches?

**Stochastic updates** were part of Frank Rosenblatt's original 1958 design. Random sampling approximates the full gradient while introducing noise that helps escape oscillation near the decision boundary—particularly important given the Perceptron's discrete error signal.

### What happens if the data isn't linearly separable?

The standard Perceptron **will not converge** on non-separable data. The implementation in `Perceptron.ipynb` demonstrates this limitation: weights continue oscillating indefinitely without finding a stable solution. This motivated the development of the **Pocket algorithm** (keeping the best-seen weights) and eventually **soft-margin SVMs** with hinge loss.