How the Perceptron Algorithm Is Implemented from Scratch in Lesson 3

The Perceptron is implemented in lessons/3-NeuralNetworks/03-Perceptron/Perceptron.ipynb using pure NumPy with stochastic weight updates on misclassified examples.

The AI-For-Beginners curriculum by Microsoft provides a hands-on introduction to neural networks through progressive lessons. Lesson 3 focuses on the Perceptron, the foundational binary classifier that laid the groundwork for modern deep learning. This implementation strips away framework abstractions to reveal how the algorithm learns through iterative error correction.

Locating the Implementation

The complete Perceptron implementation resides in the notebook lessons/3-NeuralNetworks/03-Perceptron/Perceptron.ipynb. The core training logic appears in the train function spanning lines 35–68 of that notebook. This notebook demonstrates how to build a functioning neural network component using only NumPy and Python's standard library.

Core Architecture: The train Function

The train function implements the classic Perceptron learning rule through four distinct stages. Understanding each stage clarifies how the algorithm converges to a decision boundary.

Weight Initialization with Bias Term

The implementation begins by allocating a zero-initialized weight matrix with shape (num_dims, 1). The num_dims equals the number of input features plus one additional dimension for the bias term:

weights = np.zeros((num_dims, 1))

This bias dimension allows the decision boundary to offset from the origin, critical for learning non-centered classification patterns.

Stochastic Example Selection

Rather than batch processing, the algorithm uses stochastic updates—randomly sampling one positive and one negative example per iteration:

pos = random.choice(positive_examples)
neg = random.choice(negative_examples)

This approach introduces beneficial noise that helps escape local minima and mirrors the original 1958 Rosenblatt formulation.

Prediction and Misclassification Detection

Predictions are computed via the dot product between input vectors and current weights:

z = np.dot(pos, weights)

A positive example is misclassified when z < 0 (predicted negative), triggering a weight update to correct the error. The symmetric check applies to negative examples.

Weight Update Rule

When misclassification occurs, weights are adjusted by adding or subtracting the misclassified example vector—multiplying by the learning rate. This additive correction nudges the decision boundary toward proper classification.

Why This Implementation Matters

Studying this scratch implementation reveals principles obscured by high-level frameworks:

• Linear separability requirement — The algorithm only converges when classes are linearly separable
• Role of the learning rate — Controls update magnitude and convergence stability
• Order-dependence — Stochastic sampling makes training non-deterministic across runs

The notebook includes visualization code showing the decision boundary's evolution, reinforcing theoretical concepts with empirical observation.

Comparison to Modern Approaches

Aspect	This Perceptron	Modern Neural Networks
Activation	Heaviside step function	Differentiable functions (ReLU, sigmoid, softmax)
Optimization	Stochastic per-example updates	Mini-batch gradient descent with backpropagation
Output	Binary classification	Multi-class, regression, generative tasks
Layers	Single layer	Deep architectures with hundreds of layers

Despite these limitations, the Perceptron's update rule prefigures gradient descent: both adjust parameters proportional to prediction error.

Running the Implementation

Execute the notebook sequentially to observe training dynamics:

cd lessons/3-NeuralNetworks/03-Perceptron
jupyter notebook Perceptron.ipynb

The notebook generates synthetic linearly-separable data, trains the Perceptron, and animates the separating hyperplane's convergence.

Summary

• Location: lessons/3-NeuralNetworks/03-Perceptron/Perceptron.ipynb — lines 35–68 contain the train function
• Initialization: Zero weights with bias dimension via np.zeros((num_dims, 1))
• Training: Stochastic selection of one positive and one negative example per iteration
• Update: Additive correction on misclassification using dot-product predictions
• Dependencies: Pure NumPy implementation with no deep learning frameworks required

Frequently Asked Questions

What makes this Perceptron implementation "from scratch"?

The implementation uses only NumPy arrays and Python standard library functions—no TensorFlow, PyTorch, or scikit-learn abstractions. Every operation from matrix multiplication to random sampling is explicitly coded, making the learning mechanics transparent.

How does this differ from scikit-learn's Perceptron?

Scikit-learn's Perceptron class (in sklearn.linear_model) adds production features: partial fit support, regularization, early stopping, and optimized Cython backends. The Microsoft implementation prioritizes educational clarity over computational efficiency.

Why does the algorithm randomly select examples instead of using full batches?

Stochastic updates were part of Frank Rosenblatt's original 1958 design. Random sampling approximates the full gradient while introducing noise that helps escape oscillation near the decision boundary—particularly important given the Perceptron's discrete error signal.

What happens if the data isn't linearly separable?

The standard Perceptron will not converge on non-separable data. The implementation in Perceptron.ipynb demonstrates this limitation: weights continue oscillating indefinitely without finding a stable solution. This motivated the development of the Pocket algorithm (keeping the best-seen weights) and eventually soft-margin SVMs with hinge loss.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}