How to Build a Custom Neural Network Framework from Scratch

You can build a custom neural network framework from scratch by implementing modular layer classes with forward() and backward() methods, composing them into a sequential model container, and manually executing gradient descent to update parameters without relying on high-level libraries like TensorFlow or PyTorch.

The microsoft/AI-For-Beginners repository contains pedagogical notebooks and scripts that demonstrate how to build a custom neural network framework from scratch using pure Python and NumPy. By studying these reference implementations—including Perceptron.ipynb and 02-simple-neural-network.py—you can understand the fundamental mechanics of forward propagation, back-propagation, and loss computation while maintaining full control over every computational step.

Core Architecture Components

A custom neural network framework requires five essential components: a base layer abstraction, concrete layer implementations, a model container, a loss function, and an optimizer. The following sections map each component to specific files in the microsoft/AI-For-Beginners repository.

Base Layer Class

Every neural network layer must store parameters and cache intermediate values for back-propagation. In lessons/3-NeuralNetworks/03-Perceptron/Perceptron.ipynb, the repository demonstrates this pattern with a Neuron class that implements forward and backward methods. Your base class should define these methods as interfaces, ensuring every concrete layer can participate in the computational graph.

Linear and Activation Layers

The examples/02-simple-neural-network.py file provides a reference implementation for a dense layer using NumPy matrix operations. A Linear layer initializes weights using Xavier initialization and computes the affine transformation x @ W + b during the forward pass. During back-propagation, it calculates gradients with respect to weights (dW), biases (db), and inputs using the cached input from the forward pass.

Activation functions like ReLU and Sigmoid follow the same interface. They store masks or intermediate values during forward and apply element-wise derivatives during backward. The repository's simple neural network example demonstrates how to chain these layers sequentially.

Model Container

To execute layers in sequence, you need a Model class that iterates through a list of layers. The lessons/4-ComputerVision/07-ConvNets/pytorchcv.py file illustrates this pattern using PyTorch's nn.Sequential, but the logic applies equally to custom frameworks. Your container should call forward() on each layer in order during prediction, then reverse the list to call backward() during training, passing gradients from the loss function back to the input layer.

Loss Computation and Optimization

Separating loss calculation from parameter updates allows you to swap components without modifying the training loop.

Implementing Loss Functions

The examples/04-text-sentiment.py script demonstrates categorical cross-entropy loss computation for classification tasks. For regression, you can implement mean squared error (MSE) as shown in the repository's introductory examples. A loss function must return two values: the scalar loss for monitoring and the gradient of the loss with respect to the model's output (dL/dout) to seed the backward pass.

Manual Gradient Descent

Instead of using torch.optim or tf.optimizers, update parameters manually using the gradients computed during back-propagation. The examples/02-simple-neural-network.py file shows vanilla SGD: W -= lr * dW. You can extend this pattern to implement momentum or Adam by maintaining additional state variables (velocity estimates) alongside your parameters.

End-to-End Example: Training on XOR

Below is a complete, runnable implementation that combines these components to solve the XOR problem. This code follows the architectural patterns found in 02-simple-neural-network.py and Perceptron.ipynb.

import numpy as np

# ── Base Layer ───────────────────────────────────────────────────────

class Layer:
    def forward(self, x):
        raise NotImplementedError
    def backward(self, grad):
        raise NotImplementedError

# ── Linear (Fully‑Connected) Layer ─────────────────────────────────────

class Linear(Layer):
    def __init__(self, in_features, out_features):
        limit = np.sqrt(6 / (in_features + out_features))
        self.W = np.random.uniform(-limit, limit, (in_features, out_features))
        self.b = np.zeros(out_features)
    def forward(self, x):
        self.last_input = x                # cache for back‑prop

        return x @ self.W + self.b
    def backward(self, grad):
        # grad: dL/dout

        self.dW = self.last_input.T @ grad
        self.db = grad.sum(axis=0)
        return grad @ self.W.T             # dL/dinput

# ── ReLU Activation ───────────────────────────────────────────────────

class ReLU(Layer):
    def forward(self, x):
        self.mask = x > 0
        return x * self.mask
    def backward(self, grad):
        return grad * self.mask

# ── Simple MSE Loss ───────────────────────────────────────────────────

def mse_loss(pred, target):
    diff = pred - target
    return (diff ** 2).mean(), diff * 2 / diff.size

# ── Model Container ───────────────────────────────────────────────────

class Model:
    def __init__(self, layers):
        self.layers = layers
    def forward(self, x):
        for l in self.layers:
            x = l.forward(x)
        return x
    def backward(self, loss_grad):
        for l in reversed(self.layers):
            loss_grad = l.backward(loss_grad)

# ── Training Loop ─────────────────────────────────────────────────────

def train(model, X, y, lr=0.01, epochs=500):
    for epoch in range(epochs):
        # forward

        out = model.forward(X)
        loss, grad = mse_loss(out, y)
        # backward

        model.backward(grad)
        # SGD update

        for l in model.layers:
            if isinstance(l, Linear):
                l.W -= lr * l.dW
                l.b -= lr * l.db
        if epoch % 100 == 0:
            print(f"epoch {epoch:3d} – loss {loss:.4f}")

# ── Demo on XOR problem ───────────────────────────────────────────────

X = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=float)
y = np.array([[0],[1],[1],[0]], dtype=float)

net = Model([
    Linear(2, 4),
    ReLU(),
    Linear(4, 1),
    ReLU()
])

train(net, X, y, lr=0.1, epochs=1000)

# Test

print("Predictions:", net.forward(X).round())

This implementation demonstrates vectorized NumPy operations for weight updates, caching of intermediate values for back-propagation, and the sequential execution pattern found in the repository's convolutional examples.

Extending Your Framework

Once the basic framework functions, you can add sophisticated layers and utilities following the patterns in microsoft/AI-For-Beginners.

Convolutional Layers

The lessons/4-ComputerVision/07-ConvNets/pytorchcv.py file demonstrates kernel shapes, padding strategies, and weight initialization for convolutional layers. To implement Conv2D in your custom framework, replace the matrix multiplication in Linear with im2col operations or direct convolution loops, maintaining the same forward/backward interface.

Data Pipeline Utilities

For handling real datasets, refer to lessons/4-ComputerVision/08-TransferLearning/tfcv.py. This file shows preprocessing pipelines and batch generation strategies that you can adapt into custom DataLoader classes for your framework, enabling mini-batch training instead of full-batch gradient descent.

Summary

• Implement a base Layer class with forward() and backward() methods, as demonstrated in Perceptron.ipynb.
• Create concrete layers like Linear and ReLU using NumPy operations, following the patterns in 02-simple-neural-network.py.
• Build a Model container to execute layers sequentially and reverse the order for back-propagation, mirroring the logic in pytorchcv.py.
• Compute loss and gradients separately, referencing the cross-entropy implementation in 04-text-sentiment.py.
• Update parameters manually using gradient descent, or extend the optimizer logic to include momentum and adaptive learning rates.
• Cache only necessary intermediates (like inputs and masks) to enable efficient back-propagation without excessive memory overhead.

Frequently Asked Questions

Do I need PyTorch or TensorFlow to build a neural network from scratch?

No. The microsoft/AI-For-Beginners repository demonstrates that you can build a fully functional neural network framework using only Python and NumPy. Files like 02-simple-neural-network.py implement forward propagation, back-propagation, and gradient descent without any external deep learning libraries.

What is the purpose of the backward method in a custom layer?

The backward method computes gradients of the loss with respect to the layer's inputs and parameters. According to the implementation patterns in Perceptron.ipynb, each layer receives the gradient from the subsequent layer (grad), uses cached values from the forward pass to calculate local gradients, and returns the gradient with respect to its own input for propagation to earlier layers.

How does back-propagation work without automatic differentiation?

Manual back-propagation requires each layer to store intermediate values during the forward pass (such as inputs or activation masks) and use them to compute derivatives during the backward pass. As shown in the XOR example, the Linear layer caches last_input to compute dW = last_input.T @ grad, implementing the chain rule explicitly without autograd systems.

Can this custom framework handle convolutional neural networks?

Yes. While the initial examples use fully-connected layers, the architecture supports any layer implementing the base interface. The pytorchcv.py file in the repository demonstrates convolutional kernel operations and shape handling that you can translate into custom Conv2D layers using NumPy's stride tricks or im2col transformations, maintaining the same forward and backward contract.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}