# How to Build a Custom Neural Network Framework from Scratch

> Learn to build a custom neural network framework from scratch. Implement modular layers, compose models, and master gradient descent for deep learning without TensorFlow or PyTorch.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: how-to-guide
- Published: 2026-08-18

---

**You can build a custom neural network framework from scratch by implementing modular layer classes with `forward()` and `backward()` methods, composing them into a sequential model container, and manually executing gradient descent to update parameters without relying on high-level libraries like TensorFlow or PyTorch.**

The microsoft/AI-For-Beginners repository contains pedagogical notebooks and scripts that demonstrate how to build a custom neural network framework from scratch using pure Python and NumPy. By studying these reference implementations—including `Perceptron.ipynb` and [`02-simple-neural-network.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/02-simple-neural-network.py)—you can understand the fundamental mechanics of forward propagation, back-propagation, and loss computation while maintaining full control over every computational step.

## Core Architecture Components

A custom neural network framework requires five essential components: a **base layer abstraction**, concrete layer implementations, a **model container**, a loss function, and an optimizer. The following sections map each component to specific files in the microsoft/AI-For-Beginners repository.

### Base Layer Class

Every neural network layer must store parameters and cache intermediate values for back-propagation. In `lessons/3-NeuralNetworks/03-Perceptron/Perceptron.ipynb`, the repository demonstrates this pattern with a `Neuron` class that implements `forward` and `backward` methods. Your base class should define these methods as interfaces, ensuring every concrete layer can participate in the computational graph.

### Linear and Activation Layers

The [`examples/02-simple-neural-network.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/examples/02-simple-neural-network.py) file provides a reference implementation for a **dense layer** using NumPy matrix operations. A `Linear` layer initializes weights using Xavier initialization and computes the affine transformation `x @ W + b` during the forward pass. During back-propagation, it calculates gradients with respect to weights (`dW`), biases (`db`), and inputs using the cached input from the forward pass.

Activation functions like **ReLU** and **Sigmoid** follow the same interface. They store masks or intermediate values during `forward` and apply element-wise derivatives during `backward`. The repository's simple neural network example demonstrates how to chain these layers sequentially.

### Model Container

To execute layers in sequence, you need a **Model** class that iterates through a list of layers. The [`lessons/4-ComputerVision/07-ConvNets/pytorchcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/07-ConvNets/pytorchcv.py) file illustrates this pattern using PyTorch's `nn.Sequential`, but the logic applies equally to custom frameworks. Your container should call `forward()` on each layer in order during prediction, then reverse the list to call `backward()` during training, passing gradients from the loss function back to the input layer.

## Loss Computation and Optimization

Separating loss calculation from parameter updates allows you to swap components without modifying the training loop.

### Implementing Loss Functions

The [`examples/04-text-sentiment.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/examples/04-text-sentiment.py) script demonstrates categorical cross-entropy loss computation for classification tasks. For regression, you can implement **mean squared error (MSE)** as shown in the repository's introductory examples. A loss function must return two values: the scalar loss for monitoring and the gradient of the loss with respect to the model's output (`dL/dout`) to seed the backward pass.

### Manual Gradient Descent

Instead of using `torch.optim` or `tf.optimizers`, update parameters manually using the gradients computed during back-propagation. The [`examples/02-simple-neural-network.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/examples/02-simple-neural-network.py) file shows vanilla SGD: `W -= lr * dW`. You can extend this pattern to implement momentum or Adam by maintaining additional state variables (velocity estimates) alongside your parameters.

## End-to-End Example: Training on XOR

Below is a complete, runnable implementation that combines these components to solve the XOR problem. This code follows the architectural patterns found in [`02-simple-neural-network.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/02-simple-neural-network.py) and `Perceptron.ipynb`.

```python
import numpy as np

# ── Base Layer ───────────────────────────────────────────────────────

class Layer:
    def forward(self, x):
        raise NotImplementedError
    def backward(self, grad):
        raise NotImplementedError

# ── Linear (Fully‑Connected) Layer ─────────────────────────────────────

class Linear(Layer):
    def __init__(self, in_features, out_features):
        limit = np.sqrt(6 / (in_features + out_features))
        self.W = np.random.uniform(-limit, limit, (in_features, out_features))
        self.b = np.zeros(out_features)
    def forward(self, x):
        self.last_input = x                # cache for back‑prop

        return x @ self.W + self.b
    def backward(self, grad):
        # grad: dL/dout

        self.dW = self.last_input.T @ grad
        self.db = grad.sum(axis=0)
        return grad @ self.W.T             # dL/dinput

# ── ReLU Activation ───────────────────────────────────────────────────

class ReLU(Layer):
    def forward(self, x):
        self.mask = x > 0
        return x * self.mask
    def backward(self, grad):
        return grad * self.mask

# ── Simple MSE Loss ───────────────────────────────────────────────────

def mse_loss(pred, target):
    diff = pred - target
    return (diff ** 2).mean(), diff * 2 / diff.size

# ── Model Container ───────────────────────────────────────────────────

class Model:
    def __init__(self, layers):
        self.layers = layers
    def forward(self, x):
        for l in self.layers:
            x = l.forward(x)
        return x
    def backward(self, loss_grad):
        for l in reversed(self.layers):
            loss_grad = l.backward(loss_grad)

# ── Training Loop ─────────────────────────────────────────────────────

def train(model, X, y, lr=0.01, epochs=500):
    for epoch in range(epochs):
        # forward

        out = model.forward(X)
        loss, grad = mse_loss(out, y)
        # backward

        model.backward(grad)
        # SGD update

        for l in model.layers:
            if isinstance(l, Linear):
                l.W -= lr * l.dW
                l.b -= lr * l.db
        if epoch % 100 == 0:
            print(f"epoch {epoch:3d} – loss {loss:.4f}")

# ── Demo on XOR problem ───────────────────────────────────────────────

X = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=float)
y = np.array([[0],[1],[1],[0]], dtype=float)

net = Model([
    Linear(2, 4),
    ReLU(),
    Linear(4, 1),
    ReLU()
])

train(net, X, y, lr=0.1, epochs=1000)

# Test

print("Predictions:", net.forward(X).round())

```

This implementation demonstrates **vectorized NumPy operations** for weight updates, **caching** of intermediate values for back-propagation, and the **sequential execution** pattern found in the repository's convolutional examples.

## Extending Your Framework

Once the basic framework functions, you can add sophisticated layers and utilities following the patterns in microsoft/AI-For-Beginners.

### Convolutional Layers

The [`lessons/4-ComputerVision/07-ConvNets/pytorchcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/07-ConvNets/pytorchcv.py) file demonstrates kernel shapes, padding strategies, and weight initialization for convolutional layers. To implement `Conv2D` in your custom framework, replace the matrix multiplication in `Linear` with im2col operations or direct convolution loops, maintaining the same `forward`/`backward` interface.

### Data Pipeline Utilities

For handling real datasets, refer to [`lessons/4-ComputerVision/08-TransferLearning/tfcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/08-TransferLearning/tfcv.py). This file shows preprocessing pipelines and batch generation strategies that you can adapt into custom `DataLoader` classes for your framework, enabling mini-batch training instead of full-batch gradient descent.

## Summary

- **Implement a base `Layer` class** with `forward()` and `backward()` methods, as demonstrated in `Perceptron.ipynb`.
- **Create concrete layers** like `Linear` and `ReLU` using NumPy operations, following the patterns in [`02-simple-neural-network.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/02-simple-neural-network.py).
- **Build a `Model` container** to execute layers sequentially and reverse the order for back-propagation, mirroring the logic in [`pytorchcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/pytorchcv.py).
- **Compute loss and gradients** separately, referencing the cross-entropy implementation in [`04-text-sentiment.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/04-text-sentiment.py).
- **Update parameters manually** using gradient descent, or extend the optimizer logic to include momentum and adaptive learning rates.
- **Cache only necessary intermediates** (like inputs and masks) to enable efficient back-propagation without excessive memory overhead.

## Frequently Asked Questions

### Do I need PyTorch or TensorFlow to build a neural network from scratch?

No. The microsoft/AI-For-Beginners repository demonstrates that you can build a fully functional neural network framework using only Python and NumPy. Files like [`02-simple-neural-network.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/02-simple-neural-network.py) implement forward propagation, back-propagation, and gradient descent without any external deep learning libraries.

### What is the purpose of the `backward` method in a custom layer?

The `backward` method computes gradients of the loss with respect to the layer's inputs and parameters. According to the implementation patterns in `Perceptron.ipynb`, each layer receives the gradient from the subsequent layer (`grad`), uses cached values from the forward pass to calculate local gradients, and returns the gradient with respect to its own input for propagation to earlier layers.

### How does back-propagation work without automatic differentiation?

Manual back-propagation requires each layer to store intermediate values during the forward pass (such as inputs or activation masks) and use them to compute derivatives during the backward pass. As shown in the XOR example, the `Linear` layer caches `last_input` to compute `dW = last_input.T @ grad`, implementing the chain rule explicitly without autograd systems.

### Can this custom framework handle convolutional neural networks?

Yes. While the initial examples use fully-connected layers, the architecture supports any layer implementing the base interface. The [`pytorchcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/pytorchcv.py) file in the repository demonstrates convolutional kernel operations and shape handling that you can translate into custom `Conv2D` layers using NumPy's stride tricks or im2col transformations, maintaining the same `forward` and `backward` contract.