# Deep Learning Core Phase: 13 Fundamental Concepts from First Principles

> Master deep learning fundamentals. Build neural networks from scratch covering perceptrons, backpropagation, optimizers, and regularization with 13 hands-on lessons.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-08-30

---

**The Deep Learning Core phase teaches you to build neural networks from mathematical foundations, covering perceptrons, backpropagation, optimizers, and regularization through 13 hands-on lessons that culminate in a reusable mini-framework.**

The Deep Learning Core phase (Phase 3) of the [rohitg00/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch) repository provides a complete implementation-first curriculum. You progress from a single-neuron classifier to a functioning deep-learning library, mastering every mathematical operation and design decision along the way. Each lesson includes runnable Python code located in `phases/03-deep-learning-core/` that demonstrates exactly how high-level frameworks like PyTorch and JAX implement their underlying mechanics.

## Foundational Building Blocks: Perceptrons and Backpropagation

### The Perceptron and Linear Classification

You begin by implementing the `Perceptron` class in [`phases/03-deep-learning-core/01-the-perceptron/code/perceptron.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/01-the-perceptron/code/perceptron.py) to understand how a single neuron makes binary decisions. The lesson demonstrates the **perceptron learning rule**, where weights update proportionally to the classification error. Crucially, you implement the XOR dataset to prove that a single-layer network fails on non-linearly separable problems, establishing the mathematical necessity for hidden layers.

```python
class Perceptron:
    def __init__(self, n_inputs, lr=0.1):
        self.weights = [0.0] * n_inputs
        self.bias = 0.0
        self.lr = lr

    def predict(self, inputs):
        total = sum(w * x for w, x in zip(self.weights, inputs)) + self.bias
        return 1 if total >= 0 else 0

    def train(self, data, epochs=100):
        for _ in range(epochs):
            for x, y in data:
                y_pred = self.predict(x)
                err = y - y_pred
                if err:
                    self.weights = [w + self.lr * err * xi for w, xi in zip(self.weights, x)]
                    self.bias += self.lr * err

```

### Multi-Layer Networks and Non-Linearity

In [`phases/03-deep-learning-core/02-multi-layer-networks/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/02-multi-layer-networks/code/main.py), you stack perceptrons to build a **two-layer network** that successfully solves the XOR problem. This introduces the concept of **hidden layers** and demonstrates how depth creates non-linear decision boundaries that single neurons cannot represent.

### Backpropagation Implementation

The [`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py) lesson teaches you to derive and implement the **gradient-based learning rule** manually. You code the forward pass, calculate partial derivatives for each weight, and perform backward propagation to update parameters automatically. The `TwoLayerNetwork` class demonstrates this end-to-end:

```python
class TwoLayerNetwork:
    def __init__(self, lr=0.5):
        import random, math
        random.seed(0)
        self.w_hidden = [[random.uniform(-1, 1) for _ in range(2)] for _ in range(2)]
        self.b_hidden = [random.uniform(-1, 1) for _ in range(2)]
        self.w_out = [random.uniform(-1, 1) for _ in range(2)]
        self.b_out = random.uniform(-1, 1)
        self.lr = lr

    def sigmoid(self, x):
        x = max(-500, min(500, x))
        return 1.0 / (1.0 + math.exp(-x))

    def forward(self, x):
        self.hidden = [self.sigmoid(sum(w * xi for w, xi in zip(row, x)) + b)
                       for row, b in zip(self.w_hidden, self.b_hidden)]
        z = sum(w * h for w, h in zip(self.w_out, self.hidden)) + self.b_out
        self.out = self.sigmoid(z)
        return self.out

```

## Essential Network Components

### Activation Functions

Located in [`phases/03-deep-learning-core/04-activation-functions/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/04-activation-functions/code/main.py), this module has you implement **sigmoid**, **tanh**, **ReLU**, **Leaky ReLU**, **GELU**, **Swish**, and **softmax** with their derivatives. You analyze the **vanishing gradient problem** in saturating functions like sigmoid and the **dying ReLU problem** to understand why modern architectures prefer alternatives like GELU.

```python
def get_activation(name):
    import math
    if name == "relu":
        return lambda x: max(0.0, x)
    if name == "gelu":
        return lambda x: 0.5 * x * (1 + math.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * x**3)))
    if name == "sigmoid":
        return lambda x: 1 / (1 + math.exp(-x))
    # add others as needed

```

### Loss Functions

The [`phases/03-deep-learning-core/05-loss-functions/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/05-loss-functions/code/main.py) lesson covers **Mean Squared Error (MSE)** for regression tasks and **cross-entropy** for classification. You implement both from scratch, learning when to use each based on output activation compatibility and gradient behavior.

### Optimizers

In [`phases/03-deep-learning-core/06-optimizers/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/06-optimizers/code/main.py), you build **Stochastic Gradient Descent (SGD)**, **momentum**, **RMSProp**, and **Adam** optimizers. The lesson emphasizes how **learning rate** and **momentum** parameters affect convergence speed and stability, giving you intuition for hyperparameter tuning.

## Training Stability and Regularization

### Weight Initialization

The [`phases/03-deep-learning-core/08-weight-initialization/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/08-weight-initialization/code/main.py) module explains why random initialization is necessary and implements **Xavier** and **He** initialization schemes. You learn that proper scaling of initial weights prevents **vanishing or exploding gradients** in deep networks by maintaining variance across layers.

### Regularization Techniques

Located in [`phases/03-deep-learning-core/07-regularization/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/07-regularization/code/main.py), this lesson teaches **L2 weight decay**, **dropout**, and **early stopping** to combat overfitting. You implement dropout masks during training and scaling at test time to understand how ensemble effects improve generalization.

### Learning-Rate Schedules

The [`phases/03-deep-learning-core/09-learning-rate-schedules/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/09-learning-rate-schedules/code/main.py) lesson covers **step decay**, **exponential decay**, and **cosine annealing**. You learn to decay learning rates during training to settle into sharper minima and avoid oscillation around optima.

## Production-Ready Implementation

### Mini-Framework Architecture

The [`phases/03-deep-learning-core/10-mini-framework/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/10-mini-framework/code/main.py) lesson synthesizes previous concepts into a **tiny, reusable NN library**. You design modular components—layers, activations, loss functions, and optimizers—that compose into complex architectures, mirroring the API design of production frameworks.

### Debugging Neural Networks

In [`phases/03-deep-learning-core/13-debugging-neural-networks/code/debug_neural_nets.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/13-debugging-neural-networks/code/debug_neural_nets.py), you learn systematic debugging through **gradient checking**, **loss visualization**, and intermediate activation monitoring. These techniques help you identify implementation errors when building custom architectures from scratch.

### Framework Integration

The final lessons map your hand-crafted implementations to industrial frameworks. [`phases/03-deep-learning-core/12-intro-to-jax/code/jax_intro.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/12-intro-to-jax/code/jax_intro.py) and [`phases/03-deep-learning-core/11-intro-to-pytorch/code/pytorch_intro.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/11-intro-to-pytorch/code/pytorch_intro.py) demonstrate how the mathematical concepts translate to **JAX** and **PyTorch** APIs, validating your understanding against optimized implementations.

## Summary

- **The Deep Learning Core phase** provides 13 lessons that build neural networks from mathematical first principles without relying on high-level abstractions.
- You implement the complete training pipeline—from **perceptrons** and **backpropagation** to **Adam optimization** and **learning-rate scheduling**—in pure Python.
- Key components include **activation functions** (ReLU, GELU, sigmoid), **initialization schemes** (Xavier, He), and **regularization** (dropout, L2) located in specific files under `phases/03-deep-learning-core/`.
- The curriculum culminates in a **mini-framework** that demonstrates proper software architecture for deep learning libraries before transitioning to JAX and PyTorch.

## Frequently Asked Questions

### What prerequisites are needed for the Deep Learning Core phase?

You need basic Python programming and high-school-level calculus, specifically understanding of derivatives and partial derivatives. Linear algebra fundamentals (matrix multiplication) are helpful but reviewed within the context of the code implementations in `phases/03-deep-learning-core/`.

### How does this phase differ from using PyTorch or TensorFlow directly?

Unlike framework-specific tutorials, this phase forces you to implement every mathematical operation—from the sigmoid derivative to the Adam update rule—in raw Python. When you later use `torch.nn` or `jax.numpy`, you understand exactly what happens inside the black box because you have built identical functionality in [`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py) and related files.

### Which lesson covers the most important concept for debugging neural networks?

Lesson 13, "Debugging Neural Networks" ([`phases/03-deep-learning-core/13-debugging-neural-networks/code/debug_neural_nets.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/13-debugging-neural-networks/code/debug_neural_nets.py)), is critical as it teaches **gradient checking**—numerically verifying your backpropagation implementation against finite differences. This technique catches bugs that silent failures in loss convergence would otherwise mask.

### Can I skip to the mini-framework lesson without doing the previous exercises?

Skipping is not recommended. The [`phases/03-deep-learning-core/10-mini-framework/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/10-mini-framework/code/main.py) file assumes you have implemented and understood the components from previous lessons (activations, losses, optimizers). Each lesson builds mathematical dependencies; the mini-framework is essentially a refactoring of code you write in lessons 1–9.