Deep Learning Core Phase: 13 Fundamental Concepts from First Principles

The Deep Learning Core phase teaches you to build neural networks from mathematical foundations, covering perceptrons, backpropagation, optimizers, and regularization through 13 hands-on lessons that culminate in a reusable mini-framework.

The Deep Learning Core phase (Phase 3) of the rohitg00/ai-engineering-from-scratch repository provides a complete implementation-first curriculum. You progress from a single-neuron classifier to a functioning deep-learning library, mastering every mathematical operation and design decision along the way. Each lesson includes runnable Python code located in phases/03-deep-learning-core/ that demonstrates exactly how high-level frameworks like PyTorch and JAX implement their underlying mechanics.

Foundational Building Blocks: Perceptrons and Backpropagation

The Perceptron and Linear Classification

You begin by implementing the Perceptron class in phases/03-deep-learning-core/01-the-perceptron/code/perceptron.py to understand how a single neuron makes binary decisions. The lesson demonstrates the perceptron learning rule, where weights update proportionally to the classification error. Crucially, you implement the XOR dataset to prove that a single-layer network fails on non-linearly separable problems, establishing the mathematical necessity for hidden layers.

class Perceptron:
    def __init__(self, n_inputs, lr=0.1):
        self.weights = [0.0] * n_inputs
        self.bias = 0.0
        self.lr = lr

    def predict(self, inputs):
        total = sum(w * x for w, x in zip(self.weights, inputs)) + self.bias
        return 1 if total >= 0 else 0

    def train(self, data, epochs=100):
        for _ in range(epochs):
            for x, y in data:
                y_pred = self.predict(x)
                err = y - y_pred
                if err:
                    self.weights = [w + self.lr * err * xi for w, xi in zip(self.weights, x)]
                    self.bias += self.lr * err

Multi-Layer Networks and Non-Linearity

In phases/03-deep-learning-core/02-multi-layer-networks/code/main.py, you stack perceptrons to build a two-layer network that successfully solves the XOR problem. This introduces the concept of hidden layers and demonstrates how depth creates non-linear decision boundaries that single neurons cannot represent.

Backpropagation Implementation

The phases/03-deep-learning-core/03-backpropagation/code/main.py lesson teaches you to derive and implement the gradient-based learning rule manually. You code the forward pass, calculate partial derivatives for each weight, and perform backward propagation to update parameters automatically. The TwoLayerNetwork class demonstrates this end-to-end:

class TwoLayerNetwork:
    def __init__(self, lr=0.5):
        import random, math
        random.seed(0)
        self.w_hidden = [[random.uniform(-1, 1) for _ in range(2)] for _ in range(2)]
        self.b_hidden = [random.uniform(-1, 1) for _ in range(2)]
        self.w_out = [random.uniform(-1, 1) for _ in range(2)]
        self.b_out = random.uniform(-1, 1)
        self.lr = lr

    def sigmoid(self, x):
        x = max(-500, min(500, x))
        return 1.0 / (1.0 + math.exp(-x))

    def forward(self, x):
        self.hidden = [self.sigmoid(sum(w * xi for w, xi in zip(row, x)) + b)
                       for row, b in zip(self.w_hidden, self.b_hidden)]
        z = sum(w * h for w, h in zip(self.w_out, self.hidden)) + self.b_out
        self.out = self.sigmoid(z)
        return self.out

Essential Network Components

Activation Functions

Located in phases/03-deep-learning-core/04-activation-functions/code/main.py, this module has you implement sigmoid, tanh, ReLU, Leaky ReLU, GELU, Swish, and softmax with their derivatives. You analyze the vanishing gradient problem in saturating functions like sigmoid and the dying ReLU problem to understand why modern architectures prefer alternatives like GELU.

def get_activation(name):
    import math
    if name == "relu":
        return lambda x: max(0.0, x)
    if name == "gelu":
        return lambda x: 0.5 * x * (1 + math.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * x**3)))
    if name == "sigmoid":
        return lambda x: 1 / (1 + math.exp(-x))
    # add others as needed

Loss Functions

The phases/03-deep-learning-core/05-loss-functions/code/main.py lesson covers Mean Squared Error (MSE) for regression tasks and cross-entropy for classification. You implement both from scratch, learning when to use each based on output activation compatibility and gradient behavior.

Optimizers

In phases/03-deep-learning-core/06-optimizers/code/main.py, you build Stochastic Gradient Descent (SGD), momentum, RMSProp, and Adam optimizers. The lesson emphasizes how learning rate and momentum parameters affect convergence speed and stability, giving you intuition for hyperparameter tuning.

Training Stability and Regularization

Weight Initialization

The phases/03-deep-learning-core/08-weight-initialization/code/main.py module explains why random initialization is necessary and implements Xavier and He initialization schemes. You learn that proper scaling of initial weights prevents vanishing or exploding gradients in deep networks by maintaining variance across layers.

Regularization Techniques

Located in phases/03-deep-learning-core/07-regularization/code/main.py, this lesson teaches L2 weight decay, dropout, and early stopping to combat overfitting. You implement dropout masks during training and scaling at test time to understand how ensemble effects improve generalization.

Learning-Rate Schedules

The phases/03-deep-learning-core/09-learning-rate-schedules/code/main.py lesson covers step decay, exponential decay, and cosine annealing. You learn to decay learning rates during training to settle into sharper minima and avoid oscillation around optima.

Production-Ready Implementation

Mini-Framework Architecture

The phases/03-deep-learning-core/10-mini-framework/code/main.py lesson synthesizes previous concepts into a tiny, reusable NN library. You design modular components—layers, activations, loss functions, and optimizers—that compose into complex architectures, mirroring the API design of production frameworks.

Debugging Neural Networks

In phases/03-deep-learning-core/13-debugging-neural-networks/code/debug_neural_nets.py, you learn systematic debugging through gradient checking, loss visualization, and intermediate activation monitoring. These techniques help you identify implementation errors when building custom architectures from scratch.

Framework Integration

The final lessons map your hand-crafted implementations to industrial frameworks. phases/03-deep-learning-core/12-intro-to-jax/code/jax_intro.py and phases/03-deep-learning-core/11-intro-to-pytorch/code/pytorch_intro.py demonstrate how the mathematical concepts translate to JAX and PyTorch APIs, validating your understanding against optimized implementations.

Summary

• The Deep Learning Core phase provides 13 lessons that build neural networks from mathematical first principles without relying on high-level abstractions.
• You implement the complete training pipeline—from perceptrons and backpropagation to Adam optimization and learning-rate scheduling—in pure Python.
• Key components include activation functions (ReLU, GELU, sigmoid), initialization schemes (Xavier, He), and regularization (dropout, L2) located in specific files under phases/03-deep-learning-core/.
• The curriculum culminates in a mini-framework that demonstrates proper software architecture for deep learning libraries before transitioning to JAX and PyTorch.

Frequently Asked Questions

What prerequisites are needed for the Deep Learning Core phase?

You need basic Python programming and high-school-level calculus, specifically understanding of derivatives and partial derivatives. Linear algebra fundamentals (matrix multiplication) are helpful but reviewed within the context of the code implementations in phases/03-deep-learning-core/.

How does this phase differ from using PyTorch or TensorFlow directly?

Unlike framework-specific tutorials, this phase forces you to implement every mathematical operation—from the sigmoid derivative to the Adam update rule—in raw Python. When you later use torch.nn or jax.numpy, you understand exactly what happens inside the black box because you have built identical functionality in phases/03-deep-learning-core/03-backpropagation/code/main.py and related files.

Which lesson covers the most important concept for debugging neural networks?

Lesson 13, "Debugging Neural Networks" (phases/03-deep-learning-core/13-debugging-neural-networks/code/debug_neural_nets.py), is critical as it teaches gradient checking—numerically verifying your backpropagation implementation against finite differences. This technique catches bugs that silent failures in loss convergence would otherwise mask.

Can I skip to the mini-framework lesson without doing the previous exercises?

Skipping is not recommended. The phases/03-deep-learning-core/10-mini-framework/code/main.py file assumes you have implemented and understood the components from previous lessons (activations, losses, optimizers). Each lesson builds mathematical dependencies; the mini-framework is essentially a refactoring of code you write in lessons 1–9.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}