# Relationship Between Phase 3 Deep Learning Core and Later Phases NLP Vision Agents

> Discover how Phase 3 Deep Learning Core in ai-engineering-from-scratch empowers NLP, Vision, and Agent systems with foundational math and algorithms. See the code connect ML.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-06-12

---

**Phase 3 Deep Learning Core provides the foundational mathematical and algorithmic primitives—from tensor operations to gradient-based optimization—that every subsequent phase in the curriculum builds upon, enabling the same core code to power computer vision, natural language processing, and autonomous agent systems.**

The rohitg00/ai-engineering-from-scratch repository organizes AI education into progressive phases, where the relationship between Phase 3 Deep Learning Core and later phases like NLP, Vision, and Agents forms the architectural backbone of the entire curriculum. By implementing fundamental building blocks from scratch without external deep-learning libraries, Phase 3 creates a reusable codebase that higher-level phases import directly, ensuring learners understand exactly how production AI systems operate under the hood.

## The Six Pillars Connecting Phase 3 to Advanced AI

Phase 3 deliberately implements six essential primitives that later phases treat as the engine for domain-specific applications. Understanding these connections reveals how modern AI systems share common DNA regardless of their target modality.

### Linear Algebra and Tensor Operations

All higher-level models in the curriculum store weights as tensors, applying the same `matmul`, `reshape`, and broadcasting rules introduced in Phase 3. Whether processing image pixels in Phase 4 or word embeddings in Phase 5, the underlying numerical kernels remain identical.

### Gradient-Based Optimization and Backpropagation

The generic `SGD` and `Adam` implementations written in Phase 3 power training loops across Vision CNNs (Phase 4), NLP Transformers (Phase 5 and 7), and Reinforcement Learning agents (Phase 9). Each downstream phase calls these optimizers to update model parameters without reimplementing the calculus.

### Layer Abstractions and Modular Design

The `Layer` base class defined in [`phases/03-deep-learning-core/code/nn.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/code/nn.py) serves as the inheritance point for specialized implementations. Convolutional layers, attention mechanisms, and policy network layers all subclass this foundation, maintaining consistent forward and backward pass interfaces.

### Loss Functions and Regularization

Common utilities like `CrossEntropy`, `MSE`, and `KLDiv` implemented in Phase 3 support text classification (Phase 5), image classification (Phase 4), and multi-agent reward shaping (Phase 9). This standardization ensures that the mathematical objectives remain consistent across domains.

### Training Pipelines and Data Infrastructure

The `DataLoader` and `Trainer` scaffolding created in Phase 3 reappears in every downstream lesson. By reusing the same training loop code, the curriculum demonstrates that swapping datasets represents the only change needed to move from vision tasks to language modeling, highlighting the transferability of core AI engineering skills.

### Model Checkpointing and Persistence

Checkpoint utilities introduced in Phase 3 enable Phase 19 capstone projects to resume long training runs, allow agent policies in Phase 14 to persist learned behaviors, and support fine-tuning pipelines across the curriculum.

## How Phase 3 Powers Computer Vision (Phase 4)

Computer Vision in Phase 4 builds directly atop Phase 3's linear algebra primitives. The repository demonstrates this relationship through explicit imports where convolutional networks reuse the foundational `Linear` layer.

In [`phases/04-computer-vision/code/cnn.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/code/cnn.py), the implementation imports the core layer and integrates it into the final classification head:

```python

# Phase 4 – Vision CNN using the core Linear layer (phases/04-computer-vision/code/cnn.py)

from phases.03_deep_learning_core.nn import Linear

class ConvNet:
    def __init__(self):
        self.fc = Linear(1280, 10)   # reuse the Linear class

    def forward(self, img):
        x = self.conv_layers(img)
        x = x.reshape(x.shape[0], -1)
        return self.fc(x)

```

This pattern shows that even specialized vision architectures ultimately rely on the same matrix multiplication and gradient flow mechanisms established in Phase 3.

## From Core Layers to NLP and Transformers (Phases 5 and 7)

Natural Language Processing and Transformer architectures in Phases 5 and 7 demonstrate the flexibility of Phase 3's abstractions. The attention mechanism—fundamental to modern NLP—reuses the same `Linear` primitive to project queries, keys, and values.

In [`phases/05-nlp-foundations/code/transformer.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/05-nlp-foundations/code/transformer.py), the attention implementation imports and instantiates multiple Linear layers:

```python

# Phase 5 – NLP Transformer head re‑using the same Linear layer (phases/05-nlp-foundations/code/transformer.py)

from phases.03_deep_learning_core.nn import Linear

class Attention:
    def __init__(self, d_model):
        self.q_proj = Linear(d_model, d_model)
        self.k_proj = Linear(d_model, d_model)
        self.v_proj = Linear(d_model, d_model)

    def __call__(self, x):
        Q = self.q_proj(x)
        K = self.k_proj(x)
        V = self.v_proj(x)
        # … compute scaled dot‑product attention …

        return output

```

This reuse proves that the self-attention mechanism, despite its complex mathematical description, ultimately consists of the same dense matrix operations learners implemented in Phase 3.

## Powering Reinforcement Learning and Agent Systems (Phases 9 and 14)

Reinforcement Learning and Agent Engineering in Phases 9 and 14 extend Phase 3's utilities to sequential decision-making. Policy networks— the brains of autonomous agents— import the same Linear primitives to map observations to action probabilities.

In [`phases/09-reinforcement-learning/code/policy.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/09-reinforcement-learning/code/policy.py), the policy network architecture demonstrates this inheritance:

```python

# Phase 9 – RL agent policy network shares the same Linear primitive (phases/09-reinforcement-learning/code/policy.py)

from phases.03_deep_learning_core.nn import Linear

class PolicyNetwork:
    def __init__(self, obs_dim, act_dim):
        self.fc1 = Linear(obs_dim, 64)
        self.fc2 = Linear(64, act_dim)

    def act(self, obs):
        hidden = np.tanh(self.fc1(obs))
        logits = self.fc2(hidden)
        return softmax(logits)

```

Phase 14's Agent Engineering then composes these policy networks with vision and language modules, creating unified systems where every component traces back to Phase 3's core implementations.

## Summary

The relationship between Phase 3 Deep Learning Core and later phases in rohitg00/ai-engineering-from-scratch creates a cohesive learning architecture where fundamentals directly enable advanced applications:

- **Phase 3 implements the engine**: Tensor operations, autograd, optimizers, and layer abstractions written from scratch without external dependencies.
- **Later phases import the primitives**: Vision, NLP, and Agent modules explicitly import `Linear`, optimizers, and training utilities from `phases/03-deep-learning-core/`.
- **Single codebase powers multiple modalities**: The same `matmul` kernels and gradient descent loops train image classifiers, language models, and reinforcement learning policies.
- **Checkpointing ensures continuity**: Persistence utilities created in Phase 3 support long-running experiments and model deployment in capstone projects.
- **Pedagogical clarity through reuse**: By witnessing the same code power diverse applications, learners internalize that AI engineering rests on a unified mathematical foundation rather than domain-specific magic.

## Frequently Asked Questions

### How does Phase 3 code get imported into Phase 4 Computer Vision lessons?

Phase 4 modules use explicit Python imports to access Phase 3 implementations. For example, [`phases/04-computer-vision/code/cnn.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/code/cnn.py) imports `from phases.03_deep_learning_core.nn import Linear`, allowing convolutional networks to use the exact same dense layer implementation that learners studied in the core phase. This import-based architecture ensures that vision models build transparently atop verified fundamentals.

### Why does the curriculum implement Phase 3 from scratch instead of using PyTorch or TensorFlow?

By implementing tensors, autograd, and optimizers manually in Phase 3, the curriculum removes the "framework magic" that typically obscures AI internals. When Phase 7 Transformers or Phase 9 Agents later import these primitives, learners understand precisely how matrix multiplication, gradient computation, and parameter updates occur. This foundation proves essential when debugging production systems or optimizing novel architectures.

### Can the Phase 3 utilities handle both CNNs and Transformer architectures without modification?

Yes. The `Linear` layer and optimizer implementations in Phase 3 are dimensionality-agnostic and architecture-agnostic. Phase 4 CNNs use `Linear` for final classification heads, while Phase 5 Transformers use identical `Linear` layers for query/key/value projections. The curriculum demonstrates that these diverse architectures share the same underlying linear algebra operations, differing only in how they arrange and connect these core components.

### What role does Phase 3 play in the capstone projects of Phase 19?

Phase 19 capstone projects integrate multiple domains—such as vision-based agents or language-guided robots—requiring the checkpointing, training loops, and layer utilities established in Phase 3. The curriculum's checkpoint utilities enable long-running training resumption, while the modular `Layer` base class allows students to compose heterogeneous components (convolutional encoders, attention mechanisms, and policy heads) into unified systems.