Domain-Specific Phases 4-8 in AI Engineering from Scratch: A Complete Curriculum Guide

Phases 4 through 8 of the AI Engineering from Scratch curriculum cover Computer Vision, Natural Language Processing, Speech & Audio, Transformers Deep Dive, and Generative AI, with executable code artifacts located in phases/<phase-slug>/<lesson-slug>/code/main.py for hands-on practice.

The open-source repository rohitg00/ai-engineering-from-scratch structures AI engineering education into progressive phases, with the domain-specific phases 4-8 representing the core application stack that practitioners encounter after mastering foundational mathematics. According to the ROADMAP.md file, these five phases transition learners from theoretical deep learning into specialized domains including vision, language, audio, and generative modeling.

Phase 4 – Computer Vision

Computer Vision serves as the first domain-specific phase, introducing image fundamentals and convolutional architectures. As documented in ROADMAP.md at line 96, this phase progresses from basic image processing through Convolutional Neural Networks (CNNs) to modern architectures including Vision Transformers (ViT), diffusion models, and open-vocabulary vision systems.

The phase directory phases/04-computer-vision/ contains lessons that implement these concepts from scratch. Learners explore convolution operations, pooling layers, and attention mechanisms specific to visual data before advancing to state-of-the-art approaches like CLIP and segmentation models.

Phase 5 – Natural Language Processing (NLP)

Natural Language Processing (NLP) encompasses the full text processing pipeline from raw data to generation. Located in phases/05-nlp-foundations-to-advanced/, this phase covers tokenization algorithms (including BPE), embeddings (Word2Vec, GloVe, and contextual embeddings), and the evolution from RNNs to Transformer-based language models.

The curriculum addresses both classical techniques and modern implementations, with lessons spanning text preprocessing, sequence modeling, and generation strategies. Reference the roadmap entry at line 129 for the complete lesson structure covering everything from bag-of-words to large language model foundations.

Phase 6 – Speech & Audio

Speech & Audio processing introduces multimodal learning through waveform manipulation and spectrogram analysis. The phases/06-speech-and-audio/ directory contains implementations of Automatic Speech Recognition (ASR), Text-to-Speech (TTS) systems, voice cloning techniques, and audio-language models like Whisper.

Lessons in this phase handle raw audio signal processing, Fourier transforms, and mel-spectrogram generation before implementing end-to-end speech systems. The roadmap entry at line 163 details coverage of encoder-decoder architectures specifically optimized for temporal audio sequences.

Phase 7 – Transformers Deep Dive

Transformers Deep Dive provides the architectural foundation underlying modern AI systems. Located in phases/07-transformers-deep-dive/, this phase dissects self-attention mechanisms, multi-head attention, positional encoding, and the complete encoder-decoder stack as originally proposed in "Attention Is All You Need."

The curriculum implements these components from scratch in main.py files, including optimization techniques like Flash Attention for memory efficiency. Reference line 185 in ROADMAP.md for the specific progression from attention basics to full Transformer implementations.

Phase 8 – Generative AI

Generative AI synthesizes prior knowledge to create novel content across modalities. The phases/08-generative-ai/ directory surveys Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), diffusion models, and Latent Diffusion Models (LDMs) for image, video, audio, and 3D generation.

This phase covers conditioning techniques, classifier-free guidance, and evaluation metrics including FID and IS scores. As noted at line 206 of the roadmap, lessons implement DDPM (Denoising Diffusion Probabilistic Models) from scratch and extend to modern latent diffusion architectures.

Running the Phase Code Examples

Each phase contains runnable Python artifacts in phases/<phase-slug>/<lesson-slug>/code/main.py. Execute these examples from the repository root after installing dependencies from requirements.txt.

Run a specific phase lesson using the following patterns:

# Phase 4: Execute CNN convolution implementation

import subprocess
subprocess.run([
    "python3", 
    "phases/04-computer-vision/02-convolutions-from-scratch/code/main.py"
])

# Phase 5: Run BPE tokenization demonstration

subprocess.run([
    "python3", 
    "phases/05-nlp-foundations-to-advanced/01-text-processing/code/main.py"
])

# Phase 6: Launch Whisper fine-tuning example

subprocess.run([
    "python3", 
    "phases/06-speech-and-audio/05-whisper-architecture-finetuning/code/main.py"
])

# Phase 7: Build mini-Transformer from scratch

subprocess.run([
    "python3", 
    "phases/07-transformers-deep-dive/05-full-transformer/code/main.py"
])

# Phase 8: Train tiny diffusion model (DDPM)

subprocess.run([
    "python3", 
    "phases/08-generative-ai/06-diffusion-ddpm-from-scratch/code/main.py"
])

Summary

• Phase 4 (Computer Vision) covers CNNs, ViT, and diffusion models in phases/04-computer-vision/
• Phase 5 (NLP) implements tokenization, embeddings, and text pipelines in phases/05-nlp-foundations-to-advanced/
• Phase 6 (Speech & Audio) explores ASR, TTS, and Whisper architectures in phases/06-speech-and-audio/
• Phase 7 (Transformers) provides deep implementation of attention mechanisms and the encoder-decoder stack in phases/07-transformers-deep-dive/
• Phase 8 (Generative AI) surveys VAEs, GANs, and diffusion models for multimodal generation in phases/08-generative-ai/
• All phases reference specific line numbers in ROADMAP.md and contain executable main.py files for hands-on learning

Frequently Asked Questions

What prerequisites are needed before starting Phase 4?

According to the repository structure, Phases 1-3 cover mathematical foundations, Python for AI, and deep learning basics. Learners should master linear algebra, calculus, autograd, and MLP implementation before entering the domain-specific Phase 4 curriculum.

How is the Transformer architecture covered across different phases?

The curriculum distributes Transformer knowledge across three stages: Phase 5 applies pre-trained Transformers to NLP tasks, Phase 7 implements the architecture from scratch (self-attention, multi-head attention, positional encoding), and Phase 8 utilizes Transformer backbones for generative modeling.

Where can I find the executable code for the diffusion model lesson?

The DDPM (Denoising Diffusion Probabilistic Models) from-scratch implementation resides in phases/08-generative-ai/06-diffusion-ddpm-from-scratch/code/main.py. This corresponds to the Phase 8 roadmap entry at line 206 in ROADMAP.md.

Does the curriculum cover audio generation or only speech recognition?

Phase 6 covers both directions: ASR (Automatic Speech Recognition) and TTS (Text-to-Speech) implementations, including voice cloning. Additionally, Phase 8 extends to audio generation via latent diffusion models, providing comprehensive coverage of audio AI engineering.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}