How the AI Engineering from Scratch Curriculum Teaches Multimodal AI and Vision-Language Models (VLMs)

The curriculum teaches multimodal AI through a progressive "Build-It / Use-It" approach, implementing CLIP, BLIP-2, Flamingo, and LLaVA from scratch before integrating them into production-grade RAG pipelines and multimodal agents.

The ai-engineering-from-scratch repository treats multimodal AI as a progressive stack of architectural blocks, guiding learners from low-level vision fundamentals to full-blown vision-language models (VLMs). Following the repository’s core philosophy, every concept is first implemented using only the Python standard library, then re-implemented with production-grade frameworks to illustrate the same mathematics in real-world settings. This approach ensures students understand both the theoretical foundations and practical deployment of modern VLMs.

The Progressive Architecture Roadmap

The curriculum organizes multimodal learning into four interconnected phases, each building upon the previous to create a complete pipeline from pixel to agent action.

Phase 4: Computer Vision Foundations

Before tackling multimodal fusion, students master pixel-level feature extraction in phases/04-computer-vision/14-vision-transformers/docs/en.md. Here, learners build tiny convolutional networks and a minimal Vision Transformer (ViT) implementation that converts images into patch tokens. This groundwork establishes how visual information transforms into the embedding spaces that later feed into VLMs.

Phase 12: Multimodal AI Core

The heart of the VLM curriculum resides in Phase 12, which covers dual-encoder contrastive pre-training through cross-modal fusion and finally unified multimodal agents. Students implement three critical architectures:

1. CLIP with InfoNCE loss in phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py
2. BLIP-2 Q-Former bridge in phases/12-multimodal-ai/03-blip2-qformer-bridge/code/main.py
3. Flamingo-style gated cross-attention in phases/12-multimodal-ai/04-flamingo-gated-cross-attention/code/main.py

Phase 11: Multimodal RAG Integration

In phases/11-llm-engineering/06-rag/code/main.py, the curriculum demonstrates RAG-style multimodal retrieval. Students build a simple Multimodal RAG demo that concatenates CLIP image embeddings to text prompts, enabling vision-conditioned language generation using LangChain-style wrappers.

Phase 14: Multimodal Agent Engineering

The capstone in phases/14-agent-engineering/25-multimodal-agents-computer-use/code/main.py implements closed-loop agents that consume image, audio, and text streams simultaneously. This lesson wires vision embeddings to tool-calling planners, creating agents that can see, hear, reason, and act through MCP client/server protocols.

Building a VLM from Scratch: The Six-Step Pipeline

The curriculum constructs a complete vision-language model through six incremental steps, each implemented first in pure Python then with PyTorch or JAX.

Step 1: From Pixels to Patch Tokens

Students begin with convolutional layers to understand spatial locality, then progress to ViT implementations that tokenize images into patches. This transformation from raw pixels to structured embeddings mirrors how modern VLMs process visual input.

Step 2: Dual-Encoder Contrastive Learning (CLIP)

The CLIP lesson derives the InfoNCE loss from mutual information principles, implementing a numerically-stable version that enables zero-shot classification. In phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py, students write the core training logic:

# Toy CLIP training step – InfoNCE loss

import math, numpy as np

def info_nce(sim_matrix, tau=0.07):
    # sim_matrix shape (N, N) – cosine similarities divided by temperature

    logits = sim_matrix / tau
    # row‑wise softmax → cross‑entropy against identity labels

    log_probs = logits - np.log(np.exp(logits).sum(axis=1, keepdims=True))
    loss_i2t = -np.mean(np.diag(log_probs))
    # column‑wise (caption→image) loss

    loss_t2i = -np.mean(np.diag(log_probs.T))
    return (loss_i2t + loss_t2i) / 2

This implementation teaches temperature scaling and contrastive objectives without framework abstraction.

Step 3: Scaling with SigLIP

The curriculum introduces SigLIP as an evolution of CLIP, replacing the softmax with a per-pair sigmoid loss. This modification allows batch sizes exceeding 300k without all-gather communication, teaching students how modern VLMs achieve massive scale.

Step 4: Cross-Modal Fusion with BLIP-2

Moving beyond dual encoders, students implement the Q-Former architecture in phases/12-multimodal-ai/03-blip2-qformer-bridge/code/main.py. This bridge transforms frozen image embeddings into query vectors for language models:

# BLIP‑2 Q‑Former bridge – turning image features into query vectors

def q_former(image_feats, q_proj):
    """
    image_feats: (N, D) visual tokens from frozen CLIP
    q_proj: linear layer that maps D → Q (query dim)
    Returns a set of query vectors for the LLM decoder.
    """
    return q_proj(image_feats)      # (N, Q)

This enables image-to-text generation by connecting vision encoders to frozen LLMs.

Step 5: Gated Cross-Attention (Flamingo)

The Flamingo lesson adds gated cross-attention layers that allow language models to attend to visual tokens while preserving linguistic conditioning. Implemented in phases/12-multimodal-ai/04-flamingo-gated-cross-attention/code/main.py, this architecture demonstrates how to inject vision into next-token prediction without full model retraining.

Step 6: Visual Instruction Tuning (LLaVA)

Finally, students fine-tune CLIP vision towers with visual instruction data following the LLaVA and LLaVA-OneVision recipes. This converts the VLM into an interactive assistant capable of following complex multimodal instructions.

Production Integration: RAG and Multimodal Agents

Beyond training, the curriculum emphasizes deployment through skill artifacts emitted by each lesson.

Multimodal RAG Implementation

The repository demonstrates zero-shot classification using trained embeddings:

# Zero‑shot classification using a CLIP checkpoint

def zero_shot(image_emb, class_names, tokenizer, text_encoder, temperature=0.07):
    # Build prompt templates

    prompts = [f"a photo of a {c}" for c in class_names]
    text_emb = text_encoder(tokenizer(prompts))           # (C, D)

    # Normalise embeddings

    image_emb = image_emb / np.linalg.norm(image_emb)
    text_emb  = text_emb / np.linalg.norm(text_emb, axis=1, keepdims=True)
    # Cosine similarity

    sims = image_emb @ text_emb.T / temperature
    return class_names[np.argmax(sims)]

These skills output MD-formatted artifacts (e.g., skill-clip-zero-shot.md) that integrate into any downstream pipeline.

Closed-Loop Agent Architecture

The final integration in Phase 14 creates a multimodal agent skeleton that queries CLIP skills, feeds results to a planner, and executes tool calls. Using the MCP protocol from phases/13-tools-and-protocols/07-building-an-mcp-server, agents stream vision, audio, and text between services.

Summary

• The curriculum teaches multimodal AI through a progressive stack: Vision Transformers → CLIP → BLIP-2 → Flamingo → LLaVA → Agents.
• Students implement InfoNCE loss, Q-Former bridges, and gated cross-attention from scratch in phases/12-multimodal-ai/.
• Every lesson produces reusable skill artifacts compatible with production frameworks like Hugging Face and LangChain.
• The final capstone builds closed-loop multimodal agents that combine vision, language, and tool use in phases/14-agent-engineering/.

Frequently Asked Questions

What is the "Build-It / Use-It" philosophy in multimodal AI?

The "Build-It / Use-It" approach requires students to first implement algorithms using only the Python standard library, then re-implement them with production frameworks like PyTorch or JAX. For multimodal AI, this means writing the InfoNCE loss and attention mechanisms in pure NumPy before using Hugging Face transformers, ensuring deep understanding of the mathematics behind VLMs.

How does the curriculum teach CLIP and contrastive learning?

The curriculum dedicates phases/12-multimodal-ai/02-clip-contrastive-pretraining/ to CLIP, where students derive and implement the InfoNCE loss from mutual information principles. The lesson covers temperature scaling, dual-encoder architectures, and zero-shot classification, culminating in a skill artifact that can be dropped into any RAG system.

What is the difference between BLIP-2 and Flamingo architectures in the curriculum?

BLIP-2 uses a Q-Former bridge to transform frozen image embeddings into query vectors for LLMs, enabling efficient image-to-text generation. Flamingo instead employs gated cross-attention layers inserted into existing language models, allowing the LLM to attend to visual tokens while preserving its linguistic capabilities. The curriculum implements both in phases/12-multimodal-ai/03-blip2-qformer-bridge/ and phases/12-multimodal-ai/04-flamingo-gated-cross-attention/ respectively.

How are multimodal agents implemented in the final phase?

Phase 14 (phases/14-agent-engineering/25-multimodal-agents-computer-use/) implements a multimodal agent that consumes image embeddings from CLIP, audio embeddings, and text prompts simultaneously. The agent uses a planner to route these inputs to appropriate tools, executing actions through MCP servers that stream vision and audio between services, creating a closed-loop system that can perceive and act upon the world.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}