# How the AI Engineering from Scratch Curriculum Teaches Multimodal AI and Vision-Language Models (VLMs)

> Learn multimodal AI and VLMs with the AI Engineering from Scratch curriculum. Build CLIP, BLIP-2, Flamingo, and LLaVA from scratch for production RAG pipelines and agents.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-06-16

---

**The curriculum teaches multimodal AI through a progressive "Build-It / Use-It" approach, implementing CLIP, BLIP-2, Flamingo, and LLaVA from scratch before integrating them into production-grade RAG pipelines and multimodal agents.**

The **ai-engineering-from-scratch** repository treats multimodal AI as a progressive stack of architectural blocks, guiding learners from low-level vision fundamentals to full-blown vision-language models (VLMs). Following the repository’s core philosophy, every concept is first implemented using only the Python standard library, then re-implemented with production-grade frameworks to illustrate the same mathematics in real-world settings. This approach ensures students understand both the theoretical foundations and practical deployment of modern VLMs.

## The Progressive Architecture Roadmap

The curriculum organizes multimodal learning into four interconnected phases, each building upon the previous to create a complete pipeline from pixel to agent action.

### Phase 4: Computer Vision Foundations

Before tackling multimodal fusion, students master **pixel-level feature extraction** in [`phases/04-computer-vision/14-vision-transformers/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/14-vision-transformers/docs/en.md). Here, learners build tiny convolutional networks and a minimal Vision Transformer (ViT) implementation that converts images into patch tokens. This groundwork establishes how visual information transforms into the embedding spaces that later feed into VLMs.

### Phase 12: Multimodal AI Core

The heart of the VLM curriculum resides in Phase 12, which covers **dual-encoder contrastive pre-training** through **cross-modal fusion** and finally **unified multimodal agents**. Students implement three critical architectures:

1. **CLIP** with InfoNCE loss in [`phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py)
2. **BLIP-2 Q-Former** bridge in [`phases/12-multimodal-ai/03-blip2-qformer-bridge/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/03-blip2-qformer-bridge/code/main.py)
3. **Flamingo-style gated cross-attention** in [`phases/12-multimodal-ai/04-flamingo-gated-cross-attention/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/04-flamingo-gated-cross-attention/code/main.py)

### Phase 11: Multimodal RAG Integration

In [`phases/11-llm-engineering/06-rag/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/11-llm-engineering/06-rag/code/main.py), the curriculum demonstrates **RAG-style multimodal retrieval**. Students build a simple Multimodal RAG demo that concatenates CLIP image embeddings to text prompts, enabling vision-conditioned language generation using LangChain-style wrappers.

### Phase 14: Multimodal Agent Engineering

The capstone in [`phases/14-agent-engineering/25-multimodal-agents-computer-use/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/14-agent-engineering/25-multimodal-agents-computer-use/code/main.py) implements **closed-loop agents** that consume image, audio, and text streams simultaneously. This lesson wires vision embeddings to tool-calling planners, creating agents that can see, hear, reason, and act through MCP client/server protocols.

## Building a VLM from Scratch: The Six-Step Pipeline

The curriculum constructs a complete vision-language model through six incremental steps, each implemented first in pure Python then with PyTorch or JAX.

### Step 1: From Pixels to Patch Tokens

Students begin with convolutional layers to understand spatial locality, then progress to ViT implementations that **tokenize images into patches**. This transformation from raw pixels to structured embeddings mirrors how modern VLMs process visual input.

### Step 2: Dual-Encoder Contrastive Learning (CLIP)

The CLIP lesson derives the **InfoNCE loss** from mutual information principles, implementing a numerically-stable version that enables **zero-shot classification**. In [`phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py), students write the core training logic:

```python

# Toy CLIP training step – InfoNCE loss

import math, numpy as np

def info_nce(sim_matrix, tau=0.07):
    # sim_matrix shape (N, N) – cosine similarities divided by temperature

    logits = sim_matrix / tau
    # row‑wise softmax → cross‑entropy against identity labels

    log_probs = logits - np.log(np.exp(logits).sum(axis=1, keepdims=True))
    loss_i2t = -np.mean(np.diag(log_probs))
    # column‑wise (caption→image) loss

    loss_t2i = -np.mean(np.diag(log_probs.T))
    return (loss_i2t + loss_t2i) / 2

```

This implementation teaches temperature scaling and contrastive objectives without framework abstraction.

### Step 3: Scaling with SigLIP

The curriculum introduces **SigLIP** as an evolution of CLIP, replacing the softmax with a per-pair sigmoid loss. This modification allows **batch sizes exceeding 300k** without all-gather communication, teaching students how modern VLMs achieve massive scale.

### Step 4: Cross-Modal Fusion with BLIP-2

Moving beyond dual encoders, students implement the **Q-Former** architecture in [`phases/12-multimodal-ai/03-blip2-qformer-bridge/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/03-blip2-qformer-bridge/code/main.py). This bridge transforms frozen image embeddings into query vectors for language models:

```python

# BLIP‑2 Q‑Former bridge – turning image features into query vectors

def q_former(image_feats, q_proj):
    """
    image_feats: (N, D) visual tokens from frozen CLIP
    q_proj: linear layer that maps D → Q (query dim)
    Returns a set of query vectors for the LLM decoder.
    """
    return q_proj(image_feats)      # (N, Q)

```

This enables **image-to-text generation** by connecting vision encoders to frozen LLMs.

### Step 5: Gated Cross-Attention (Flamingo)

The Flamingo lesson adds **gated cross-attention layers** that allow language models to attend to visual tokens while preserving linguistic conditioning. Implemented in [`phases/12-multimodal-ai/04-flamingo-gated-cross-attention/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/04-flamingo-gated-cross-attention/code/main.py), this architecture demonstrates how to inject vision into next-token prediction without full model retraining.

### Step 6: Visual Instruction Tuning (LLaVA)

Finally, students fine-tune CLIP vision towers with **visual instruction data** following the LLaVA and LLaVA-OneVision recipes. This converts the VLM into an **interactive assistant** capable of following complex multimodal instructions.

## Production Integration: RAG and Multimodal Agents

Beyond training, the curriculum emphasizes deployment through **skill artifacts** emitted by each lesson.

### Multimodal RAG Implementation

The repository demonstrates zero-shot classification using trained embeddings:

```python

# Zero‑shot classification using a CLIP checkpoint

def zero_shot(image_emb, class_names, tokenizer, text_encoder, temperature=0.07):
    # Build prompt templates

    prompts = [f"a photo of a {c}" for c in class_names]
    text_emb = text_encoder(tokenizer(prompts))           # (C, D)

    # Normalise embeddings

    image_emb = image_emb / np.linalg.norm(image_emb)
    text_emb  = text_emb / np.linalg.norm(text_emb, axis=1, keepdims=True)
    # Cosine similarity

    sims = image_emb @ text_emb.T / temperature
    return class_names[np.argmax(sims)]

```

These skills output **MD-formatted artifacts** (e.g., [`skill-clip-zero-shot.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/skill-clip-zero-shot.md)) that integrate into any downstream pipeline.

### Closed-Loop Agent Architecture

The final integration in Phase 14 creates a **multimodal agent skeleton** that queries CLIP skills, feeds results to a planner, and executes tool calls. Using the MCP protocol from `phases/13-tools-and-protocols/07-building-an-mcp-server`, agents stream vision, audio, and text between services.

## Summary

- The curriculum teaches **multimodal AI** through a progressive stack: Vision Transformers → CLIP → BLIP-2 → Flamingo → LLaVA → Agents.
- Students implement **InfoNCE loss**, **Q-Former bridges**, and **gated cross-attention** from scratch in `phases/12-multimodal-ai/`.
- Every lesson produces **reusable skill artifacts** compatible with production frameworks like Hugging Face and LangChain.
- The final capstone builds **closed-loop multimodal agents** that combine vision, language, and tool use in `phases/14-agent-engineering/`.

## Frequently Asked Questions

### What is the "Build-It / Use-It" philosophy in multimodal AI?

The "Build-It / Use-It" approach requires students to first implement algorithms using only the Python standard library, then re-implement them with production frameworks like PyTorch or JAX. For multimodal AI, this means writing the **InfoNCE loss** and **attention mechanisms** in pure NumPy before using Hugging Face transformers, ensuring deep understanding of the mathematics behind VLMs.

### How does the curriculum teach CLIP and contrastive learning?

The curriculum dedicates `phases/12-multimodal-ai/02-clip-contrastive-pretraining/` to CLIP, where students derive and implement the **InfoNCE loss** from mutual information principles. The lesson covers temperature scaling, dual-encoder architectures, and zero-shot classification, culminating in a skill artifact that can be dropped into any RAG system.

### What is the difference between BLIP-2 and Flamingo architectures in the curriculum?

**BLIP-2** uses a **Q-Former** bridge to transform frozen image embeddings into query vectors for LLMs, enabling efficient image-to-text generation. **Flamingo** instead employs **gated cross-attention layers** inserted into existing language models, allowing the LLM to attend to visual tokens while preserving its linguistic capabilities. The curriculum implements both in `phases/12-multimodal-ai/03-blip2-qformer-bridge/` and `phases/12-multimodal-ai/04-flamingo-gated-cross-attention/` respectively.

### How are multimodal agents implemented in the final phase?

Phase 14 (`phases/14-agent-engineering/25-multimodal-agents-computer-use/`) implements a **multimodal agent** that consumes image embeddings from CLIP, audio embeddings, and text prompts simultaneously. The agent uses a planner to route these inputs to appropriate tools, executing actions through MCP servers that stream vision and audio between services, creating a closed-loop system that can perceive and act upon the world.