How the AI Engineering Curriculum Teaches Multimodal AI with Vision Transformers and VLMs

The curriculum builds a complete multimodal AI stack by teaching the three-component ViT-Projector-LLM pattern used in production Vision-Language Models, covering everything from patch tokenization to unified single-image/multi-image/video architectures like LLaVA-OneVision.

The rohitg00/ai-engineering-from-scratch repository provides a systematic, production-oriented approach to mastering multimodal AI with Vision Transformers and VLMs. By progressing from foundational image tokenization to advanced unified multimodal systems, learners implement the exact architectural patterns driving modern AI applications. The curriculum emphasizes hands-on code implementations that mirror the training pipelines and inference optimizations found in deployed systems.

The Three-Component VLM Architecture

Every production Vision-Language Model in the curriculum follows a standardized three-component pattern implemented in phases/04-computer-vision/25-vision-language-models/docs/en.md. This architecture consists of a Vision Transformer (ViT) encoder that converts images into patch-tokens, a projector (MLP or Q-Former) that aligns visual embeddings with the language model's space, and a decoder-only LLM that processes the mixed token stream.

The MinimalVLM class demonstrates how these components connect during the forward pass:

class MinimalVLM(nn.Module):
    def __init__(self, vit, projector, llm, image_token_id):
        super().__init__()
        self.vit = vit
        self.projector = projector
        self.llm = llm
        self.image_token_id = image_token_id

    def forward(self, image, input_ids, attention_mask):
        vision_tokens = self.vit(image)                     # (B, N, d_vit)

        vision_embeds = self.projector(vision_tokens)       # (B, N, d_llm)

        text_embeds = self.llm.get_input_embeddings()(input_ids)  # (B, M, d_llm)

        merged = self._merge(text_embeds, vision_embeds, input_ids)
        return self.llm(inputs_embeds=merged, attention_mask=attention_mask)

    def _merge(self, text_embeds, vision_embeds, input_ids):
        out = text_embeds.clone()
        expected = vision_embeds.size(1)
        for b in range(input_ids.size(0)):
            positions = (input_ids[b] == self.image_token_id).nonzero(as_tuple=True)[0]
            if len(positions) != expected:
                raise ValueError(
                    f"batch {b} has {len(positions)} image tokens but vision_embeds has {expected} patches.")
            out[b, positions] = vision_embeds[b]
        return out

This implementation shows how vision embeddings replace special image token placeholders in the text embedding matrix, enabling the LLM to process both modalities within its existing context window.

The Complete Multimodal AI Curriculum Pipeline

The curriculum structures multimodal AI education across five distinct stages, each targeting specific competencies required for building production VLMs.

Foundations - Vision Transformers (ViT)

Located in phases/07-transformers-deep-dive/09-vision-transformers/docs/en.md, this module establishes the "patch-token primitive" that makes multimodal AI possible. Learners implement how 2-D images are tokenized exactly like sentences through the patchify function:

def patchify(image, P):
    H = len(image)
    W = len(image[0])
    patches = []
    for i in range(0, H, P):
        for j in range(0, W, P):
            patch = []
            for di in range(P):
                for dj in range(P):
                    patch.extend(image[i + di][j + dj])
            patches.append(patch)
    return patches

The lesson covers patchify, linear embedding, CLS tokens, and the full transformer encoder—demonstrating how a pure-transformer architecture processes visual information without convolutional layers.

Contrastive Pre-training - CLIP

The curriculum teaches joint image-text representation learning through contrastive loss methods documented in the capstone materials. This stage explains how aligned vision-text encoders become the frozen backbones for downstream VLMs. The contrastive pre-training approach enables zero-shot capabilities and establishes the embedding space that later projector layers will navigate.

Vision-Language Integration

The phases/04-computer-vision/25-vision-language-models/docs/en.md file details the complete connection between ViT encoder, projector, and LLM. Key concepts include DeepStack (utilizing multi-level ViT features) and the three-stage training pipeline: alignment, pre-training, and instruction-tuning. This module explains why most production systems keep both the ViT encoder and LLM frozen during fine-tuning, updating only the projector parameters.

Unified Multimodal Architecture - LLaVA-OneVision

The phases/12-multimodal-ai/08-llava-onevision-single-multi-video/docs/en.md lesson introduces a unified model handling single-image, multi-image, and video inputs through a constant visual-token budget of approximately 3,000–4,000 tokens. The curriculum demonstrates how a fixed token budget across modalities allows one model to dominate specialist models while preserving LLM context limits.

The token-budget planner illustrates this optimization strategy:

# given a target mix (e.g. 40% single‑image, 30% multi‑image, 30% video)

# allocate resolution & pooling so each scenario respects ~3‑4k visual tokens

def plan_budget(mix):
    budget = 3500
    # simple heuristic: use higher resolution for single‑image, pool more for video

    single_res = 384          # ≈ 3000 tokens per image

    multi_res  = 384          # ≈ 600 tokens per image × 5 images ≈ 3000

    video_frames = 16
    video_pool = 2            # 2×2 bilinear pool → ~200 tokens per frame

    return {
        "single": {"res": single_res, "tokens": budget},
        "multi":  {"res": multi_res,  "images": 5, "tokens": budget},
        "video":  {"frames": video_frames, "pool": video_pool, "tokens": budget},
    }

This approach yields emergent capabilities including multi-camera reasoning, set-of-mark prompting, and iPhone-screenshot agent behaviors through a curriculum ordering of single-image → multi-image → video.

Capstone Application

The phases/19-capstone-projects/62-vision-language-pretraining/docs/en.md capstone provides a comprehensive hands-on project integrating the encoder, projector, and LLM into a real-world VLM pipeline. Learners implement the full lifecycle from data preprocessing through contrastive pre-training to instruction-tuning, producing a deployable multimodal system.

Key Architectural Takeaways from the Source Code

The curriculum emphasizes four critical design principles evident in the source files:

• ViT as a universal modality tokenizer – By treating images as token sequences in phases/07-transformers-deep-dive/09-vision-transformers/docs/en.md, the same transformer architecture ingests video frames, audio spectrograms, or robot action tokens (explored in Phase 12's multimodal RAG lessons).

• Fixed visual-token budget – LLaVA-OneVision in phases/12-multimodal-ai/08-llava-onevision-single-multi-video/docs/en.md proves that maintaining ~3,500 tokens across single-image, multi-image, and video scenarios lets a single unified model outperform specialist models.

• Curriculum ordering matters – Training first on high-resolution single images builds a strong perceptual base; subsequent stages add compositional and temporal reasoning without degrading visual fidelity.

• Projector-centric fine-tuning – As implemented in phases/04-computer-vision/25-vision-language-models/docs/en.md, production VLMs freeze the ViT encoder and LLM, updating only the projector via LoRA or QLoRA. This approach dramatically reduces compute requirements while achieving state-of-the-art multimodal performance.

Summary

• The ViT-Projector-LLM pattern forms the architectural backbone of all production-ready Vision-Language Models taught in the curriculum.
• The curriculum progresses systematically from patch tokenization fundamentals to unified multimodal architectures handling diverse input types.
• LLaVA-OneVision demonstrates that constant visual-token budgets across modalities enable superior performance compared to specialist models.
• Projector-only fine-tuning via LoRA/QLoRA provides an efficient pathway to customize VLMs without retraining billion-parameter encoders or language models.

Frequently Asked Questions

What is the three-component architecture pattern for Vision-Language Models taught in the curriculum?

The curriculum teaches a standardized architecture consisting of a Vision Transformer (ViT) encoder that generates patch-tokens from images, a projector (MLP or Q-Former) that maps these tokens into the language model's embedding space, and a decoder-only LLM that processes the merged visual and text tokens. This pattern, fully implemented in phases/04-computer-vision/25-vision-language-models/docs/en.md, underlies every major production VLM.

How does the curriculum address different visual input types like single-image, multi-image, and video?

Through the LLaVA-OneVision module in phases/12-multimodal-ai/08-llava-onevision-single-multi-video/docs/en.md, the curriculum implements a unified architecture using a fixed visual-token budget of approximately 3,500 tokens regardless of input type. The system allocates higher resolution for single images while applying pooling strategies for video frames, enabling multi-camera reasoning and video understanding within consistent computational constraints.

Why does the curriculum emphasize fine-tuning only the projector component in multimodal AI systems?

According to the implementation in phases/04-computer-vision/25-vision-language-models/docs/en.md, freezing the ViT encoder and LLM while updating only the projector via LoRA or QLoRA dramatically reduces computational requirements. This approach preserves the pre-trained knowledge in the vision and language components while efficiently adapting the bridge between modalities, achieving state-of-the-art results with minimal trainable parameters.

What is the visual-token budget and why is it critical for multimodal AI performance?

The visual-token budget refers to the fixed number of tokens (typically 3,000–4,000) allocated to represent visual inputs within the LLM's context window. As demonstrated in phases/12-multimodal-ai/08-llava-onevision-single-multi-video/docs/en.md, maintaining this budget constant across single-images, multi-images, and video ensures predictable memory usage and leaves sufficient context capacity for text instructions. This constraint forces efficient representation learning and enables a single model to handle diverse scenarios effectively.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}