Vision and Multimodal AI Topics in ai‑engineering‑from‑scratch: From Classic CNNs to Modern Vision‑Language Models

The rohitg00/ai‑engineering‑from‑scratch repository delivers a complete curriculum spanning classic CNNs (LeNet through ResNet), Vision Transformers, CLIP‑style contrastive learning, and production‑grade Vision‑Language Model implementations with self‑contained reference code.

This educational codebase walks through the full spectrum of vision and multimodal AI, organized into discrete phases and lessons that each pair architectural theory with runnable Python implementations. Whether you are implementing a 1998‑style LeNet‑5 or a modern ViT‑MLP‑LLM stack, the repository provides specific file paths, exact method signatures, and unit tests to validate your understanding.

Classic Convolutional Neural Networks (CNNs)

The foundational computer vision phase covers the evolutionary arc of convolutional architectures, from the earliest fully‑connected conv‑pool pipelines to residual networks.

From LeNet to ResNet

Lesson 03‑cnns‑lenet‑to‑resnet tracks the historical progression of CNNs through five landmark architectures. According to phases/04-computer-vision/03-cnns-lenet-to-resnet/docs/en.md, you will implement:

• LeNet‑5 – 2 conv layers with tanh activations followed by average pooling and fully‑connected stacks.
• AlexNet – ReLU non‑linearities, dropout regularization, and GPU‑split training patterns.
• VGG – Strictly 3×3 convolutional stacks demonstrating how depth compensates for kernel size.
• Inception – Parallel kernels of varying sizes capturing multi‑scale features.
• ResNet – Identity skip connections that solve vanishing gradients in deep stacks.

The accompanying main.py in phases/04-computer-vision/03-cnns-lenet-to-resnet/code/ contains sub‑200‑line implementations of each block, including ResNetBasicBlock with its conv‑norm‑relu‑skip pattern.

Detection, Segmentation, and Tracking

Beyond classification, the curriculum extends CNNs into structured prediction tasks:

• YOLO (Lesson 06) – One‑stage object detection with anchor‑free bounding‑box regression and focal loss, detailed in phases/04-computer-vision/06-object-detection-yolo/docs/en.md.
• U‑Net (Lesson 07) – Encoder‑decoder architecture with skip connections for semantic segmentation, located at phases/04-computer-vision/07-semantic-segmentation-unet/docs/en.md.
• Mask R‑CNN (Lesson 08) – Two‑stage detector adding a mask head atop RoIAlign, implemented in phases/04-computer-vision/08-instance-segmentation-mask-rcnn/docs/en.md.
• DeepSORT (Lesson 27) – Multi‑object tracking that couples CNN‑based ReID embeddings with Kalman filtering, described in phases/04-computer-vision/27-multi-object-tracking/docs/en.md.

Advanced Vision: 3D and Video

The repository also covers neural rendering and generative video models:

• 3D Gaussian Splatting (Lesson 22) – Differentiable rasterization of point clouds using view‑dependent 3‑D Gaussians (phases/04-computer-vision/22-3d-gaussian-splatting/docs/en.md).
• Video Diffusion (Lesson 28) – Latent diffusion models with temporal attention for frame prediction, found in phases/04-computer-vision/28-world-models-video-diffusion/docs/en.md.

Vision Transformers (ViT)

Moving beyond convolutions, the curriculum introduces the Vision Transformer paradigm where images are treated as sequences of patches.

Lesson 14‑vision‑transformers in phases/04-computer-vision/14-vision-transformers/docs/en.md implements the canonical ViT: a 2‑D image is split into patch embeddings via a Conv2d projection, prepended with a class token, and processed through a standard Transformer encoder with self‑attention and positional embeddings.

For advanced variants, 09‑vision‑transformers in phases/07-transformers-deep-dive/09‑vision‑transformers/docs/en.md discusses multi‑scale architectures like Swin and DINOv2, analyzing patch‑size trade‑offs and scaling laws relevant to large‑scale pre‑training.

Multimodal Foundations – CLIP and Contrastive Pre‑training

The bridge between vision and language begins with CLIP‑style contrastive learning.

Lesson 02‑clip‑contrastive‑pretraining (phases/12-multimodal-ai/02-clip-contrastive-pretraining/docs/en.md) demonstrates how to align image and text embeddings using InfoNCE loss with temperature scaling. The code includes the dual‑encoder setup: a Vision Transformer processes images while a Transformer encoder (or text‑only Transformer) processes tokenized captions, with both embeddings projected into a shared latent space.

Supporting infrastructure for multimodal architectures includes:

• 58‑vision‑encoder‑patches – Patch‑embedding front‑ends (stride‑based tokenization) common to CLIP‑ViT and SigLIP, located in phases/19-capstone-projects/58-vision-encoder-patches/docs/en.md.
• 60‑projection‑layer‑modality‑align – Small MLP projector (or Q‑former) that maps vision token dimensions to LLM embedding spaces using cosine‑alignment objectives (phases/19-capstone-projects/60-projection-layer-modality-align/docs/en.md).
• 61‑cross‑attention‑fusion – Cross‑attention layers allowing text tokens to attend to image tokens, critical for VLM decoders (phases/19-capstone-projects/61-cross-attention-fusion/docs/en.md).

Vision‑Language Models (VLMs) – The ViT‑MLP‑LLM Pattern

The flagship Vision‑Language Model lesson is 25‑vision‑language‑models (phases/04-computer-vision/25-vision-language-models/docs/en.md). It defines the canonical three‑component architecture:

1. Vision Encoder – Pre‑trained ViT (CLIP‑L/14, SigLIP, or DINOv3) extracting patch tokens.
2. Projector – 2‑4‑layer MLP mapping vit_dim (e.g., 768) to llm_dim (e.g., 4096).
3. LLM – Decoder‑only language model (Qwen‑3‑VL, LLaVA‑Next, etc.) consuming projected visual tokens.

The lesson introduces DeepStack (lines 53‑56), a technique that extracts features from multiple ViT depths to improve spatial grounding. A comparison table (lines 67‑79) catalogs open‑source VLM families by parameter count, context window, and benchmark performance.

Production‑Ready VLM Code

The repository provides a complete MinimalVLM implementation:

import torch
import torch.nn as nn

class Projector(nn.Module):
    def __init__(self, vit_dim=768, llm_dim=4096, hidden=4096):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(vit_dim, hidden),
            nn.GELU(),
            nn.Linear(hidden, llm_dim),
        )

    def forward(self, x):
        return self.net(x)          # (B, N_patches, llm_dim)

class MinimalVLM(nn.Module):
    def __init__(self, vit, projector, llm, image_token_id):
        super().__init__()
        self.vit = vit
        self.projector = projector
        self.llm = llm
        self.image_token_id = image_token_id

    def forward(self, image, input_ids, attention_mask):
        vision_tokens = self.vit(image)               # (B, N, d_vit)

        vision_embeds = self.projector(vision_tokens)  # (B, N, d_llm)

        text_embeds = self.llm.get_input_embeddings()(input_ids)
        merged = self._merge(text_embeds, vision_embeds, input_ids)
        return self.llm(inputs_embeds=merged, attention_mask=attention_mask)

    def _merge(self, text_embeds, vision_embeds, input_ids):
        out = text_embeds.clone()
        expected = vision_embeds.size(1)
        for b in range(input_ids.size(0)):
            pos = (input_ids[b] == self.image_token_id).nonzero(as_tuple=True)[0]
            out[b, pos] = vision_embeds[b]
        return out

Source: Lines 30‑71 of phases/04-computer-vision/25-vision-language-models/docs/en.md.

The same file also provides the CMER (Cross‑Modal Error Rate) metric for monitoring VLM hallucinations in production:

import torch.nn.functional as F

def cross_modal_error_rate(image_emb, text_emb, text_confidence,
                          sim_threshold=0.25, conf_threshold=0.8):
    image_emb = F.normalize(image_emb, dim=-1)
    text_emb  = F.normalize(text_emb, dim=-1)
    sim = (image_emb * text_emb).sum(dim=-1)
    high_conf_low_sim = (text_confidence > conf_threshold) & (sim < sim_threshold)
    return high_conf_low_sim.float().mean().item()

Source: Lines 84‑95 of phases/04-computer-vision/25-vision-language-models/docs/en.md.

Multimodal Capstone Projects

The curriculum culminates in end‑to‑end projects assembling the above components:

• 62‑vision‑language‑pretraining – Full VLM pre‑training script with ranking and generation objectives, including data‑mixing strategies (phases/19-capstone-projects/62-vision-language-pretraining/docs/en.md).
• 04‑multimodal‑document‑qa – Retrieval‑augmented pipeline for OCR‑aware question answering over scanned documents using Qwen‑VL‑OCR and InternVL (phases/12-multimodal-ai/04-multimodal-document-qa/docs/en.md).
• 21‑embodied‑vlas‑openvla‑pi0‑groot – Vision‑Language‑Action models for robot control, extending the VLM encoder with an action prediction head for RT‑2 and OpenVLA architectures (phases/12-multimodal-ai/21-embodied-vlas-openvla-pi0-groot/docs/en.md).

Summary

• The ai‑engineering‑from‑scratch curriculum spans from 1990s CNNs (LeNet‑5) to modern Vision‑Language Models, with each lesson containing reference implementations under 200 lines.
• Classic CNNs (Lessons 03, 04, 06‑08) cover LeNet through Mask R‑CNN, including YOLO for detection and U‑Net for segmentation.
• Vision Transformers (Lessons 14, 09) explain patch‑embedding encoders and scaling laws, forming the backbone for modern VLMs.
• Multimodal foundations (Lessons 02, 58‑61) detail CLIP contrastive pre‑training, projection layers, and cross‑attention fusion mechanisms.
• Vision‑Language Models (Lesson 25) implement the canonical ViT → MLP → LLM stack with the DeepStack technique for improved grounding and a production CMER metric.
• Capstone projects (Lessons 62, 04, 21) provide end‑to‑end pipelines for VLM pre‑training, document QA, and embodied robot control.

Frequently Asked Questions

What CNN architectures are covered in the curriculum?

The repository tracks the complete evolution of CNNs from LeNet‑5 (1998) through AlexNet, VGG, Inception, and ResNet, each implemented in phases/04-computer-vision/03-cnns-lenet-to-resnet/code/main.py. Lessons also extend to YOLO for detection, U‑Net for segmentation, and Mask R‑CNN for instance segmentation with mask heads.

How does the repository explain Vision‑Language Model architectures?

Lesson 25‑vision‑language‑models documents the canonical ViT → MLP → LLM pattern: a pretrained Vision Transformer extracts patch tokens, a projector MLP aligns dimensions to the LLM’s embedding space, and a decoder‑only language model generates text. The lesson includes training stages (alignment, pre‑training, instruction tuning) and the DeepStack technique for multi‑scale feature extraction.

What is the DeepStack technique mentioned in the VLM lesson?

DeepStack (lines 53‑56 of phases/04-computer-vision/25-vision-language-models/docs/en.md) is a feature aggregation method that stacks intermediate representations from multiple depths of the Vision Transformer to improve spatial grounding in Vision‑Language Models, particularly for fine‑grained visual question answering.

Are there runnable code examples or only theoretical explanations?

Every lesson includes a self‑contained main.py implementation (typically under 200 lines) along with unit tests in code/tests/. For example, the CNN lesson provides runnable LeNet‑5 and ResNet blocks, while the VLM lesson includes the complete MinimalVLM class, Projector implementation, and the CMER monitoring metric.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}