# Vision and Multimodal AI Topics in ai‑engineering‑from‑scratch: From Classic CNNs to Modern Vision‑Language Models

> Explore vision and multimodal AI topics in ai-engineering-from-scratch, from classic CNNs like LeNet and ResNet to modern Vision Transformers and CLIP-style models. Learn practical implementations.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-06-23

---

**The rohitg00/ai‑engineering‑from‑scratch repository delivers a complete curriculum spanning classic CNNs (LeNet through ResNet), Vision Transformers, CLIP‑style contrastive learning, and production‑grade Vision‑Language Model implementations with self‑contained reference code.**

This educational codebase walks through the full spectrum of vision and multimodal AI, organized into discrete phases and lessons that each pair architectural theory with runnable Python implementations. Whether you are implementing a 1998‑style LeNet‑5 or a modern ViT‑MLP‑LLM stack, the repository provides specific file paths, exact method signatures, and unit tests to validate your understanding.

## Classic Convolutional Neural Networks (CNNs)

The foundational computer vision phase covers the evolutionary arc of convolutional architectures, from the earliest fully‑connected conv‑pool pipelines to residual networks.

### From LeNet to ResNet

Lesson **`03‑cnns‑lenet‑to‑resnet`** tracks the historical progression of CNNs through five landmark architectures. According to [`phases/04-computer-vision/03-cnns-lenet-to-resnet/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/03-cnns-lenet-to-resnet/docs/en.md), you will implement:

* **LeNet‑5** – 2 conv layers with `tanh` activations followed by average pooling and fully‑connected stacks.
* **AlexNet** – ReLU non‑linearities, dropout regularization, and GPU‑split training patterns.
* **VGG** – Strictly 3×3 convolutional stacks demonstrating how depth compensates for kernel size.
* **Inception** – Parallel kernels of varying sizes capturing multi‑scale features.
* **ResNet** – Identity skip connections that solve vanishing gradients in deep stacks.

The accompanying [`main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/main.py) in `phases/04-computer-vision/03-cnns-lenet-to-resnet/code/` contains sub‑200‑line implementations of each block, including `ResNetBasicBlock` with its conv‑norm‑relu‑skip pattern.

### Detection, Segmentation, and Tracking

Beyond classification, the curriculum extends CNNs into structured prediction tasks:

* **YOLO (Lesson 06)** – One‑stage object detection with anchor‑free bounding‑box regression and focal loss, detailed in [`phases/04-computer-vision/06-object-detection-yolo/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/06-object-detection-yolo/docs/en.md).
* **U‑Net (Lesson 07)** – Encoder‑decoder architecture with skip connections for semantic segmentation, located at [`phases/04-computer-vision/07-semantic-segmentation-unet/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/07-semantic-segmentation-unet/docs/en.md).
* **Mask R‑CNN (Lesson 08)** – Two‑stage detector adding a mask head atop RoIAlign, implemented in [`phases/04-computer-vision/08-instance-segmentation-mask-rcnn/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/08-instance-segmentation-mask-rcnn/docs/en.md).
* **DeepSORT (Lesson 27)** – Multi‑object tracking that couples CNN‑based ReID embeddings with Kalman filtering, described in [`phases/04-computer-vision/27-multi-object-tracking/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/27-multi-object-tracking/docs/en.md).

### Advanced Vision: 3D and Video

The repository also covers neural rendering and generative video models:

* **3D Gaussian Splatting (Lesson 22)** – Differentiable rasterization of point clouds using view‑dependent 3‑D Gaussians ([`phases/04-computer-vision/22-3d-gaussian-splatting/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/22-3d-gaussian-splatting/docs/en.md)).
* **Video Diffusion (Lesson 28)** – Latent diffusion models with temporal attention for frame prediction, found in [`phases/04-computer-vision/28-world-models-video-diffusion/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/28-world-models-video-diffusion/docs/en.md).

## Vision Transformers (ViT)

Moving beyond convolutions, the curriculum introduces the **Vision Transformer** paradigm where images are treated as sequences of patches.

Lesson **`14‑vision‑transformers`** in [`phases/04-computer-vision/14-vision-transformers/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/14-vision-transformers/docs/en.md) implements the canonical ViT: a 2‑D image is split into patch embeddings via a `Conv2d` projection, prepended with a class token, and processed through a standard Transformer encoder with self‑attention and positional embeddings.

For advanced variants, **`09‑vision‑transformers`** in `phases/07-transformers-deep-dive/09‑vision‑transformers/docs/en.md` discusses multi‑scale architectures like **Swin** and **DINOv2**, analyzing patch‑size trade‑offs and scaling laws relevant to large‑scale pre‑training.

## Multimodal Foundations – CLIP and Contrastive Pre‑training

The bridge between vision and language begins with **CLIP‑style contrastive learning**.

Lesson **`02‑clip‑contrastive‑pretraining`** ([`phases/12-multimodal-ai/02-clip-contrastive-pretraining/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/02-clip-contrastive-pretraining/docs/en.md)) demonstrates how to align image and text embeddings using **InfoNCE loss** with temperature scaling. The code includes the dual‑encoder setup: a Vision Transformer processes images while a Transformer encoder (or text‑only Transformer) processes tokenized captions, with both embeddings projected into a shared latent space.

Supporting infrastructure for multimodal architectures includes:

* **`58‑vision‑encoder‑patches`** – Patch‑embedding front‑ends (stride‑based tokenization) common to CLIP‑ViT and SigLIP, located in [`phases/19-capstone-projects/58-vision-encoder-patches/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/58-vision-encoder-patches/docs/en.md).
* **`60‑projection‑layer‑modality‑align`** – Small MLP projector (or Q‑former) that maps vision token dimensions to LLM embedding spaces using cosine‑alignment objectives ([`phases/19-capstone-projects/60-projection-layer-modality-align/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/60-projection-layer-modality-align/docs/en.md)).
* **`61‑cross‑attention‑fusion`** – Cross‑attention layers allowing text tokens to attend to image tokens, critical for VLM decoders ([`phases/19-capstone-projects/61-cross-attention-fusion/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/61-cross-attention-fusion/docs/en.md)).

## Vision‑Language Models (VLMs) – The ViT‑MLP‑LLM Pattern

The flagship **Vision‑Language Model** lesson is **`25‑vision‑language‑models`** ([`phases/04-computer-vision/25-vision-language-models/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/25-vision-language-models/docs/en.md)). It defines the canonical three‑component architecture:

1. **Vision Encoder** – Pre‑trained ViT (CLIP‑L/14, SigLIP, or DINOv3) extracting patch tokens.
2. **Projector** – 2‑4‑layer MLP mapping `vit_dim` (e.g., 768) to `llm_dim` (e.g., 4096).
3. **LLM** – Decoder‑only language model (Qwen‑3‑VL, LLaVA‑Next, etc.) consuming projected visual tokens.

The lesson introduces **DeepStack** (lines 53‑56), a technique that extracts features from multiple ViT depths to improve spatial grounding. A comparison table (lines 67‑79) catalogs open‑source VLM families by parameter count, context window, and benchmark performance.

### Production‑Ready VLM Code

The repository provides a complete **MinimalVLM** implementation:

```python
import torch
import torch.nn as nn

class Projector(nn.Module):
    def __init__(self, vit_dim=768, llm_dim=4096, hidden=4096):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(vit_dim, hidden),
            nn.GELU(),
            nn.Linear(hidden, llm_dim),
        )

    def forward(self, x):
        return self.net(x)          # (B, N_patches, llm_dim)

class MinimalVLM(nn.Module):
    def __init__(self, vit, projector, llm, image_token_id):
        super().__init__()
        self.vit = vit
        self.projector = projector
        self.llm = llm
        self.image_token_id = image_token_id

    def forward(self, image, input_ids, attention_mask):
        vision_tokens = self.vit(image)               # (B, N, d_vit)

        vision_embeds = self.projector(vision_tokens)  # (B, N, d_llm)

        text_embeds = self.llm.get_input_embeddings()(input_ids)
        merged = self._merge(text_embeds, vision_embeds, input_ids)
        return self.llm(inputs_embeds=merged, attention_mask=attention_mask)

    def _merge(self, text_embeds, vision_embeds, input_ids):
        out = text_embeds.clone()
        expected = vision_embeds.size(1)
        for b in range(input_ids.size(0)):
            pos = (input_ids[b] == self.image_token_id).nonzero(as_tuple=True)[0]
            out[b, pos] = vision_embeds[b]
        return out

```

*Source:* Lines 30‑71 of [`phases/04-computer-vision/25-vision-language-models/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/25-vision-language-models/docs/en.md).

The same file also provides the **CMER (Cross‑Modal Error Rate)** metric for monitoring VLM hallucinations in production:

```python
import torch.nn.functional as F

def cross_modal_error_rate(image_emb, text_emb, text_confidence,
                          sim_threshold=0.25, conf_threshold=0.8):
    image_emb = F.normalize(image_emb, dim=-1)
    text_emb  = F.normalize(text_emb, dim=-1)
    sim = (image_emb * text_emb).sum(dim=-1)
    high_conf_low_sim = (text_confidence > conf_threshold) & (sim < sim_threshold)
    return high_conf_low_sim.float().mean().item()

```

*Source:* Lines 84‑95 of [`phases/04-computer-vision/25-vision-language-models/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/25-vision-language-models/docs/en.md).

## Multimodal Capstone Projects

The curriculum culminates in end‑to‑end projects assembling the above components:

* **`62‑vision‑language‑pretraining`** – Full VLM pre‑training script with ranking and generation objectives, including data‑mixing strategies ([`phases/19-capstone-projects/62-vision-language-pretraining/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/62-vision-language-pretraining/docs/en.md)).
* **`04‑multimodal‑document‑qa`** – Retrieval‑augmented pipeline for OCR‑aware question answering over scanned documents using Qwen‑VL‑OCR and InternVL ([`phases/12-multimodal-ai/04-multimodal-document-qa/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/04-multimodal-document-qa/docs/en.md)).
* **`21‑embodied‑vlas‑openvla‑pi0‑groot`** – Vision‑Language‑Action models for robot control, extending the VLM encoder with an action prediction head for RT‑2 and OpenVLA architectures ([`phases/12-multimodal-ai/21-embodied-vlas-openvla-pi0-groot/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/21-embodied-vlas-openvla-pi0-groot/docs/en.md)).

## Summary

* The **ai‑engineering‑from‑scratch** curriculum spans from 1990s CNNs (LeNet‑5) to modern Vision‑Language Models, with each lesson containing reference implementations under 200 lines.
* **Classic CNNs** (Lessons 03, 04, 06‑08) cover LeNet through Mask R‑CNN, including YOLO for detection and U‑Net for segmentation.
* **Vision Transformers** (Lessons 14, 09) explain patch‑embedding encoders and scaling laws, forming the backbone for modern VLMs.
* **Multimodal foundations** (Lessons 02, 58‑61) detail CLIP contrastive pre‑training, projection layers, and cross‑attention fusion mechanisms.
* **Vision‑Language Models** (Lesson 25) implement the canonical ViT → MLP → LLM stack with the DeepStack technique for improved grounding and a production CMER metric.
* **Capstone projects** (Lessons 62, 04, 21) provide end‑to‑end pipelines for VLM pre‑training, document QA, and embodied robot control.

## Frequently Asked Questions

### What CNN architectures are covered in the curriculum?

The repository tracks the complete evolution of CNNs from **LeNet‑5** (1998) through **AlexNet**, **VGG**, **Inception**, and **ResNet**, each implemented in [`phases/04-computer-vision/03-cnns-lenet-to-resnet/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/03-cnns-lenet-to-resnet/code/main.py). Lessons also extend to **YOLO** for detection, **U‑Net** for segmentation, and **Mask R‑CNN** for instance segmentation with mask heads.

### How does the repository explain Vision‑Language Model architectures?

Lesson **25‑vision‑language‑models** documents the canonical **ViT → MLP → LLM** pattern: a pretrained Vision Transformer extracts patch tokens, a projector MLP aligns dimensions to the LLM’s embedding space, and a decoder‑only language model generates text. The lesson includes training stages (alignment, pre‑training, instruction tuning) and the **DeepStack** technique for multi‑scale feature extraction.

### What is the DeepStack technique mentioned in the VLM lesson?

**DeepStack** (lines 53‑56 of [`phases/04-computer-vision/25-vision-language-models/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/25-vision-language-models/docs/en.md)) is a feature aggregation method that stacks intermediate representations from multiple depths of the Vision Transformer to improve spatial grounding in Vision‑Language Models, particularly for fine‑grained visual question answering.

### Are there runnable code examples or only theoretical explanations?

Every lesson includes a self‑contained [`main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/main.py) implementation (typically under 200 lines) along with unit tests in `code/tests/`. For example, the CNN lesson provides runnable LeNet‑5 and ResNet blocks, while the VLM lesson includes the complete `MinimalVLM` class, `Projector` implementation, and the **CMER** monitoring metric.