# How Vision-Language Models (VLMs) like CLIP and LLaVA are Implemented from Scratch

> Learn how to implement Vision-Language Models (VLMs) like CLIP and LLaVA from scratch. Understand the architecture and training process for aligning visual and text data.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-06-22

---

**Vision-Language Models combine frozen Vision Transformers with text encoders via contrastive learning (CLIP) or lightweight projection layers (LLaVA) to align visual and linguistic representations in a shared embedding space.**

The rohitg00/ai-engineering-from-scratch repository provides a complete curriculum for building Vision-Language Models (VLMs) from first principles. This guide walks through the essential implementations of CLIP's contrastive pre-training and LLaVA's two-stage alignment pipeline, referencing the actual source files and runnable code examples from the multimodal AI phases.

## Vision Transformer Foundation: Patch Tokens and Encoders

Every modern VLM begins with a **Vision Transformer (ViT)** that converts images into sequence tokens. According to the curriculum in [`phases/12-multimodal-ai/01-vision-transformer-patch-tokens/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/01-vision-transformer-patch-tokens/docs/en.md), the implementation follows three strict steps.

### Patch Extraction and Linear Projection

An input image is divided into fixed-size patches (typically 16×16 pixels). Each patch is flattened and projected to a token vector via a trainable linear layer. A learnable `[CLS]` token is prepended to the sequence; its final hidden state serves as the global image embedding.

### The ViT Backbone Architecture

The token sequence passes through standard transformer blocks featuring pre-LayerNorm, GELU activations, and MLP sublayers. The repository notes that later lessons extend this basic recipe to support variable-resolution images through AnyRes, NaFlex, and M-RoPE techniques, but the core mechanism remains patch-level tokenization.

## CLIP: Contrastive Language-Image Pre-training

CLIP establishes a joint embedding space by training separate visual and text encoders with a contrastive objective. The implementation details in [`phases/12-multimodal-ai/02-clip-contrastive-pretraining/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/02-clip-contrastive-pretraining/docs/en.md) reveal the mathematical foundation.

### Dual Encoder Architecture

CLIP utilizes two frozen towers: a ViT visual encoder and a text transformer. For a batch of *N* image-caption pairs, the model computes an *N×N* similarity matrix **S = I·Tᵀ / τ**, where *I* represents image embeddings, *T* represents text embeddings, and *τ* is a learned temperature parameter initialized around 0.07.

### InfoNCE Loss Implementation

The training objective applies bidirectional cross-entropy. Each row (image-to-text) and each column (text-to-image) serves as separate softmax targets, with losses averaged across both directions. The temperature *τ* is optimized in log-space to prevent training collapse.

The repository provides a minimal implementation in [`phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py):

```python
import torch
import torch.nn.functional as F

def info_nce_loss(image_emb, text_emb, temperature=0.07):
    # L2-normalize embeddings

    image_emb = F.normalize(image_emb, dim=-1)
    text_emb  = F.normalize(text_emb, dim=-1)

    # Similarity matrix (N×N)

    logits = image_emb @ text_emb.t() / temperature

    # Targets are the diagonal (i == j)

    labels = torch.arange(logits.size(0), device=logits.device)

    # Row-wise cross-entropy

    loss_i2t = F.cross_entropy(logits, labels)
    # Column-wise cross-entropy (transpose)

    loss_t2i = F.cross_entropy(logits.t(), labels)

    return (loss_i2t + loss_t2i) / 2.0

```

## LLaVA: Aligning Visual Encoders with LLMs

While CLIP produces discriminative embeddings, LLaVA enables generative capabilities by bridging vision encoders with large language models. The architecture follows a two-stage pipeline documented in [`phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md).

### Stage 1: Projection Alignment

In the first stage, the CLIP visual encoder (ViT-L) remains frozen. The trainable component is a **2-layer MLP projector** (`Linear → GELU → Linear`) that maps CLIP image embeddings (e.g., 1024 dimensions) to the LLM's hidden dimension (e.g., 4096). The loss function is standard language-model cross-entropy on caption tokens, using the projected image embedding as a pseudo-token prefix.

The reference implementation in [`phases/12-multimodal-ai/05-llava-visual-instruction-tuning/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/05-llava-visual-instruction-tuning/code/main.py) defines the projector as:

```python
import torch.nn as nn

class LLaVAProjector(nn.Module):
    def __init__(self, vision_dim=1024, llm_dim=4096):
        super().__init__()
        self.proj = nn.Sequential(
            nn.Linear(vision_dim, llm_dim),
            nn.GELU(),
            nn.Linear(llm_dim, llm_dim)
        )

    def forward(self, vision_emb):
        # vision_emb: (B, vision_dim)

        return self.proj(vision_emb)  # (B, llm_dim)

```

### Stage 2: Visual Instruction Tuning

After projection alignment, the LLM (LLaMA, Vicuna, or similar) is unfrozen. The model trains on image-prompt pairs generated automatically from COCO captions using GPT-4. During this phase, the model learns to answer questions, generate descriptions, and follow complex instructions while conditioning on the visual tokens provided by the frozen encoder and trained projector.

## End-to-End Inference Pipeline

Combining these components yields a complete VLM inference flow. The following pseudo-code from the curriculum demonstrates the integration:

```python

# 1️⃣ Encode image → CLIP ViT-L embedding

img_emb = clip_encoder(image)          # shape (B, 1024)

# 2️⃣ Project into LLM space

proj_emb = projector(img_emb)          # shape (B, 4096)

# 3️⃣ Append as a pseudo-token to the LLM input

llm_input = torch.cat([proj_emb, llm_token_embeddings], dim=1)

# 4️⃣ Run LLM (frozen) to generate a caption / answer

output_ids = llm.generate(llm_input)

```

A full runnable demo appears in [`phases/12-multimodal-ai/05-llava-visual-instruction-tuning/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/05-llava-visual-instruction-tuning/code/main.py).

## Vision Encoder Selection and Projection Variants

The repository catalogs several vision encoder families and their VLM applications in [`phases/12-multimodal-ai/02-clip-contrastive-pretraining/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/02-clip-contrastive-pretraining/docs/en.md):

- **CLIP ViT-L/14**: Dominant for semantic similarity and zero-shot classification; used in classic CLIP and early VLMs.
- **SigLIP 2 (SO400m/14)**: Offers faster training and higher accuracy for contrastive pre-training; prevalent in 2025-2026 VLMs like Qwen-VL and InternViT.
- **DINOv2 ViT-g**: Provides dense visual features suited for segmentation and detection tasks requiring fine-grained perception.
- **ViT-MAE**: Pre-trained via masked image modeling, suitable for reconstruction-focused generation tasks.

### Projection Layer Designs

The canonical **two-layer MLP** serves as the baseline for LLaVA 1.5, BLIP-2's Q-Former, and many contemporary architectures. However, [`phases/12-multimodal-ai/60-projection-layer-modality-align/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/60-projection-layer-modality-align/docs/en.md) documents advanced alternatives including GLU variants and attention-based bridges used in LLaVA-NeXT and Qwen-VL. The choice between simple MLPs and complex bridges depends on the trade-off between parameter efficiency and alignment quality.

## Summary

- **Vision Transformers** convert images to tokens via patch extraction and a `[CLS]` token, forming the foundation of all VLMs.
- **CLIP** trains dual encoders with InfoNCE loss to create a shared embedding space where cosine similarity measures image-text relevance.
- **LLaVA** adds a lightweight 2-layer MLP projector that aligns frozen CLIP embeddings to frozen LLMs, followed by visual instruction tuning on GPT-4 generated data.
- The curriculum provides complete implementations in [`phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py) and [`phases/12-multimodal-ai/05-llava-visual-instruction-tuning/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/05-llava-visual-instruction-tuning/code/main.py).
- Encoder selection (CLIP, SigLIP 2, DINOv2, MAE) depends on whether the application requires semantic ranking, dense vision, or reconstruction capabilities.

## Frequently Asked Questions

### What is the difference between CLIP and LLaVA architectures?

CLIP uses a **contrastive** objective to align image and text embeddings in a shared space for retrieval and classification, keeping both encoders frozen after training. LLaVA uses a **generative** approach, adding a trainable projection layer to feed visual embeddings into a language model, enabling text generation conditioned on images.

### Why does LLaVA use a 2-layer MLP projector instead of a single linear layer?

According to [`phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md), the intermediate GELU activation and non-linear transformation in a 2-layer MLP provide sufficient capacity to transform the visual feature space (optimized for contrastive learning) into the linguistic feature space of the LLM without adding excessive parameters. A single linear layer often lacks the expressiveness to bridge this modality gap effectively.

### How does the InfoNCE temperature parameter affect CLIP training?

The temperature parameter *τ* scales the similarity logits in the InfoNCE loss. The repository notes it is initialized to approximately 0.07 and optimized in log-space. Lower temperatures sharpen the softmax distribution, making the model more confident in positive pairs but potentially causing collapse; higher temperatures soften the distribution. Learning *τ* automatically allows the model to find the optimal sharpness for the specific embedding space.

### Can I use DINOv2 instead of CLIP for the visual encoder in a LLaVA-style model?

Yes. While CLIP provides semantic embeddings optimized for image-text alignment, [`phases/12-multimodal-ai/02-clip-contrastive-pretraining/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/02-clip-contrastive-pretraining/docs/en.md) indicates that DINOv2 ViT-g offers superior dense visual features for tasks requiring fine-grained spatial understanding. However, using DINOv2 may require longer projection alignment training or a more complex bridge architecture, as its features are not pre-aligned with linguistic concepts.