How Vision-Language Models (VLMs) like CLIP and LLaVA are Implemented from Scratch

Vision-Language Models combine frozen Vision Transformers with text encoders via contrastive learning (CLIP) or lightweight projection layers (LLaVA) to align visual and linguistic representations in a shared embedding space.

The rohitg00/ai-engineering-from-scratch repository provides a complete curriculum for building Vision-Language Models (VLMs) from first principles. This guide walks through the essential implementations of CLIP's contrastive pre-training and LLaVA's two-stage alignment pipeline, referencing the actual source files and runnable code examples from the multimodal AI phases.

Vision Transformer Foundation: Patch Tokens and Encoders

Every modern VLM begins with a Vision Transformer (ViT) that converts images into sequence tokens. According to the curriculum in phases/12-multimodal-ai/01-vision-transformer-patch-tokens/docs/en.md, the implementation follows three strict steps.

Patch Extraction and Linear Projection

An input image is divided into fixed-size patches (typically 16×16 pixels). Each patch is flattened and projected to a token vector via a trainable linear layer. A learnable [CLS] token is prepended to the sequence; its final hidden state serves as the global image embedding.

The ViT Backbone Architecture

The token sequence passes through standard transformer blocks featuring pre-LayerNorm, GELU activations, and MLP sublayers. The repository notes that later lessons extend this basic recipe to support variable-resolution images through AnyRes, NaFlex, and M-RoPE techniques, but the core mechanism remains patch-level tokenization.

CLIP: Contrastive Language-Image Pre-training

CLIP establishes a joint embedding space by training separate visual and text encoders with a contrastive objective. The implementation details in phases/12-multimodal-ai/02-clip-contrastive-pretraining/docs/en.md reveal the mathematical foundation.

Dual Encoder Architecture

CLIP utilizes two frozen towers: a ViT visual encoder and a text transformer. For a batch of N image-caption pairs, the model computes an N×N similarity matrix S = I·Tᵀ / τ, where I represents image embeddings, T represents text embeddings, and τ is a learned temperature parameter initialized around 0.07.

InfoNCE Loss Implementation

The training objective applies bidirectional cross-entropy. Each row (image-to-text) and each column (text-to-image) serves as separate softmax targets, with losses averaged across both directions. The temperature τ is optimized in log-space to prevent training collapse.

The repository provides a minimal implementation in phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py:

import torch
import torch.nn.functional as F

def info_nce_loss(image_emb, text_emb, temperature=0.07):
    # L2-normalize embeddings

    image_emb = F.normalize(image_emb, dim=-1)
    text_emb  = F.normalize(text_emb, dim=-1)

    # Similarity matrix (N×N)

    logits = image_emb @ text_emb.t() / temperature

    # Targets are the diagonal (i == j)

    labels = torch.arange(logits.size(0), device=logits.device)

    # Row-wise cross-entropy

    loss_i2t = F.cross_entropy(logits, labels)
    # Column-wise cross-entropy (transpose)

    loss_t2i = F.cross_entropy(logits.t(), labels)

    return (loss_i2t + loss_t2i) / 2.0

LLaVA: Aligning Visual Encoders with LLMs

While CLIP produces discriminative embeddings, LLaVA enables generative capabilities by bridging vision encoders with large language models. The architecture follows a two-stage pipeline documented in phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md.

Stage 1: Projection Alignment

In the first stage, the CLIP visual encoder (ViT-L) remains frozen. The trainable component is a 2-layer MLP projector (Linear → GELU → Linear) that maps CLIP image embeddings (e.g., 1024 dimensions) to the LLM's hidden dimension (e.g., 4096). The loss function is standard language-model cross-entropy on caption tokens, using the projected image embedding as a pseudo-token prefix.

The reference implementation in phases/12-multimodal-ai/05-llava-visual-instruction-tuning/code/main.py defines the projector as:

import torch.nn as nn

class LLaVAProjector(nn.Module):
    def __init__(self, vision_dim=1024, llm_dim=4096):
        super().__init__()
        self.proj = nn.Sequential(
            nn.Linear(vision_dim, llm_dim),
            nn.GELU(),
            nn.Linear(llm_dim, llm_dim)
        )

    def forward(self, vision_emb):
        # vision_emb: (B, vision_dim)

        return self.proj(vision_emb)  # (B, llm_dim)

Stage 2: Visual Instruction Tuning

After projection alignment, the LLM (LLaMA, Vicuna, or similar) is unfrozen. The model trains on image-prompt pairs generated automatically from COCO captions using GPT-4. During this phase, the model learns to answer questions, generate descriptions, and follow complex instructions while conditioning on the visual tokens provided by the frozen encoder and trained projector.

End-to-End Inference Pipeline

Combining these components yields a complete VLM inference flow. The following pseudo-code from the curriculum demonstrates the integration:

# 1️⃣ Encode image → CLIP ViT-L embedding

img_emb = clip_encoder(image)          # shape (B, 1024)

# 2️⃣ Project into LLM space

proj_emb = projector(img_emb)          # shape (B, 4096)

# 3️⃣ Append as a pseudo-token to the LLM input

llm_input = torch.cat([proj_emb, llm_token_embeddings], dim=1)

# 4️⃣ Run LLM (frozen) to generate a caption / answer

output_ids = llm.generate(llm_input)

A full runnable demo appears in phases/12-multimodal-ai/05-llava-visual-instruction-tuning/code/main.py.

Vision Encoder Selection and Projection Variants

The repository catalogs several vision encoder families and their VLM applications in phases/12-multimodal-ai/02-clip-contrastive-pretraining/docs/en.md:

• CLIP ViT-L/14: Dominant for semantic similarity and zero-shot classification; used in classic CLIP and early VLMs.
• SigLIP 2 (SO400m/14): Offers faster training and higher accuracy for contrastive pre-training; prevalent in 2025-2026 VLMs like Qwen-VL and InternViT.
• DINOv2 ViT-g: Provides dense visual features suited for segmentation and detection tasks requiring fine-grained perception.
• ViT-MAE: Pre-trained via masked image modeling, suitable for reconstruction-focused generation tasks.

Projection Layer Designs

The canonical two-layer MLP serves as the baseline for LLaVA 1.5, BLIP-2's Q-Former, and many contemporary architectures. However, phases/12-multimodal-ai/60-projection-layer-modality-align/docs/en.md documents advanced alternatives including GLU variants and attention-based bridges used in LLaVA-NeXT and Qwen-VL. The choice between simple MLPs and complex bridges depends on the trade-off between parameter efficiency and alignment quality.

Summary

• Vision Transformers convert images to tokens via patch extraction and a [CLS] token, forming the foundation of all VLMs.
• CLIP trains dual encoders with InfoNCE loss to create a shared embedding space where cosine similarity measures image-text relevance.
• LLaVA adds a lightweight 2-layer MLP projector that aligns frozen CLIP embeddings to frozen LLMs, followed by visual instruction tuning on GPT-4 generated data.
• The curriculum provides complete implementations in phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py and phases/12-multimodal-ai/05-llava-visual-instruction-tuning/code/main.py.
• Encoder selection (CLIP, SigLIP 2, DINOv2, MAE) depends on whether the application requires semantic ranking, dense vision, or reconstruction capabilities.

Frequently Asked Questions

What is the difference between CLIP and LLaVA architectures?

CLIP uses a contrastive objective to align image and text embeddings in a shared space for retrieval and classification, keeping both encoders frozen after training. LLaVA uses a generative approach, adding a trainable projection layer to feed visual embeddings into a language model, enabling text generation conditioned on images.

Why does LLaVA use a 2-layer MLP projector instead of a single linear layer?

According to phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md, the intermediate GELU activation and non-linear transformation in a 2-layer MLP provide sufficient capacity to transform the visual feature space (optimized for contrastive learning) into the linguistic feature space of the LLM without adding excessive parameters. A single linear layer often lacks the expressiveness to bridge this modality gap effectively.

How does the InfoNCE temperature parameter affect CLIP training?

The temperature parameter τ scales the similarity logits in the InfoNCE loss. The repository notes it is initialized to approximately 0.07 and optimized in log-space. Lower temperatures sharpen the softmax distribution, making the model more confident in positive pairs but potentially causing collapse; higher temperatures soften the distribution. Learning τ automatically allows the model to find the optimal sharpness for the specific embedding space.

Can I use DINOv2 instead of CLIP for the visual encoder in a LLaVA-style model?

Yes. While CLIP provides semantic embeddings optimized for image-text alignment, phases/12-multimodal-ai/02-clip-contrastive-pretraining/docs/en.md indicates that DINOv2 ViT-g offers superior dense visual features for tasks requiring fine-grained spatial understanding. However, using DINOv2 may require longer projection alignment training or a more complex bridge architecture, as its features are not pre-aligned with linguistic concepts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}