CLIP, LLaVA, and Chameleon: Multimodal Architectures Covered in AI Engineering From Scratch

The ai-engineering-from-scratch curriculum implements three foundational multimodal architectures—CLIP (contrastive dual-tower), LLaVA (late-fusion adapter), and Chameleon (early-fusion token-only)—providing runnable code and production-grade training pipelines for each.

The rohitg00/ai-engineering-from-scratch repository teaches these multimodal architectures through hands-on lessons that cover theory, implementation, and practical fine-tuning. Each module includes reference implementations, mathematical derivations, and skill-based pipelines you can execute from scratch.

CLIP: Contrastive Vision-Language Pretraining

CLIP (Contrastive Language-Image Pre-training) represents the canonical dual-tower architecture that established the modern standard for vision-language alignment.

Architecture Components

In phases/12-multimodal-ai/02-clip-contrastive-pretraining/docs/en.md, the curriculum breaks down CLIP into two parallel encoders:

• Image tower: A Vision Transformer (ViT) such as ViT-L/14 that maps input images to 768-dimensional vectors.
• Text tower: A transformer encoder that processes caption text (with prompt templates) into 768-dimensional embeddings.
• Late-fusion mechanism: Both outputs are L2-normalized and compared via cosine similarity in a shared embedding space.

Contrastive Loss Implementation

The training objective uses InfoNCE (symmetric cross-entropy) over a batch of N image-caption pairs:

# Conceptual implementation from phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py

# The loss computes symmetric cross-entropy between image and text embeddings

loss = - (log(exp(z_i @ c_i / tau) / sum(exp(z_i @ c_j / tau))) + 
          log(exp(c_i @ z_i / tau) / sum(exp(c_i @ z_j / tau))))

Where τ represents a learned temperature parameter. This contrastive approach enables zero-shot classification by ranking image embeddings against text embeddings generated from class templates.

LLaVA: Late-Fusion Visual Instruction Tuning

LLaVA (Large Language and Vision Assistant) introduces the adapter-style late-fusion pattern that powers most open-weight VLMs built after 2024.

The Projector Design

According to phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md, the architecture connects frozen pretrained models through a lightweight bridge:

• Vision encoder: A frozen CLIP or SigLIP checkpoint (the same two-tower used in CLIP).
• Language model: A frozen Llama-compatible LLM (e.g., LLaMA-7B).
• Projection head: A two-layer MLP (Linear → GELU → Linear) mapping the vision encoder's hidden dimension (e.g., 1024) to the LLM's embedding dimension (e.g., 2048).

The projector contains approximately 1 million parameters, making it data-efficient to train while preserving the knowledge in frozen encoders.

Two-Stage Training Pipeline

The lesson in phases/12-multimodal-ai/07-open-weight-vlm-recipes/docs/en.md outlines a strict training protocol:

1. Alignment stage: Only the projector is trained using CLIP-style contrastive loss while both vision and language towers remain frozen.
2. Instruction tuning stage: The LLM is unfrozen and fine-tuned on multimodal instruction datasets (LLaVA-Inst-150k, ShareGPT-4V), while the vision encoder stays frozen.

This separation prevents overfitting and maintains the vision encoder's generalizable features.

Chameleon: Early-Fusion Token-Only Models

Chameleon represents a paradigm shift to early-fusion token-only architectures, where images and text share a unified vocabulary and processing pipeline.

Unified VQ-VAE Tokenization

As detailed in phases/12-multimodal-ai/11-chameleon-early-fusion-tokens/docs/en.md, the architecture uses a single tokenizer for both modalities:

• Image tokenizer: A VQ-VAE (Vector Quantized Variational AutoEncoder) encodes 512×512 images into 1024 discrete tokens.
• Codebook specifications: 8192 entries (K=8192), each 256-dimensional.
• Shared vocabulary: Image tokens, text BPE tokens, and special separators (<IMG>, <SEP>) share a single vocabulary of approximately 40,000 entries.

Single Transformer Processing

Unlike CLIP and LLaVA, Chameleon uses one autoregressive transformer that processes a concatenated mixed-modality stream:

[<IMG> image_token_1 ... image_token_1024 <SEP> text_token_1 ... text_token_M]

The model trains with a standard next-token-prediction loss across the entire sequence, enabling it to generate interleaved text and image outputs.

Training Stability Techniques

The curriculum emphasizes three critical implementation details from the Chameleon paper that enable stable training at 34 billion parameters:

• QK-Norm: Normalizing query and key vectors before attention computations.
• Dropout placement: Strategic positioning of dropout layers to prevent overfitting without harming convergence.
• LayerNorm ordering: Careful attention to pre-norm versus post-norm configurations to prevent "norm-explosion" during large-scale training.

Practical Code Examples

The repository provides minimal, runnable implementations for each architecture.

CLIP Zero-Shot Classification

# phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py

from pathlib import Path
from clip_demo import clip_zero_shot

images = [Path("imgs/cat.jpg"), Path("imgs/dog.jpg")]
classes = ["a photo of a cat", "a photo of a dog", "a photo of a horse"]

preds = clip_zero_shot(images, classes, checkpoint="openai/clip-vit-large-patch14")
for img, top in zip(images, preds):
    print(f"{img.name} → {top[0]} (score {top[1]:.3f})")

LLaVA Two-Stage Fine-tuning

# phases/12-multimodal-ai/05-llava-visual-instruction-tuning/code/main.py

from llava_finetune import train_llava

# Stage 1: Projector alignment (only MLP trainable)

train_llava(
    vision_checkpoint="openai/clip-vit-large-patch14",
    llm_checkpoint="meta-llama/Llama-2-7b-chat",
    projector_type="mlp-2",
    phase="alignment"
)

# Stage 2: Instruction tuning (unfreeze LLM)

train_llava(
    vision_checkpoint="openai/clip-vit-large-patch14",
    llm_checkpoint="meta-llama/Llama-2-7b-chat",
    projector_type="mlp-2",
    phase="instruction",
    data="data/llava-instruct-150k.json"
)

Chameleon Mixed-Modality Generation

# phases/12-multimodal-ai/11-chameleon-early-fusion-tokens/code/main.py

from chameleon_demo import generate_mixed

prompt = "Describe the scene and then draw it:"
output = generate_mixed(prompt, tokenizer="chameleon-vqvae", max_len=2048)

print("Generated token stream:")
print(output)

# Output interleaves text tokens and image tokens marked with <IMG>

Summary

• CLIP provides a dual-tower contrastive foundation using ViT and text transformers with InfoNCE loss, enabling zero-shot transfer through shared embedding spaces.
• LLaVA demonstrates late-fusion adaptation through a lightweight two-layer MLP projector that bridges frozen vision encoders and frozen LLMs via two-stage training.
• Chameleon implements early-fusion tokenization using VQ-VAE to convert images into discrete tokens that share a vocabulary with text, processed by a single autoregressive transformer with specialized stability techniques (QK-Norm, specific LayerNorm ordering).

Frequently Asked Questions

What is the key difference between CLIP and LLaVA architectures?

CLIP uses dual encoders that process images and text separately, comparing their outputs via contrastive loss. LLaVA employs late-fusion where a frozen CLIP vision encoder feeds into a frozen LLM through a trainable projection layer, allowing the model to generate text conditioned on visual inputs rather than just scoring similarity.

How does Chameleon differ from LLaVA in processing multimodal inputs?

Chameleon uses early-fusion where images are tokenized into discrete VQ-VAE tokens that share the same vocabulary as text tokens, then processed by a single transformer. LLaVA maintains separate encoders and uses a projector to align continuous visual features with text embeddings, making Chameleon more unified but requiring careful training stability tricks like QK-Norm.

What files contain the reference implementations for these architectures?

The source code resides in phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py for CLIP, phases/12-multimodal-ai/05-llava-visual-instruction-tuning/code/main.py for LLaVA, and phases/12-multimodal-ai/11-chameleon-early-fusion-tokens/code/main.py for Chameleon, with documentation in the corresponding docs/en.md files.

Why does the LLaVA training use two separate stages?

The two-stage approach in phases/12-multimodal-ai/05-llava-visual-instruction-tuning/code/main.py first aligns the vision and language modalities by training only the small projector (≈1M parameters), then fine-tunes the LLM on instruction data. This prevents the LLM from overfitting early and preserves the vision encoder's generalizable representations learned during CLIP pretraining.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}