# CLIP, LLaVA, and Chameleon: Multimodal Architectures Covered in AI Engineering From Scratch

> Explore robust implementations of CLIP, LLaVA, and Chameleon multimodal architectures in ai-engineering-from-scratch. Get runnable code and production pipelines for these foundational models.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-06-14

---

**The ai-engineering-from-scratch curriculum implements three foundational multimodal architectures—CLIP (contrastive dual-tower), LLaVA (late-fusion adapter), and Chameleon (early-fusion token-only)—providing runnable code and production-grade training pipelines for each.**

The rohitg00/ai-engineering-from-scratch repository teaches these multimodal architectures through hands-on lessons that cover theory, implementation, and practical fine-tuning. Each module includes reference implementations, mathematical derivations, and skill-based pipelines you can execute from scratch.

## CLIP: Contrastive Vision-Language Pretraining

CLIP (Contrastive Language-Image Pre-training) represents the canonical **dual-tower architecture** that established the modern standard for vision-language alignment.

### Architecture Components

In [`phases/12-multimodal-ai/02-clip-contrastive-pretraining/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/02-clip-contrastive-pretraining/docs/en.md), the curriculum breaks down CLIP into two parallel encoders:

- **Image tower**: A Vision Transformer (ViT) such as ViT-L/14 that maps input images to 768-dimensional vectors.
- **Text tower**: A transformer encoder that processes caption text (with prompt templates) into 768-dimensional embeddings.
- **Late-fusion mechanism**: Both outputs are **L2-normalized** and compared via cosine similarity in a shared embedding space.

### Contrastive Loss Implementation

The training objective uses **InfoNCE** (symmetric cross-entropy) over a batch of *N* image-caption pairs:

```python

# Conceptual implementation from phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py

# The loss computes symmetric cross-entropy between image and text embeddings

loss = - (log(exp(z_i @ c_i / tau) / sum(exp(z_i @ c_j / tau))) + 
          log(exp(c_i @ z_i / tau) / sum(exp(c_i @ z_j / tau))))

```

Where **τ** represents a learned temperature parameter. This contrastive approach enables **zero-shot classification** by ranking image embeddings against text embeddings generated from class templates.

## LLaVA: Late-Fusion Visual Instruction Tuning

LLaVA (Large Language and Vision Assistant) introduces the **adapter-style late-fusion** pattern that powers most open-weight VLMs built after 2024.

### The Projector Design

According to [`phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md), the architecture connects frozen pretrained models through a lightweight bridge:

- **Vision encoder**: A frozen CLIP or SigLIP checkpoint (the same two-tower used in CLIP).
- **Language model**: A frozen Llama-compatible LLM (e.g., LLaMA-7B).
- **Projection head**: A **two-layer MLP** (Linear → GELU → Linear) mapping the vision encoder's hidden dimension (e.g., 1024) to the LLM's embedding dimension (e.g., 2048).

The projector contains approximately **1 million parameters**, making it data-efficient to train while preserving the knowledge in frozen encoders.

### Two-Stage Training Pipeline

The lesson in [`phases/12-multimodal-ai/07-open-weight-vlm-recipes/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/07-open-weight-vlm-recipes/docs/en.md) outlines a strict training protocol:

1. **Alignment stage**: Only the projector is trained using CLIP-style contrastive loss while both vision and language towers remain frozen.
2. **Instruction tuning stage**: The LLM is unfrozen and fine-tuned on multimodal instruction datasets (LLaVA-Inst-150k, ShareGPT-4V), while the vision encoder stays frozen.

This separation prevents overfitting and maintains the vision encoder's generalizable features.

## Chameleon: Early-Fusion Token-Only Models

Chameleon represents a paradigm shift to **early-fusion token-only** architectures, where images and text share a unified vocabulary and processing pipeline.

### Unified VQ-VAE Tokenization

As detailed in [`phases/12-multimodal-ai/11-chameleon-early-fusion-tokens/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/11-chameleon-early-fusion-tokens/docs/en.md), the architecture uses a single tokenizer for both modalities:

- **Image tokenizer**: A **VQ-VAE** (Vector Quantized Variational AutoEncoder) encodes 512×512 images into 1024 discrete tokens.
- **Codebook specifications**: 8192 entries (K=8192), each 256-dimensional.
- **Shared vocabulary**: Image tokens, text BPE tokens, and special separators (`<IMG>`, `<SEP>`) share a single vocabulary of approximately 40,000 entries.

### Single Transformer Processing

Unlike CLIP and LLaVA, Chameleon uses **one autoregressive transformer** that processes a concatenated mixed-modality stream:

```

[<IMG> image_token_1 ... image_token_1024 <SEP> text_token_1 ... text_token_M]

```

The model trains with a standard **next-token-prediction loss** across the entire sequence, enabling it to generate interleaved text and image outputs.

### Training Stability Techniques

The curriculum emphasizes three critical implementation details from the Chameleon paper that enable stable training at 34 billion parameters:

- **QK-Norm**: Normalizing query and key vectors before attention computations.
- **Dropout placement**: Strategic positioning of dropout layers to prevent overfitting without harming convergence.
- **LayerNorm ordering**: Careful attention to pre-norm versus post-norm configurations to prevent "norm-explosion" during large-scale training.

## Practical Code Examples

The repository provides minimal, runnable implementations for each architecture.

### CLIP Zero-Shot Classification

```python

# phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py

from pathlib import Path
from clip_demo import clip_zero_shot

images = [Path("imgs/cat.jpg"), Path("imgs/dog.jpg")]
classes = ["a photo of a cat", "a photo of a dog", "a photo of a horse"]

preds = clip_zero_shot(images, classes, checkpoint="openai/clip-vit-large-patch14")
for img, top in zip(images, preds):
    print(f"{img.name} → {top[0]} (score {top[1]:.3f})")

```

### LLaVA Two-Stage Fine-tuning

```python

# phases/12-multimodal-ai/05-llava-visual-instruction-tuning/code/main.py

from llava_finetune import train_llava

# Stage 1: Projector alignment (only MLP trainable)

train_llava(
    vision_checkpoint="openai/clip-vit-large-patch14",
    llm_checkpoint="meta-llama/Llama-2-7b-chat",
    projector_type="mlp-2",
    phase="alignment"
)

# Stage 2: Instruction tuning (unfreeze LLM)

train_llava(
    vision_checkpoint="openai/clip-vit-large-patch14",
    llm_checkpoint="meta-llama/Llama-2-7b-chat",
    projector_type="mlp-2",
    phase="instruction",
    data="data/llava-instruct-150k.json"
)

```

### Chameleon Mixed-Modality Generation

```python

# phases/12-multimodal-ai/11-chameleon-early-fusion-tokens/code/main.py

from chameleon_demo import generate_mixed

prompt = "Describe the scene and then draw it:"
output = generate_mixed(prompt, tokenizer="chameleon-vqvae", max_len=2048)

print("Generated token stream:")
print(output)

# Output interleaves text tokens and image tokens marked with <IMG>

```

## Summary

- **CLIP** provides a **dual-tower contrastive** foundation using ViT and text transformers with InfoNCE loss, enabling zero-shot transfer through shared embedding spaces.
- **LLaVA** demonstrates **late-fusion adaptation** through a lightweight two-layer MLP projector that bridges frozen vision encoders and frozen LLMs via two-stage training.
- **Chameleon** implements **early-fusion tokenization** using VQ-VAE to convert images into discrete tokens that share a vocabulary with text, processed by a single autoregressive transformer with specialized stability techniques (QK-Norm, specific LayerNorm ordering).

## Frequently Asked Questions

### What is the key difference between CLIP and LLaVA architectures?

CLIP uses **dual encoders** that process images and text separately, comparing their outputs via contrastive loss. LLaVA employs **late-fusion** where a frozen CLIP vision encoder feeds into a frozen LLM through a trainable projection layer, allowing the model to generate text conditioned on visual inputs rather than just scoring similarity.

### How does Chameleon differ from LLaVA in processing multimodal inputs?

Chameleon uses **early-fusion** where images are tokenized into discrete VQ-VAE tokens that share the same vocabulary as text tokens, then processed by a single transformer. LLaVA maintains separate encoders and uses a projector to align continuous visual features with text embeddings, making Chameleon more unified but requiring careful training stability tricks like QK-Norm.

### What files contain the reference implementations for these architectures?

The source code resides in [`phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py) for CLIP, [`phases/12-multimodal-ai/05-llava-visual-instruction-tuning/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/05-llava-visual-instruction-tuning/code/main.py) for LLaVA, and [`phases/12-multimodal-ai/11-chameleon-early-fusion-tokens/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/11-chameleon-early-fusion-tokens/code/main.py) for Chameleon, with documentation in the corresponding [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) files.

### Why does the LLaVA training use two separate stages?

The two-stage approach in [`phases/12-multimodal-ai/05-llava-visual-instruction-tuning/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/05-llava-visual-instruction-tuning/code/main.py) first aligns the vision and language modalities by training only the small projector (≈1M parameters), then fine-tunes the LLM on instruction data. This prevents the LLM from overfitting early and preserves the vision encoder's generalizable representations learned during CLIP pretraining.