# CLIP, LLaVA, and FLAVA: Which Multimodal AI Architectures Are Taught in AI Engineering From Scratch

> Learn CLIP and LLaVA through hands-on AI engineering from scratch. Explore which multimodal AI architectures are covered, excluding FLAVA in this curriculum.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-06-18

---

**The AI Engineering From Scratch curriculum teaches CLIP and LLaVA through hands-on implementations, but does not include FLAVA in the current lesson set.**

The rohitg00/ai-engineering-from-scratch repository provides production-ready code for understanding modern multimodal AI architectures. This guide examines which vision-language models are covered, including complete implementation details for CLIP's contrastive learning framework and LLaVA's late-fusion approach, while clarifying why FLAVA is absent from the current materials.

## CLIP: Contrastive Language-Image Pre-training

The curriculum dedicates a full lesson to CLIP, implementing the foundational two-tower architecture that revolutionized zero-shot computer vision.

### Dual Encoder Architecture

According to the source code in [`phases/12-multimodal-ai/02-clip-contrastive-pretraining/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/02-clip-contrastive-pretraining/docs/en.md), CLIP consists of separate image and text encoders. The image encoder uses a Vision Transformer (ViT), while the text encoder employs a standard transformer language model. These encoders map images and text into a shared embedding space where matching pairs align.

### InfoNCE Loss Implementation

The training objective relies on a symmetric InfoNCE loss that pulls matching image-text pairs together while pushing non-matching pairs apart. In [`phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py), the curriculum implements this bidirectional contrastive loss:

```python

# File: phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py

def info_nce_loss(image_emb, text_emb, temperature=0.07):
    """Bidirectional InfoNCE used in CLIP and friends."""
    # Normalise embeddings

    image_emb = image_emb / np.linalg.norm(image_emb, axis=1, keepdims=True)
    text_emb  = text_emb  / np.linalg.norm(text_emb,  axis=1, keepdims=True)

    # Similarity matrix

    logits = image_emb @ text_emb.T / temperature
    # Symmetric cross‑entropy

    loss_i = -np.log(np.exp(np.diag(logits)) / np.exp(logits).sum(axis=1))
    loss_t = -np.log(np.exp(np.diag(logits)) / np.exp(logits).sum(axis=0))
    return (loss_i + loss_t).mean()

```

The implementation includes a learned temperature parameter (`τ`) that controls the softness of the softmax distribution during training.

### Zero-Shot Capabilities

After training, CLIP enables zero-shot image classification through cosine similarity between image embeddings and text embeddings of candidate labels. The curriculum includes a dedicated skill file at [`phases/12-multimodal-ai/02-clip-contrastive-pretraining/outputs/skill-clip-zero-shot.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/02-clip-contrastive-pretraining/outputs/skill-clip-zero-shot.md) demonstrating how to use the model for classification without task-specific fine-tuning.

## LLaVA: Large Language-and-Vision Assistant

The curriculum covers LLaVA as the canonical example of late-fusion multimodal AI architectures, teaching how to bridge frozen vision encoders with large language models.

### Late-Fusion Architecture

As documented in [`phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md), LLaVA freezes both the vision encoder (typically CLIP or SigLIP) and the large language model (LLaMA-style). This architecture prevents catastrophic forgetting of pre-trained representations while learning cross-modal alignment through a lightweight adapter.

### Two-Layer MLP Projector

The crucial innovation is the projection layer that maps visual features into the LLM's token space. The implementation in [`phases/19-capstone-projects/60-projection-layer-modality-align/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/60-projection-layer-modality-align/code/main.py) defines the canonical LLaVA-1.5 projector:

```python

# File: phases/19-capstone-projects/60-projection-layer-modality-align/code/main.py

import torch.nn as nn

class LlavaProjector(nn.Module):
    """Two‑layer MLP (linear → GELU → linear) used by LLaVA‑style VLMs."""
    def __init__(self, in_dim, hidden_dim, out_dim):
        super().__init__()
        self.fc1 = nn.Linear(in_dim, hidden_dim)
        self.gelu = nn.GELU()
        self.fc2 = nn.Linear(hidden_dim, out_dim)

    def forward(self, x):
        return self.fc2(self.gelu(self.fc1(x)))

```

This `Linear → GELU → Linear` bridge converts CLIP visual embeddings (e.g., dimension 768) into the LLM's hidden dimension (e.g., 1024), serving as the modality alignment mechanism.

### Two-Stage Training Pipeline

The curriculum emphasizes LLaVA's staged training approach:

1. **Alignment stage**: The projector learns contrastive alignment between visual queries and the LLM's CLS token while both encoders remain frozen.
2. **Instruction tuning stage**: The full model (projector + LLM) undergoes fine-tuning on multimodal instruction datasets like LLaVA-Instruct and ShareGPT-4V.

The lesson also references modern variants including LLaVA-NeXT and LLaVA-OneVision, which extend this recipe with higher-resolution encoders and tiled input processing.

## FLAVA: Not Covered in Current Curriculum

Despite being a significant open-source vision-language model, **FLAVA is not taught** in the AI Engineering From Scratch repository. A comprehensive search of the codebase reveals no FLAVA modules, lesson folders, or implementation files.

The curriculum focuses on the most pedagogically valuable architectures that emerged before 2026, prioritizing CLIP and LLaVA for their foundational impact on the field. Adding FLAVA would require creating a new lesson following the repository's template structure, including [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md), [`code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/code/main.py), unit tests, and [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) files.

## Key Implementation Files

The following source files contain the canonical implementations of these multimodal AI architectures:

- [`phases/12-multimodal-ai/02-clip-contrastive-pretraining/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/02-clip-contrastive-pretraining/docs/en.md) – Theory of CLIP, training objectives, and scaling laws.
- [`phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py) – Minimal InfoNCE loss implementation in Python/NumPy.
- [`phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md) – LLaVA architecture overview and training methodologies.
- [`phases/19-capstone-projects/60-projection-layer-modality-align/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/60-projection-layer-modality-align/code/main.py) – Two-layer MLP projector implementation.
- [`phases/12-multimodal-ai/07-open-weight-vlm-recipes/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/07-open-weight-vlm-recipes/code/main.py) – Comparative recipes for VLM families including LLaVA-1.5 and LLaVA-NeXT.

## Summary

- **CLIP** is taught through a complete lesson on contrastive vision-language pretraining, including the dual encoder architecture and InfoNCE loss implementation.
- **LLaVA** is covered as the standard late-fusion VLM, with detailed code for the two-layer MLP projector and two-stage training pipeline.
- **FLAVA** is not included in the curriculum; the repository contains no lessons or code for this architecture.
- Both CLIP and LLaVA implementations include working Python code that demonstrates the core mathematical concepts behind these multimodal AI architectures.

## Frequently Asked Questions

### Does the curriculum cover FLAVA?

No, the AI Engineering From Scratch curriculum does not include FLAVA. The repository focuses on CLIP and LLaVA as the primary examples of multimodal architectures, with no existing lesson files, documentation, or code implementations for FLAVA.

### What is the difference between CLIP and LLaVA in the curriculum?

CLIP teaches **contrastive pre-training** using dual encoders and InfoNCE loss to align image-text pairs in a shared embedding space. LLaVA teaches **late-fusion architecture** that connects frozen vision encoders to frozen language models through a learned projector, enabling visual instruction following.

### How does the LLaVA projector work?

The LLaVA projector is a two-layer MLP (`Linear → GELU → Linear`) defined in [`phases/19-capstone-projects/60-projection-layer-modality-align/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/60-projection-layer-modality-align/code/main.py). It maps frozen CLIP visual embeddings into the token space of the frozen LLM, allowing the language model to process visual information without retraining either backbone.

### Can I use the CLIP implementation for zero-shot classification?

Yes, the CLIP lesson includes zero-shot capabilities. The implementation in [`phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/02-clip-contrastive-pretraining/code/main.py) supports computing similarity between image embeddings and text embeddings, enabling classification without task-specific training data.