# What Datasets Are Used in the AI Engineering From Scratch Project?

> Explore datasets used in the AI Engineering From Scratch project. Learn about real-world corpora like Common Crawl and C4, plus synthetic data for LLM training on laptops.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: getting-started
- Published: 2026-06-19

---

**The ai-engineering-from-scratch curriculum uses a realistic mix of real-world corpora (Common Crawl, C4, Wikipedia, GitHub code, arXiv) and synthetic mock datasets (200-pair image-caption sets, shape datasets, generated papers) to mirror large-scale LLM training while remaining runnable on laptops.**

This open-source educational repository by rohitg00 provides a comprehensive guide to building AI systems from the ground up. The datasets used in the ai-engineering-from-scratch project reflect the exact data mixtures found in modern production models like Llama 3, balancing massive public crawls with lightweight synthetic alternatives for rapid prototyping.

## Web Crawl and Filtered Corpora

The curriculum begins with the foundational text sources that power today's large language models.

According to [`phases/10-llms-from-scratch/03-data-pipelines/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/03-data-pipelines/docs/en.md), the following web-scale datasets form the backbone of pre-training:

- **Common Crawl** (≈250 TB raw) – An unfiltered snapshot of the web requiring heavy deduplication and quality filtering to remove low-quality content.
- **C4** (Colossal Clean Crawled Corpus) – A curated web corpus of approximately 5 TB that has already undergone pre-filtering for quality.
- **RefinedWeb** – Another high-quality filtered web dataset of similar scale (≈5 TB) used as an alternative to raw Common Crawl.

These sources provide the bulk of token data needed for base model training, representing the "low-quality, high-quantity" paradigm that requires sophisticated cleaning pipelines.

## Encyclopedic and Academic Sources

High-quality factual and scientific text provides grounding for knowledge-intensive tasks.

The repository identifies these key sources in [`phases/10-llms-from-scratch/03-data-pipelines/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/03-data-pipelines/docs/en.md):

- **Wikipedia** (≈20 GB) – High-quality encyclopedic text used for factual grounding.
- **arXiv** – A collection of scientific papers (≈100 GB) spanning STEM disciplines.
- **S2ORC** (Semantic Scholar Open Research Corpus) – Another high-quality academic paper corpus (≈100 GB) enabling scientific QA and citation-aware generation.

These datasets improve the model's reasoning capabilities and ability to generate accurate technical content.

## Code and Book Corpora

Long-form narrative and source code improve specialized generation tasks.

The curriculum includes:

- **GitHub Code** (≈1 TB+) – Medium-quality source code with many duplicates, essential for training code-generation models like StarCoder.
- **BookCorpus / The Pile** (≈100 GB) – High-quality narrative text from books that improves long-form generation and storytelling capabilities.

As documented in [`phases/10-llms-from-scratch/03-data-pipelines/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/03-data-pipelines/docs/en.md), these sources target specific capabilities beyond general web text.

## Community Forums and Dialogue Data

Conversational data trains models for chat and instruction-following behavior.

The repository utilizes:

- **StackOverflow** – Technical Q&A content (≈100 GB) useful for coding assistance models.
- **Reddit** – Conversational forum data (≈100 GB) providing dialogue-like threads for chat-style fine-tuning.

These medium-quality sources supply the conversational patterns necessary for interactive AI systems.

## Multimodal Image-Text Datasets

For vision-language model training, the curriculum includes specific paired datasets.

According to [`phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md) and [`phases/12-multimodal-ai/10-internvl3-native-multimodal/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/10-internvl3-native-multimodal/docs/en.md):

- **CC3M** (Conceptual Captions 3 Million) – Image-caption pairs for initial vision-language alignment.
- **LAION-CC-SBU** (≈558k pairs) – A specific subset used in the LLaVA visual instruction tuning lessons.
- **OBELICS** – A large interleaved image-text corpus for native multimodal training.
- **MMC4** – Multimodal C4, offering interleaved documents with images and text.

These datasets enable the pre-training of vision-language models (VLMs) like CLIP, BLIP-2, and LLaVA.

## Synthetic Mock Datasets for Capstone Projects

To enable hands-on learning without terabyte-scale downloads, the repository provides deterministic synthetic datasets.

As implemented in [`phases/19-capstone-projects/62-vision-language-pretraining/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/62-vision-language-pretraining/docs/en.md), these include:

- **200-pair mock image-caption corpus** – A lightweight replacement for LAION that preserves the same code paths without requiring downloads.
- **Synthetic shape dataset** – Used in [`phases/19-capstone-projects/58-vision-encoder-patches/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/58-vision-encoder-patches/docs/en.md), featuring 4 classes (circle, square, triangle, star) for vision encoder patch experiments.
- **Literature retrieval corpus** – 100 generated papers described in [`phases/19-capstone-projects/51-literature-retrieval/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/51-literature-retrieval/docs/en.md) for retrieval-augmented generation demos.
- **SMS spam/ham generator** – 800 generated sentences provided in [`phases/19-capstone-projects/38-classifier-finetuning/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/38-classifier-finetuning/docs/en.md) for classification fine-tuning exercises.

These synthetic datasets allow the curriculum to demonstrate end-to-end pipelines on modest hardware.

## Loading and Preprocessing Code Examples

The repository provides practical patterns for loading these datasets using standard tools.

To load the curated web corpus (C4):

```python
from datasets import load_dataset

c4 = load_dataset("c4", "en", split="train")  # ≈5 TB pre-filtered web text

print(f"C4 sample: {c4[0]['text'][:200]}")

```

For multimodal training with image-text pairs:

```python
from datasets import load_dataset

laion = load_dataset("laion/cc_sbu", split="train[:1000]")  # tiny slice for demo

for rec in laion[:3]:
    print("Image URL:", rec["image_url"])
    print("Caption:", rec["caption"])

```

To create the synthetic shape dataset used in vision encoder lessons:

```python
import random
import itertools

shapes = ["circle", "square", "triangle", "star"]

def synthetic_shapes(num=200):
    for _ in range(num):
        label = random.choice(shapes)
        # A trivial "image" representation – just a token string

        yield f"{label} shape", label

synthetic = list(itertools.islice(synthetic_shapes(), 5))
print(synthetic)

```

For token-level pre-training with custom text:

```python
from pathlib import Path

corpus = Path("data/small_gutenberg.txt").read_text()
tokens = tokenizer.encode(corpus)

# Pack into fixed-length sequences (seq_len=2048)

seq_len = 2048
sequences = [tokens[i:i+seq_len] for i in range(0, len(tokens), seq_len)]
print(f"Number of training sequences: {len(sequences)}")

```

## Summary

- The ai-engineering-from-scratch curriculum mirrors the data mix of modern LLMs, including approximately 50% web text, 25% code, and 25% academic and book sources.
- **Real-world datasets** include Common Crawl, C4, Wikipedia, GitHub, arXiv, and LAION-CC-SBU, documented in [`phases/10-llms-from-scratch/03-data-pipelines/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/03-data-pipelines/docs/en.md).
- **Multimodal datasets** like OBELICS and MMC4 support vision-language training as detailed in [`phases/12-multimodal-ai/10-internvl3-native-multimodal/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/10-internvl3-native-multimodal/docs/en.md).
- **Synthetic mock datasets** (200-pair image-caption sets, shape datasets, generated papers) enable rapid iteration without large downloads, found in the capstone project phase files.
- All loading patterns follow the exact implementations shown in the data pipeline lessons, ensuring compatibility with production-scale training workflows.

## Frequently Asked Questions

### What is the largest dataset mentioned in the ai-engineering-from-scratch project?

The **Common Crawl** dataset represents the largest source at approximately 250 TB of raw web data. However, the curriculum recommends using filtered versions like **C4** or **RefinedWeb** (≈5 TB each) to avoid the extensive filtering requirements of the raw crawl.

### How does the project handle multimodal data for vision-language models?

The curriculum uses **CC3M**, **LAION-CC-SBU**, and interleaved corpora like **OBELICS** and **MMC4** for vision-language training. As documented in [`phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md), these provide image-caption pairs necessary for pre-training models like LLaVA and BLIP-2.

### Why does the repository include synthetic datasets instead of only real-world data?

Synthetic datasets (such as the 200-pair mock image-caption corpus and 4-class shape dataset) allow students to run **end-to-end demo pipelines** on laptops without downloading terabytes of data. These mocks preserve the exact code paths used in production while making the curriculum accessible.

### Where can I find the documentation for data preprocessing in this project?

The primary documentation for data pipelines resides in [`phases/10-llms-from-scratch/03-data-pipelines/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/03-data-pipelines/docs/en.md), which covers downloading, cleaning, deduplication, and packing sequences for pre-training. Capstone-specific data generation is documented in the respective project files under `phases/19-capstone-projects/`.