What Datasets Are Used in the AI Engineering From Scratch Project?

The ai-engineering-from-scratch curriculum uses a realistic mix of real-world corpora (Common Crawl, C4, Wikipedia, GitHub code, arXiv) and synthetic mock datasets (200-pair image-caption sets, shape datasets, generated papers) to mirror large-scale LLM training while remaining runnable on laptops.

This open-source educational repository by rohitg00 provides a comprehensive guide to building AI systems from the ground up. The datasets used in the ai-engineering-from-scratch project reflect the exact data mixtures found in modern production models like Llama 3, balancing massive public crawls with lightweight synthetic alternatives for rapid prototyping.

Web Crawl and Filtered Corpora

The curriculum begins with the foundational text sources that power today's large language models.

According to phases/10-llms-from-scratch/03-data-pipelines/docs/en.md, the following web-scale datasets form the backbone of pre-training:

• Common Crawl (≈250 TB raw) – An unfiltered snapshot of the web requiring heavy deduplication and quality filtering to remove low-quality content.
• C4 (Colossal Clean Crawled Corpus) – A curated web corpus of approximately 5 TB that has already undergone pre-filtering for quality.
• RefinedWeb – Another high-quality filtered web dataset of similar scale (≈5 TB) used as an alternative to raw Common Crawl.

These sources provide the bulk of token data needed for base model training, representing the "low-quality, high-quantity" paradigm that requires sophisticated cleaning pipelines.

Encyclopedic and Academic Sources

High-quality factual and scientific text provides grounding for knowledge-intensive tasks.

The repository identifies these key sources in phases/10-llms-from-scratch/03-data-pipelines/docs/en.md:

• Wikipedia (≈20 GB) – High-quality encyclopedic text used for factual grounding.
• arXiv – A collection of scientific papers (≈100 GB) spanning STEM disciplines.
• S2ORC (Semantic Scholar Open Research Corpus) – Another high-quality academic paper corpus (≈100 GB) enabling scientific QA and citation-aware generation.

These datasets improve the model's reasoning capabilities and ability to generate accurate technical content.

Code and Book Corpora

Long-form narrative and source code improve specialized generation tasks.

The curriculum includes:

• GitHub Code (≈1 TB+) – Medium-quality source code with many duplicates, essential for training code-generation models like StarCoder.
• BookCorpus / The Pile (≈100 GB) – High-quality narrative text from books that improves long-form generation and storytelling capabilities.

As documented in phases/10-llms-from-scratch/03-data-pipelines/docs/en.md, these sources target specific capabilities beyond general web text.

Community Forums and Dialogue Data

Conversational data trains models for chat and instruction-following behavior.

The repository utilizes:

• StackOverflow – Technical Q&A content (≈100 GB) useful for coding assistance models.
• Reddit – Conversational forum data (≈100 GB) providing dialogue-like threads for chat-style fine-tuning.

These medium-quality sources supply the conversational patterns necessary for interactive AI systems.

Multimodal Image-Text Datasets

For vision-language model training, the curriculum includes specific paired datasets.

According to phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md and phases/12-multimodal-ai/10-internvl3-native-multimodal/docs/en.md:

• CC3M (Conceptual Captions 3 Million) – Image-caption pairs for initial vision-language alignment.
• LAION-CC-SBU (≈558k pairs) – A specific subset used in the LLaVA visual instruction tuning lessons.
• OBELICS – A large interleaved image-text corpus for native multimodal training.
• MMC4 – Multimodal C4, offering interleaved documents with images and text.

These datasets enable the pre-training of vision-language models (VLMs) like CLIP, BLIP-2, and LLaVA.

Synthetic Mock Datasets for Capstone Projects

To enable hands-on learning without terabyte-scale downloads, the repository provides deterministic synthetic datasets.

As implemented in phases/19-capstone-projects/62-vision-language-pretraining/docs/en.md, these include:

• 200-pair mock image-caption corpus – A lightweight replacement for LAION that preserves the same code paths without requiring downloads.
• Synthetic shape dataset – Used in phases/19-capstone-projects/58-vision-encoder-patches/docs/en.md, featuring 4 classes (circle, square, triangle, star) for vision encoder patch experiments.
• Literature retrieval corpus – 100 generated papers described in phases/19-capstone-projects/51-literature-retrieval/docs/en.md for retrieval-augmented generation demos.
• SMS spam/ham generator – 800 generated sentences provided in phases/19-capstone-projects/38-classifier-finetuning/docs/en.md for classification fine-tuning exercises.

These synthetic datasets allow the curriculum to demonstrate end-to-end pipelines on modest hardware.

Loading and Preprocessing Code Examples

The repository provides practical patterns for loading these datasets using standard tools.

To load the curated web corpus (C4):

from datasets import load_dataset

c4 = load_dataset("c4", "en", split="train")  # ≈5 TB pre-filtered web text

print(f"C4 sample: {c4[0]['text'][:200]}")

For multimodal training with image-text pairs:

from datasets import load_dataset

laion = load_dataset("laion/cc_sbu", split="train[:1000]")  # tiny slice for demo

for rec in laion[:3]:
    print("Image URL:", rec["image_url"])
    print("Caption:", rec["caption"])

To create the synthetic shape dataset used in vision encoder lessons:

import random
import itertools

shapes = ["circle", "square", "triangle", "star"]

def synthetic_shapes(num=200):
    for _ in range(num):
        label = random.choice(shapes)
        # A trivial "image" representation – just a token string

        yield f"{label} shape", label

synthetic = list(itertools.islice(synthetic_shapes(), 5))
print(synthetic)

For token-level pre-training with custom text:

from pathlib import Path

corpus = Path("data/small_gutenberg.txt").read_text()
tokens = tokenizer.encode(corpus)

# Pack into fixed-length sequences (seq_len=2048)

seq_len = 2048
sequences = [tokens[i:i+seq_len] for i in range(0, len(tokens), seq_len)]
print(f"Number of training sequences: {len(sequences)}")

Summary

• The ai-engineering-from-scratch curriculum mirrors the data mix of modern LLMs, including approximately 50% web text, 25% code, and 25% academic and book sources.
• Real-world datasets include Common Crawl, C4, Wikipedia, GitHub, arXiv, and LAION-CC-SBU, documented in phases/10-llms-from-scratch/03-data-pipelines/docs/en.md.
• Multimodal datasets like OBELICS and MMC4 support vision-language training as detailed in phases/12-multimodal-ai/10-internvl3-native-multimodal/docs/en.md.
• Synthetic mock datasets (200-pair image-caption sets, shape datasets, generated papers) enable rapid iteration without large downloads, found in the capstone project phase files.
• All loading patterns follow the exact implementations shown in the data pipeline lessons, ensuring compatibility with production-scale training workflows.

Frequently Asked Questions

What is the largest dataset mentioned in the ai-engineering-from-scratch project?

The Common Crawl dataset represents the largest source at approximately 250 TB of raw web data. However, the curriculum recommends using filtered versions like C4 or RefinedWeb (≈5 TB each) to avoid the extensive filtering requirements of the raw crawl.

How does the project handle multimodal data for vision-language models?

The curriculum uses CC3M, LAION-CC-SBU, and interleaved corpora like OBELICS and MMC4 for vision-language training. As documented in phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md, these provide image-caption pairs necessary for pre-training models like LLaVA and BLIP-2.

Why does the repository include synthetic datasets instead of only real-world data?

Synthetic datasets (such as the 200-pair mock image-caption corpus and 4-class shape dataset) allow students to run end-to-end demo pipelines on laptops without downloading terabytes of data. These mocks preserve the exact code paths used in production while making the curriculum accessible.

Where can I find the documentation for data preprocessing in this project?

The primary documentation for data pipelines resides in phases/10-llms-from-scratch/03-data-pipelines/docs/en.md, which covers downloading, cleaning, deduplication, and packing sequences for pre-training. Capstone-specific data generation is documented in the respective project files under phases/19-capstone-projects/.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}