NLP Topics Covered in the AI Engineering from Scratch Curriculum: 29 Lessons from Foundations to Advanced

The AI Engineering from Scratch curriculum includes 29 comprehensive NLP lessons spanning text processing, embeddings, sequence modeling, transformers, retrieval-augmented generation, and modern LLM evaluation frameworks.

Natural language processing (NLP) forms the backbone of modern AI applications, from search engines to conversational agents. The rohitg00/ai-engineering-from-scratch repository dedicates Phase 5 to building NLP expertise progressively—starting with basic text preprocessing and culminating in production-grade LLM systems. Each lesson pairs theory with runnable code, following the repository's hands-on philosophy.

---

Complete NLP Curriculum Overview

Phase 5 lives in phases/05-nlp-foundations-to-advanced/ and contains 29 sequentially organized lessons. Each lesson follows a consistent structure: documentation (docs/en.md), implementation code (code/main.py), and unit tests (code/tests/).

Text Processing and Classical NLP (Lessons 1–7)

These foundational lessons establish core NLP pipeline components:

• 01‑text‑processing — Tokenization, stemming, lemmatization, and normalization
• 02‑bag‑of‑words‑tfidf — Sparse vector representations and TF-IDF weighting
• 03‑word‑embeddings‑word2vec — Continuous bag-of-words and skip-gram architectures
• 04‑glove‑fasttext‑subword — Pre-trained embeddings and subword information
• 05‑sentiment‑analysis — Binary and multiclass text classification
• 06‑named‑entity‑recognition — Sequence labeling for entity extraction
• 07‑pos‑tagging‑parsing — Part-of-speech tagging and syntactic parsing

Deep Learning for NLP (Lessons 8–13)

These lessons transition from classical methods to neural architectures:

• 08‑cnns‑rnns‑for‑text — Convolutional and recurrent neural networks on sequences
• 09‑sequence‑to‑sequence — Encoder-decoder architectures for transduction tasks
• 10‑attention‑mechanism — Scaled dot-product attention and self-attention
• 11‑machine‑translation — Neural machine translation systems
• 12‑text‑summarization — Extractive and abstractive summarization
• 13‑question‑answering — Reading comprehension and QA architectures

Information Retrieval and Generation (Lessons 14–17)

• 14‑information‑retrieval‑search — BM25, dense retrieval, and vector search
• 15‑topic‑modeling — LDA, NMF, and probabilistic topic models
• 16‑text‑generation‑pre‑transformer — N-gram and neural language models before transformers
• 17‑chatbots‑rule‑to‑neural — Dialogue management evolution from rule-based to neural systems

Multilingual and Modern Tokenization (Lessons 18–21)

• 18‑multilingual‑nlp — Cross-lingual transfer and zero-shot learning
• 19‑subword‑tokenization — Byte-Pair Encoding (BPE), WordPiece, and SentencePiece
• 20‑structured‑outputs‑constrained‑decoding — Grammar-constrained generation and JSON mode
• 21‑nli‑textual‑entailment — Natural language inference and contradiction detection

Embeddings, RAG, and Knowledge Extraction (Lessons 22–26)

• 22‑embedding‑models‑deep‑dive — Dense, sparse, and multi-vector representations
• 23‑chunking‑strategies‑rag — Document splitting for retrieval-augmented generation
• 24‑coreference‑resolution — Pronoun resolution and entity linking across mentions
• 25‑entity‑linking — Disambiguation and knowledge base alignment
• 26‑relation‑extraction‑kg — Triple extraction and knowledge graph construction

LLM Evaluation and Dialogue Systems (Lessons 27–29)

• 27‑llm‑evaluation‑frameworks — RAGAS, DeepEval, G-Eval, and custom metrics
• 28‑long‑context‑evaluation — NIAH, RULER, LongBench, and MRCR benchmarks
• 29‑dialogue‑state‑tracking — Slot-filling and LLM-driven state management

---

Code Examples from the Curriculum

The repository emphasizes "build it from scratch" implementations. Below are representative snippets from early and mid-curriculum lessons.

Basic Tokenization (Lesson 01)

From phases/05-nlp-foundations-to-advanced/01-text-processing/code/main.py, a minimal whitespace tokenizer demonstrates foundational preprocessing:

def simple_tokenizer(text: str) -> list[str]:
    """Naive whitespace-based tokenizer for educational purposes."""
    return text.lower().split()


print(simple_tokenizer("Hello, World! This is a test."))

# Output: ['hello,', 'world!', 'this', 'is', 'a', 'test.']

This implementation intentionally avoids external libraries to illustrate tokenization concepts before introducing NLTK, spaCy, or Hugging Face tokenizers in later lessons.

Sentiment Classification (Lesson 05)

The phases/05-nlp-foundations-to-advanced/05-sentiment-analysis/code/main.py file implements a complete Naive Bayes classifier:

from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer

# Minimal training data

texts = ["I love this movie", "I hate this film", "What a great show"]
labels = [1, 0, 1]  # 1 = positive, 0 = negative

vectorizer = CountVectorizer()
X_train = vectorizer.fit_transform(texts)

classifier = MultinomialNB()
classifier.fit(X_train, labels)


def predict_sentiment(sentence: str) -> str:
    """Classify sentiment of input sentence."""
    X_new = vectorizer.transform([sentence])
    prediction = classifier.predict(X_new)[0]
    return "positive" if prediction == 1 else "negative"


print(predict_sentiment("What a wonderful experience!"))

# Output: positive

Each lesson's code/ directory contains runnable scripts with corresponding unit tests in code/tests/.

---

Key Source Files and Structure

Understanding the repository layout helps navigate the NLP curriculum effectively.

File Path	Purpose
phases/05-nlp-foundations-to-advanced/README.md	Phase overview, learning objectives, and lesson index
phases/05-nlp-foundations-to-advanced/<lesson>/docs/en.md	Comprehensive lesson documentation
phases/05-nlp-foundations-to-advanced/<lesson>/code/main.py	Primary implementation file
phases/05-nlp-foundations-to-advanced/<lesson>/code/tests/	Unit tests (minimum 5 per lesson)
phases/19-capstone-projects/30-bpe-tokenizer-from-scratch/code/main.py	Reference BPE tokenizer implementation
ROADMAP.md	Curriculum completion status and estimated hours
glossary/terms.md	Centralized definitions for NLP terminology

The phases/05-nlp-foundations-to-advanced/ directory follows strict naming conventions: two-digit lesson prefix, descriptive kebab-case name, and standardized subdirectories (docs/, code/, assets/).

---

Summary

The AI Engineering from Scratch NLP curriculum delivers 29 progressive lessons covering:

• Classical NLP: Tokenization, TF-IDF, word embeddings, and syntactic analysis
• Neural architectures: CNNs, RNNs, seq2seq, and attention mechanisms
• Transformer-era techniques: Subword tokenization, multilingual models, and constrained decoding
• Production systems: RAG chunking strategies, knowledge graph construction, and entity resolution
• Evaluation frameworks: LLM benchmarks, long-context testing, and dialogue state tracking

Each lesson pairs theoretical explanation with minimal, runnable Python code in main.py, reinforced by unit tests. The curriculum bridges foundational concepts to state-of-the-art implementations as maintained in the rohitg00/ai-engineering-from-scratch repository.

---

Frequently Asked Questions

How long does it take to complete the NLP curriculum?

The 29 lessons span approximately 120–150 hours of study and implementation, according to estimates in ROADMAP.md. Each lesson includes 2–4 hours of reading, 3–6 hours of coding exercises, and 1–2 hours of review. The capstone projects in Phase 19 (particularly lesson 30's BPE tokenizer) require additional integration time.

Does the curriculum require prior machine learning experience?

Phase 5 assumes completion of Phases 1–4, which cover Python fundamentals, linear algebra, calculus, and classical ML. Lessons 01–07 are accessible with basic Python, while Lessons 08–29 require understanding of neural networks, backpropagation, and PyTorch/TensorFlow basics covered in earlier phases.

Are the NLP implementations production-ready?

The implementations emphasize educational clarity over production optimization. Code is intentionally minimal to illustrate core mechanisms—similar to Andrej Karpathy's "micrograd" approach. The repository notes that production deployments should leverage optimized libraries (Hugging Face Transformers, vLLM, etc.) after understanding the underlying principles.

What hardware is needed for the deep learning lessons?

Lessons 08–13 (CNNs/RNNs) run on CPU, though GPU acceleration speeds training. Lessons 19–29 involving transformer-scale models include memory-efficient implementations and optional cloud deployment guidance. The curriculum specifically addresses hardware constraints in phases/05-nlp-foundations-to-advanced/22-embedding-models-deep-dive/docs/en.md with quantization and distillation techniques.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}