NLP Foundations to Advanced Phase 5: Complete 25-Lesson Curriculum (rohitg00/ai-engineering-from-scratch)

Phase 5 covers 25 self-contained lessons progressing from classical text preprocessing (tokenization, TF-IDF, word embeddings) through neural sequence models, multilingual NLP, and modern LLM evaluation frameworks including long-context benchmarks.

The rohitg00/ai-engineering-from-scratch repository delivers a systematic curriculum for production-grade AI engineering. According to the source code organization in phases/05-nlp-foundations-to-advanced/, this phase bridges traditional natural language processing pipelines with state-of-the-art transformer-based techniques. Each lesson ships with runnable implementations, deterministic unit tests, and skill artifacts.

Text Preprocessing and Classical Representations

The curriculum begins with foundational text processing and vector-space models.

• 01-text-processing (docs/en.md): Basic tokenization, stemming, lemmatization, and preprocessing pitfalls. Implementation located at phases/05-nlp-foundations-to-advanced/01-text-processing/code/main.py.

• 02-bag-of-words-tfidf (docs/en.md): Classical vector-space representations and TF-IDF weighting schemes for text classification tasks.

• 03-word-embeddings-word2vec (docs/en.md): Learning dense word vectors from raw corpora using the skip-gram and CBOW architectures.

• 04-glove-fasttext-subword (docs/en.md): Pre-trained embedding matrices (GloVe, FastText) and sub-word representation strategies for handling out-of-vocabulary terms.

Core NLP Tasks and Linguistic Analysis

These lessons cover structured prediction tasks essential for information extraction.

• 05-sentiment-analysis (docs/en.md): Classical text classification methodologies and the limitations of naive baseline approaches.

• 06-named-entity-recognition (docs/en.md): Sequence tagging models for identifying entities (people, organizations, locations) in unstructured text.

• 07-pos-tagging-parsing (docs/en.md): Part-of-speech tagging pipelines and dependency parsing algorithms for syntactic analysis.

Neural Architectures and Sequence Modeling

This section introduces the neural building blocks that power modern NLP.

• 09-sequence-to-sequence (docs/en.md): Encoder-decoder architectures, teacher forcing strategies, and beam search decoding algorithms. Note: Lesson numbering skips 08 intentionally per repository conventions.

• 10-attention-mechanism (docs/en.md): Implementation of scaled dot-product attention and its variants. The reference code demonstrates the core computation:

# phases/05-nlp-foundations-to-advanced/10-attention-mechanism/code/main.py

def scaled_dot_product_attention(q, k, v, mask=None):
    dk = q.shape[-1]
    scores = (q @ k.transpose(-2, -1)) / math.sqrt(dk)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, -1e9)
    attn = torch.softmax(scores, dim=-1)
    return attn @ v

• 11-machine-translation (docs/en.md): Neural machine translation foundations, covering BLEU evaluation metrics and parallel dataset pipelines.

Knowledge Systems, Retrieval, and Extraction

These lessons address information retrieval and knowledge graph construction.

• 13-question-answering (docs/en.md): Building extractive and generative QA systems with Retrieval-Augmented Generation (RAG) architectures.

• 14-information-retrieval-search (docs/en.md): Classical IR models (BM25) versus dense retrieval methods and modern neural search implementations.

• 15-topic-modeling (docs/en.md): Unsupervised discovery of latent topics using LDA (Latent Dirichlet Allocation) and NMF (Non-negative Matrix Factorization).

• 24-coreference-resolution (docs/en.md): Algorithms for linking mentions referring to the same real-world entity across documents.

• 25-entity-linking (docs/en.md): Disambiguating entity mentions against structured knowledge bases (e.g., Wikipedia).

• 26-relation-extraction-kg (docs/en.md): Constructing knowledge graphs from text using pattern-based and supervised relation extraction techniques.

Generation, Translation, and Conversational AI

Covering language generation paradigms from statistical models to neural dialogue systems.

• 16-text-generation-pre-transformer (docs/en.md): N-gram language models, smoothing techniques, and perplexity-based evaluation metrics.

• 17-chatbots-rule-to-neural (docs/en.md): Evolution from rule-based dialog systems to modern neural conversational agents.

• 29-dialogue-state-tracking (docs/en.md): Structured tracking of user intents and slots in task-oriented conversational agents.

Multilingual NLP and Modern Tokenization

Addressing cross-lingual transfer and sub-word algorithms essential for transformer models.

• 18-multilingual-nlp (docs/en.md): Zero-shot cross-lingual transfer, multilingual fine-tuning strategies, and cross-lingual evaluation protocols.

• 19-subword-tokenization (docs/en.md): Algorithmic implementations of Byte-Pair Encoding (BPE), WordPiece, and SentencePiece. The lesson includes a from-scratch BPE trainer:

# phases/05-nlp-foundations-to-advanced/19-subword-tokenization/code/main.py

def learn_bpe(corpus: List[str], vocab_size: int) -> List[str]:
    """Greedy learning of BPE merges."""
    tokens = [list(word) + ['</w>'] for word in corpus]
    while len(vocab) < vocab_size:
        pairs = Counter(pair for word in tokens for pair in zip(word, word[1:]))
        most_common = pairs.most_common(1)[0][0]
        # merge the most common pair …

Structured Outputs and Reasoning

Advanced techniques for controlling generation and testing reasoning capabilities.

• 20-structured-outputs-constrained-decoding (docs/en.md): Constraining language model generation using JSON schemas and finite-state automata.

• 21-nli-textual-entailment (docs/en.md): Natural Language Inference (NLI) as a testbed for evaluating textual reasoning and entailment recognition.

LLM Evaluation and Long-Context Methods

Modern evaluation frameworks specific to large language models.

• 27-llm-evaluation-frameworks (docs/en.md): Comprehensive metrics for assessing faithfulness, relevance, and G-Eval automated evaluation for generative models.

• 28-long-context-evaluation (docs/en.md): Benchmarking very long input contexts using NIAH (Needle-in-a-Haystack), RULER, and LongBench protocols:

# phases/05-nlp-foundations-to-advanced/28-long-context-evaluation/code/main.py

def nia_h_eval(model, prompt, haystack):
    """Insert a long filler into the prompt and check if model still
    retrieves the relevant answer."""
    long_prompt = prompt + " " + " filler" * 1000
    return model.generate(long_prompt) == model.generate(prompt)

Repository Structure and Learning Artifacts

Every lesson in phases/05-nlp-foundations-to-advanced/ follows a standardized pedagogical structure:

• README.md (phase root): High-level overview and syllabus linking all 25 lessons.
• <lesson-slug>/docs/en.md: Comprehensive narrative, learning objectives, and "Build It" implementation instructions.
• <lesson-slug>/code/main.py: Minimal, self-contained reference implementation (typically Python).
• <lesson-slug>/code/tests.py: Five or more deterministic unit tests verifying implementation correctness.
• <lesson-slug>/outputs/skill-*.md: Reusable skill artifacts for downstream project integration.
• <lesson-slug>/quiz.json: Six-question assessment (pre-check-post) for knowledge validation.

Summary

• Phase 5 contains 25 lessons spanning classical NLP (lessons 01-07), neural architectures (09-11), knowledge extraction (13-15, 24-26), and modern LLM evaluation (27-28).
• The curriculum intentionally skips lesson numbers 08, 12, 22, and 23 to reserve space for future content or merged material.
• Each lesson provides runnable code in main.py, verification via tests.py, and theoretical documentation in docs/en.md.
• Advanced topics include constrained decoding, multilingual zero-shot transfer, and long-context benchmarking (NIAH/RULER).

Frequently Asked Questions

What programming languages are used in Phase 5?

The primary language is Python, as evidenced by the main.py files throughout phases/05-nlp-foundations-to-advanced/. Some lessons may include supplementary TypeScript implementations (main.ts), but all core algorithms and test suites are written in Python 3.

Does Phase 5 cover transformer architectures from scratch?

Yes. While the phase focuses on "foundations to advanced," it includes the prerequisite neural components: lesson 09-sequence-to-sequence covers encoder-decoder models, and lesson 10-attention-mechanism implements scaled dot-product attention—the core operation powering transformer architectures. These provide the conceptual foundation for understanding modern transformer models.

How are the lessons organized for self-study?

Each lesson follows a consistent four-part structure: (1) narrative documentation explaining concepts, (2) runnable reference code, (3) deterministic unit tests to verify your implementation, and (4) a quiz.json for self-assessment. Additionally, skill-*.md artifacts allow learners to export specific techniques (like BPE tokenization or NER tagging) directly into production projects.

Are there gaps in the lesson numbering?

Yes. The repository uses intentional numbering gaps (specifically lessons 08, 12, 22, and 23) as placeholders for future content or to indicate where lessons were consolidated. The 25 available lessons (01-07, 09-11, 13-21, 24-29) form a complete, unbroken curriculum from text preprocessing to LLM evaluation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}