LLM Engineering Phases 9-12: From Reinforcement Learning to Multimodal AI

The rohitg00/ai-engineering-from-scratch repository structures its advanced curriculum into four consecutive LLM Engineering phases—Phase 9 (Reinforcement Learning), Phase 10 (LLMs from Scratch), Phase 11 (LLM Engineering), and Phase 12 (Multimodal AI)—each located in the phases/ directory with dedicated README files and runnable code entry points.

These LLM Engineering phases bridge the gap between theoretical foundations and production deployment, guiding practitioners from Markov Decision Processes to multimodal system architecture. According to the repository source code, each phase builds upon the previous to create a complete pipeline for designing, training, and deploying large language model systems.

Phase 9 – Reinforcement Learning: Foundation of RLHF

Phase 9 establishes the mathematical underpinnings necessary for modern alignment techniques like RLHF (Reinforcement Learning from Human Feedback). Located in phases/09-reinforcement-learning/README.md, this phase focuses on Markov Decision Processes, value functions, and policy evaluation.

Core Concepts and Implementation

The curriculum emphasizes agents that learn by interaction. You will implement MDPs (Markov Decision Processes), compute value grids, and analyze how policy quality relates to expected return. These concepts form the bedrock for understanding how language models are fine-tuned using reinforcement learning signals.

The primary entry point for this phase demonstrates state-action-reward loops:

python3 phases/09-reinforcement-learning/01-mdps-states-actions-rewards/code/main.py

Running this script initializes the reinforcement learning environment and executes the foundational algorithms implemented in the repository.

Phase 10 – LLMs from Scratch: Tokenization to Training

Phase 10 transitions from theoretical RL to practical language model construction. As documented in phases/10-llms-from-scratch/README.md, this phase covers tokenization, data preprocessing, model architecture design, and training loops for tiny LLMs.

Building the Tokenizer and Model

You will write a tokenizer from the ground up, construct the neural architecture, and train a miniature LLM. The phase also includes analysis of compression ratios and data efficiency metrics critical for understanding modern transformer scaling laws.

Execute the tokenizer implementation using:

python3 phases/10-llms-from-scratch/01-tokenizers/code/main.py

This script in phases/10-llms-from-scratch/01-tokenizers/code/main.py serves as the gateway to understanding how raw text transforms into model-ready tensors.

Phase 11 – LLM Engineering: Production Deployment

Phase 11, detailed in phases/11-llm-engineering/README.md, focuses specifically on LLM Engineering—the discipline of moving from pretrained checkpoints to production-ready services. This phase covers prompt engineering strategies, fine-tuning methodologies, inference scaling techniques, and deployment patterns.

Prompt Engineering and Fine-tuning

The curriculum teaches structured approaches to prompt engineering and supervised fine-tuning (SFT). You will evaluate different prompt strategies for accuracy and latency, then implement parameter-efficient fine-tuning on pretrained models to adapt them for specific downstream tasks.

Inference Scaling and API Deployment

Practical deployment skills include creating API-served inference services and optimizing batch sizes for throughput. The phase addresses the engineering challenges of serving large models at scale, including model sharding and request queuing strategies.

Start the prompt engineering exercises with:

python3 phases/11-llm-engineering/01-prompt-engineering/code/main.py

This entry point in phases/11-llm-engineering/01-prompt-engineering/code/main.py demonstrates the core inference patterns used throughout the phase.

Phase 12 – Multimodal AI: Cross-Modal Integration

Phase 12 culminates the curriculum with Multimodal AI, located in phases/12-multimodal-ai/README.md. This phase integrates text, vision, and audio into unified model architectures using multimodal embeddings and cross-modal attention mechanisms.

Multimodal Architectures

You will build a multimodal encoder-decoder capable of processing heterogeneous data types. The curriculum covers zero-shot image-captioning tasks and audio-text alignment techniques, teaching how to align embedding spaces across different modalities.

The practical implementation begins with:

python3 phases/12-multimodal-ai/01-multimodal-embeddings/code/main.py

Located at phases/12-multimodal-ai/01-multimodal-embeddings/code/main.py, this script initializes the multimodal embedding layers that bridge vision and language representations.

Repository Navigation and Execution

All four LLM Engineering phases follow a consistent directory structure within the phases/ folder. Each phase contains a README.md file explaining theoretical concepts and a code/ subdirectory containing runnable implementations.

To execute the complete curriculum sequentially:

1. Phase 9: Run main.py in 01-mdps-states-actions-rewards/code/ to master reinforcement learning foundations
2. Phase 10: Execute the tokenizer script in 01-tokenizers/code/ to understand LLM internals
3. Phase 11: Launch the prompt engineering module in 01-prompt-engineering/code/ for deployment skills
4. Phase 12: Start with 01-multimodal-embeddings/code/ to build cross-modal systems

Summary

The LLM Engineering phases 9-12 in rohitg00/ai-engineering-from-scratch provide a systematic progression from theoretical fundamentals to production systems:

• Phase 9 establishes reinforcement learning foundations necessary for RLHF alignment techniques
• Phase 10 teaches end-to-end language model construction, from tokenization to training loops
• Phase 11 covers production engineering, including prompt optimization, fine-tuning, and API deployment
• Phase 12 integrates multimodal capabilities, combining text with vision and audio through cross-modal attention

Each phase contains specific entry-point scripts runnable from the repository root, enabling hands-on experimentation with the core concepts.

Frequently Asked Questions

What prerequisites are needed for LLM Engineering Phase 11?

According to the repository structure, learners should complete Phase 10 (LLMs from Scratch) before advancing to Phase 11. Understanding tokenization and basic training loops from phases/10-llms-from-scratch/ provides the necessary context for prompt engineering and fine-tuning operations covered in Phase 11.

How does Phase 9 relate to modern LLM training?

Phase 9 covers Markov Decision Processes and policy evaluation, which form the mathematical foundation for RLHF (Reinforcement Learning from Human Feedback). The value functions and policy optimization techniques implemented in phases/09-reinforcement-learning/ directly translate to the alignment methods used in production language models like GPT-4 and Claude.

What distinguishes Phase 10 from Phase 11 in the curriculum?

Phase 10 focuses on building language models from scratch, covering architecture and training in phases/10-llms-from-scratch/, while Phase 11 focuses on engineering existing models—optimizing inference, designing prompts, and deploying services. Phase 10 is about creation; Phase 11 is about productionization and scaling.

Which phase should I start with for multimodal AI applications?

Start with Phase 12, but ensure you have completed Phase 11 first. The multimodal implementations in phases/12-multimodal-ai/ assume familiarity with transformer architectures and inference patterns established in the earlier LLM Engineering phases, particularly the embedding handling and attention mechanisms covered in Phase 11.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}