How Transformers, LLMs, and Agent Engineering Phases Connect in Modern AI

The relationship between Transformers, LLMs, and Agent Engineering is a causal progression where transformer architectures serve as the computational foundation for Large Language Models, which then function as the reasoning core for autonomous agent systems.

The rohitg00/ai-engineering-from-scratch repository structures these concepts into a vertical curriculum of 20 progressive phases, demonstrating how Transformers, LLMs, and Agent Engineering phases form a strict hierarchical dependency. This architectural lineage establishes that mastering transformer attention mechanisms is prerequisite to building performant LLMs, which in turn are required for engineering robust autonomous agents capable of tool use and reasoning.

The Architectural Hierarchy: From Attention Blocks to Agent Loops

The curriculum explicitly maps a causal chain in its README.md phase diagram:

Phase 7 — Transformers → Phase 10 — LLMs from Scratch → Phase 11 — LLM Engineering → Phase 14 — Agent Engineering

This progression reflects how theoretical components become practical systems.

Phase 7: Transformer Architecture as the Foundation

In phases/07-transformers-deep-dive/05-full-transformer/docs/en.md, the curriculum defines the Transformer as the canonical building block for modern AI. This phase teaches the full spectrum of transformer variants:

• Encoder-only architectures (akin to BERT)
• Decoder-only architectures ( GPT-style causal transformers)
• Encoder-decoder architectures (T5-style sequence-to-sequence models)

The implementation details cover modern production-grade modifications including pre-norm layer ordering, RMSNorm stabilization, SwiGLU activation functions, Grouped Query Attention (GQA), and Rotary Position Embeddings (RoPE). These components form the computational substrate that enables the scale and performance required for Large Language Models.

Phase 10: Assembling LLMs from Transformer Blocks

Phase 10 bridges the gap between abstract architecture and concrete models. According to phases/10-llms-from-scratch/01-tokenizers/docs/en.md, this phase demonstrates how to assemble a mini-GPT by stacking the transformer blocks learned in Phase 7.

The critical components include:

1. Byte Pair Encoding (BPE) tokenizers built from first principles
2. Data pipeline engineering for large-scale corpus processing
3. Stacking decoder-only transformer blocks to create the model backbone

This phase transforms the static transformer block into a trainable language model capable of next-token prediction.

Phase 11: Production-Grade LLM Engineering

Between Phase 10 and Phase 14, Phase 11 — LLM Engineering focuses on deploying these models effectively. This includes prompt engineering, context management, Retrieval-Augmented Generation (RAG), and inference optimization techniques. These skills ensure the LLM operates reliably before being wrapped into autonomous systems.

Phase 14: Agent Engineering with ReAct Patterns

Phase 14 treats the LLM as a tool inside an autonomous control loop. As documented in phases/14-agent-engineering/01-the-agent-loop/docs/en.md, the Agent Loop lesson implements a minimal ReAct (Reasoning + Acting) style controller.

The architecture follows this execution pattern:

• The LLM receives a query and generates a reasoning trace
• The output is parsed for tool calls (e.g., search functions, API calls)
• The agent executes external tools and feeds results back into the LLM context
• The loop iterates until a final answer is produced

This demonstrates how Agents rely entirely on LLMs as their cognitive engine, which in turn rely on the transformer attention mechanisms learned in Phase 7.

Implementation: From Transformer Blocks to Agent Execution

The following code illustrates the curriculum's progression from fundamental blocks to autonomous systems.

First, the transformer-based LLM architecture from Phase 7 and 10:

import torch
import torch.nn as nn

class MiniGPT(nn.Module):
    def __init__(self, d_model=128, n_head=4, depth=2):
        super().__init__()
        self.embed = nn.Embedding(50257, d_model)  # vocab size from Phase 10 tokenizer

        self.pos = nn.Parameter(torch.randn(1, 512, d_model))
        self.blocks = nn.ModuleList([
            nn.TransformerDecoderLayer(
                d_model, n_head, 
                dim_feedforward=4*d_model,
                activation='gelu', 
                batch_first=True
            )
            for _ in range(depth)
        ])
        self.lm_head = nn.Linear(d_model, 50257, bias=False)

    def forward(self, ids):
        x = self.embed(ids) + self.pos[:, :ids.size(1)]
        for blk in self.blocks:
            x = blk(tgt=x, memory=None)  # decoder-only, no cross-attention

        return self.lm_head(x)

This model stacks the transformer blocks defined in Phase 7 and initializes them with the tokenization scheme from Phase 10.

Next, the agent loop from Phase 14 that consumes this LLM:

def run_agent(query: str, llm, max_steps=5):
    history = [{"role": "user", "content": query}]
    
    for _ in range(max_steps):
        # Invoke LLM reasoning (simulated here)

        next_token_logits = llm(torch.tensor([[1, 2, 3]]))
        
        # Parse for tool calls and execute

        # In production: detect "Tool:search(query='...')" patterns

        # and route to Python functions

        
        return "agent answer (generated by LLM)"

This minimal implementation mirrors the ReAct pattern found in phases/14-agent-engineering/01-the-agent-loop/docs/en.md, showing how the LLM serves as the reasoning engine within the agent's control flow.

Summary

• Transformers (Phase 7) supply the attention-based architectural primitives—multi-head attention, feed-forward networks, and positional encoding—required for modern language modeling.
• LLMs (Phase 10) are concrete implementations built by stacking transformer blocks, training them on tokenized corpora, and optimizing for next-token prediction.
• Agent Engineering (Phase 14) wraps these LLMs into autonomous systems using ReAct-style loops that enable reasoning, planning, and tool invocation.
• The dependency is strictly hierarchical: you cannot engineer robust agents without understanding LLM behavior, and you cannot build performant LLMs without mastering transformer attention mechanisms.

Frequently Asked Questions

What is the specific relationship between Transformers and LLMs?

Transformers are the architectural foundation, while LLMs are the concrete models built upon that foundation. According to the curriculum in phases/07-transformers-deep-dive/05-full-transformer/docs/en.md, an LLM is essentially a stack of transformer decoder blocks (or encoder-decoder blocks) trained on massive text corpora to predict the next token. Without the attention mechanism and layer normalization techniques defined in Phase 7, the language models in Phase 10 would lack the capacity to model long-range dependencies in text.

Why is Phase 7 (Transformers) a prerequisite for Phase 14 (Agent Engineering)?

Phase 14's agent systems rely on LLMs as their reasoning engine, and those LLMs are built from the transformer blocks defined in Phase 7. If you understand how Grouped Query Attention (GQA) and RMSNorm affect model outputs—as taught in phases/07-transformers-deep-dive/05-full-transformer/docs/en.md—you can better predict why an agent might hallucinate or how to structure its context window for more reliable tool use. The curriculum explicitly positions Phase 7 as foundational because architectural decisions at the transformer level determine the LLM's capabilities that agents later exploit.

How does the ReAct loop in Phase 14 utilize the LLM?

The ReAct loop treats the LLM as both a reasoning engine and a policy controller. As shown in phases/14-agent-engineering/01-the-agent-loop/docs/en.md, the agent prompts the LLM to generate a thought process, then parses that output for structured tool calls (e.g., search, calculate). The LLM's ability to generate these structured outputs depends on the training and architectural features (like RoPE and SwiGLU) from Phase 7. The loop iterates: the LLM reasons, the agent acts, and the results are fed back into the LLM's context until the task is complete.

What specific transformer components are essential for the LLMs built in Phase 10?

The curriculum emphasizes several modern transformer modifications in Phase 7 that are essential for Phase 10's LLM implementations: pre-norm architectures that stabilize gradients in deep networks, Rotary Position Embeddings (RoPE) that generalize to longer sequences than seen during training, and SwiGLU activation functions that improve feed-forward network expressiveness. These components, detailed in phases/07-transformers-deep-dive/05-full-transformer/docs/en.md, enable the mini-GPT built in phases/10-llms-from-scratch/01-tokenizers/docs/en.md to train effectively and generalize beyond its training data.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}