How Retrieval-Augmented Generation (RAG) Combines Embeddings and Language Models: A Complete Technical Guide

Retrieval-Augmented Generation (RAG) augments a generative language model with an external knowledge base by converting documents into dense vector embeddings, retrieving relevant chunks via similarity search, and injecting them into the LLM's context window to produce grounded, up-to-date responses.

This article examines the implementation of Retrieval-Augmented Generation in the rohitg00/ai-engineering-from-scratch repository, where the RAG pipeline is built from first principles using modular Python components. Understanding how embeddings and language models interact in this architecture is essential for building production-ready AI systems that can reason over private or continually updated data.

The Five-Stage RAG Architecture

According to the source code in phases/11-llm-engineering/06-rag/code/main.py, the RAG pipeline consists of five tightly coupled stages that bridge static knowledge stores with generative capabilities. The embedding model handles the deterministic transformation of text into vectors, while the LLM manages non-deterministic generation and reasoning.

• Document Ingestion & Chunking: Raw texts are split into manageable segments (100–500 tokens) to balance context granularity with retrieval precision.
• Embedding Generation: A lightweight transformer maps each chunk into high-dimensional vectors stored in a vector database.
• Query Embedding: User prompts undergo the same vectorization process as the corpus.
• Similarity Search: Nearest-neighbor algorithms retrieve the top-k most relevant chunks using cosine similarity or inner product metrics.
• Augmented Generation: Retrieved chunks are concatenated with the original query to constrain the LLM's output to factual, retrieved content.

Loading and Chunking Documents

The ingestion phase begins in phases/11-llm-engineering/06-rag/code/main.py (lines 45–62), where documents are loaded and segmented using a simple sliding-window approach. This preprocessing ensures that embedding models receive text within their context windows while preserving semantic boundaries.

from pathlib import Path

def load_and_chunk(path: Path, chunk_size: int = 300):
    text = Path(path).read_text()
    # naive splitter – split on newlines, then on spaces

    chunks = [text[i:i+chunk_size] for i in range(0, len(text), chunk_size)]
    return chunks

Source: phases/11-llm-engineering/06-rag/code/main.py#L45-L62

Chunking strategy directly impacts retrieval quality. Smaller chunks yield higher relevance precision but may sacrifice broader context, while larger chunks capture more context but dilute specific signal.

Generating Dense Vector Embeddings

Each chunk is converted to a dense vector using the embed() function defined in phases/11-llm-engineering/04-embeddings/code/embeddings.py. This abstraction layer wraps a sentence-transformer model such as sentence-transformers/all-MiniLM-L6-v2, producing fixed-size vectors that encode semantic meaning.

from embeddings import embed

# lines 78-84 in main.py

chunk_embeddings = [embed(chunk) for chunk in chunks]

Source: phases/11-llm-engineering/06-rag/code/main.py#L78-L84

The embedding step is computationally efficient compared to LLM inference, allowing millions of documents to be vectorized offline and stored for rapid retrieval without invoking the expensive generative model.

Building the Vector Store with FAISS

Once embeddings are generated, the system constructs an in-memory FAISS index for high-performance similarity search. In phases/11-llm-engineering/06-rag/code/main.py (lines 90–98), an IndexFlatL2 is initialized and populated with the stacked chunk vectors.

import faiss
import numpy as np

dim = len(chunk_embeddings[0])
index = faiss.IndexFlatL2(dim)
index.add(np.stack(chunk_embeddings))

Source: phases/11-llm-engineering/06-rag/code/main.py#L90-L98

FAISS optimizes the search operation using spatial partitioning and quantization techniques, enabling millisecond-scale retrieval across millions of vectors even on CPU-only environments.

Retrieval: Similarity Search at Inference Time

When a user submits a query, the system embeds the query using the same model applied during indexing, then executes a nearest-neighbor lookup. The retrieve() function in phases/11-llm-engineering/06-rag/code/main.py (lines 110–117) demonstrates this bi-encoder architecture.

def retrieve(query: str, k: int = 5):
    q_vec = embed(query)
    distances, ids = index.search(np.array([q_vec]), k)
    return [chunks[i] for i in ids[0]]

Source: phases/11-llm-engineering/06-rag/code/main.py#L110-L117

This retrieval stage acts as a dynamic knowledge filter, selecting only the passages statistically most likely to contain the answer before any generation occurs.

Augmented Generation: Prompt Assembly and LLM Inference

The final stage combines retrieved evidence with the user question. In phases/11-llm-engineering/06-rag/code/main.py (lines 130–138), the generate_answer() function constructs a structured prompt that includes a system instruction, the retrieved context, and the original query.

def generate_answer(query: str):
    retrieved_chunks = retrieve(query)
    prompt = "\n\n".join([
        "You are a helpful assistant. Use the following context:",
        *retrieved_chunks,
        f"Question: {query}"
    ])
    return llm.generate(prompt)   # wrapper around an OpenAI or Claude model

Source: phases/11-llm-engineering/06-rag/code/main.py#L130-L138

By grounding the LLM in retrieved snippets rather than parametric knowledge alone, RAG eliminates hallucinations on factual queries and extends the model's effective knowledge boundary to include documents ingested after the LLM's training cutoff.

Summary

• RAG separates knowledge storage from generation: Embeddings handle semantic search deterministically, while LLMs focus on reasoning and fluency.
• The pipeline is modular: Each stage—chunking, embedding, indexing, retrieval, and generation—is implemented as discrete functions in phases/11-llm-engineering/06-rag/code/main.py.
• FAISS provides scalable search: The vector store enables efficient nearest-neighbor lookups without retraining the underlying language model.
• Prompt engineering connects components: Retrieved chunks are injected into the context window via structured prompting to constrain outputs to factual grounding.
• Code artifacts document the architecture: The repository includes skill files like phases/11-llm-engineering/06-rag/outputs/skill-rag-pipeline.md that formalize this five-stage workflow for reproducibility.

Frequently Asked Questions

How do embeddings enable semantic search in RAG?

Embeddings map discrete text into continuous vector spaces where semantic similarity correlates with geometric proximity. When the embedding model encodes both documents and queries into the same high-dimensional space, mathematical distance metrics like cosine similarity or L2 norm accurately identify conceptually related content regardless of lexical overlap.

Why separate chunking from embedding generation?

Chunking occurs before embedding to ensure each vector represents a coherent semantic unit that fits within the embedding model's input constraints (typically 512 tokens). Processing individual chunks rather than entire documents allows granular retrieval—enabling the system to surface specific paragraphs relevant to a query rather than returning whole documents that may contain irrelevant information.

What role does FAISS play in the RAG pipeline?

FAISS (Facebook AI Similarity Search) serves as the vector database that indexes pre-computed embeddings and executes efficient approximate nearest-neighbor searches. By storing vectors in optimized data structures like IndexFlatL2, FAISS reduces retrieval latency from linear to sub-linear time complexity, making real-time RAG feasible even with millions of indexed documents.

Can the LLM be swapped without re-indexing the embeddings?

Yes. The embedding layer and vector store are decoupled from the generative model. You can replace the LLM (e.g., switching from GPT-3.5 to Claude or a local Llama model) without regenerating embeddings or rebuilding the FAISS index, provided the new model accepts the same prompt format. This modularity allows independent optimization of retrieval accuracy and generation quality.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}