How RAG Retrieval Algorithms Compare: Sparse, Dense, and Hybrid Search Explained

Term-based retrievers like BM25 offer speed and exact matching, while embedding-based retrievers capture semantic similarity, and hybrid approaches combine both to maximize RAG performance.

Retrieval-augmented generation (RAG) splits the inference pipeline into retrieval → generation, making the choice of retrieval algorithm the single most important factor in answer quality. According to the chiphuyen/aie-book repository, the decision between sparse and dense methods involves fundamental trade-offs between computational cost and semantic understanding. This guide compares the four main families of retrievers using concrete examples from the source material.

The Four Main Families of RAG Retrievers

Term-Based (Sparse) Retrievers

Term-based retrievers rely on classic inverted-index search algorithms like TF-IDF and BM25 that match lexical tokens between queries and documents. As noted in chapter-summaries.md (lines 134-138), these retrievers are "much lighter to implement and can provide strong baselines."

Strengths:

• Very fast to index and query with low memory overhead
• Provides exact matching guarantees without hallucinated relevance
• Works out-of-the-box via backends like Elasticsearch, Apache Solr, or Lucene

Weaknesses:

• Misses semantic matches when queries use different wording or synonyms
• Sensitive to stop-words, spelling variations, and tokenization artifacts
• Performance degrades with short documents containing few overlapping terms

Embedding-Based (Dense) Retrievers

Embedding-based retrievers encode queries and documents into high-dimensional vectors using transformer models, measuring similarity via dot-product or cosine distance. The same source file notes that these are "more computationally intensive but have the potential to outperform term-based algorithms."

Strengths:

• Captures semantic similarity beyond exact word overlap
• Robust to paraphrasing, synonyms, and multilingual variations
• Supports multimodal data when using appropriate encoders

Weaknesses:

• Requires dedicated embedding models (often transformers) resulting in higher compute costs
• Approximate nearest-neighbor (ANN) search may introduce small retrieval errors
• Index construction is expensive for large corpora

Hybrid Retrievers

Hybrid retrievers combine sparse scores (e.g., BM25) with dense scores through linear interpolation or learned rerankers. This approach leverages the precision of term-based methods and the recall of embeddings.

Strengths:

• Often yields the best end-to-end RAG performance in practice
• Tunable blend weights allow domain-specific optimization
• Balances exact matching with semantic understanding

Weaknesses:

• Requires complex infrastructure managing two indexes with synchronization pipelines
• Slightly higher latency due to dual retrieval passes

Reranker-Enhanced Pipelines

Reranker pipelines employ a secondary model—typically a cross-encoder—to re-score top-k candidates from a primary retriever.

Strengths:

• Dramatically improves relevance when primary retrievers are noisy
• Allows fine-grained control with domain-specific training data

Weaknesses:

• Adds extra latency and GPU costs
• Requires labeled relevance data for effective training

Architectural Flow of a RAG Retrieval Pipeline

The standard retrieval sequence follows these steps:

1. Document preprocessing – Chunk source material (e.g., 300-word windows) and store raw text plus pre-computed embeddings for dense retrievers
2. Retriever query – Generate a retrieval query (often the user question itself)
3. Search execution – Execute sparse (BM25), dense (vector similarity), or hybrid (combined scoring) retrieval
4. Reranking (optional) – Run a cross-encoder on the top-k results to produce final ordering
5. Augmentation – Concatenate selected passages to the LLM prompt
6. Generation – Produce the final answer conditioned on retrieved context

Code Implementation Examples

Implementing BM25 with Elasticsearch

from elasticsearch import Elasticsearch

es = Elasticsearch(hosts=["http://localhost:9200"])

def bm25_search(query, index="my_docs", k=5):
    body = {
        "size": k,
        "query": {
            "match": {
                "text": {
                    "query": query,
                    "operator": "and"
                }
            }
        }
    }
    resp = es.search(index=index, body=body)
    return [hit["_source"]["text"] for hit in resp["hits"]["hits"]]

Dense Retrieval with FAISS

import faiss
import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

# Assume `corpus` is a list of raw strings

embeddings = model.encode(corpus, normalize_embeddings=True)

# Build index

dim = embeddings.shape[1]
index = faiss.IndexFlatIP(dim)  # inner-product (cosine) similarity

index.add(embeddings)

def dense_search(query, k=5):
    q_vec = model.encode([query], normalize_embeddings=True)
    D, I = index.search(q_vec, k)  # D: scores, I: indices

    return [corpus[i] for i in I[0]]

Hybrid Retrieval Combining BM25 and Dense Vectors

def hybrid_search(query, k=5, alpha=0.5):
    # Retrieve larger pools from both methods

    bm25_hits = bm25_search(query, k=10)
    dense_hits = dense_search(query, k=10)

    # Linear interpolation on ranks (lower rank = higher relevance)

    combined = {}
    for rank, doc in enumerate(bm25_hits):
        combined[doc] = combined.get(doc, 0) + alpha * (10 - rank)
    for rank, doc in enumerate(dense_hits):
        combined[doc] = combined.get(doc, 0) + (1 - alpha) * (10 - rank)

    # Return top-k by combined score

    return [doc for doc, _ in sorted(
        combined.items(), 
        key=lambda kv: kv[1], 
        reverse=True
    )[:k]]

Reranking with Cross-Encoders

from sentence_transformers import CrossEncoder
import numpy as np

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-12-v2")

def rerank(query, candidates, top_k=3):
    scores = reranker.predict([(query, cand) for cand in candidates])
    sorted_idx = np.argsort(scores)[::-1][:top_k]
    return [candidates[i] for i in sorted_idx]

End-to-End RAG Pipeline

def rag(query):
    # Hybrid retrieval with alpha=0.7 favoring BM25

    candidates = hybrid_search(query, k=10, alpha=0.7)
    
    # Rerank to keep top-3 most relevant

    docs = rerank(query, candidates, top_k=3)
    
    # Build augmented prompt

    prompt = "Answer the question using only the following information:\n\n"
    for d in docs:
        prompt += f"- {d}\n"
    prompt += f"\nQuestion: {query}"
    
    # Send prompt to LLM (implementation dependent)

    return prompt

Selecting the Right Retriever for Your Use Case

Situation	Recommended Retriever
Prototype or small corpus (≤10k docs)	BM25/Elasticsearch – simple, cheap, fast
Semantic search requirements (different phrasing, multilingual)	Dense vector retriever (FAISS, Milvus)
High-recall, high-precision (legal/medical QA)	Hybrid + reranker
Real-time constraints (sub-100ms latency)	Sparse + small dense index, or lightweight cross-encoder
Multimodal data (images + text)	Dense encoders in common vector space

Source File References in the AI Engineering Book

The architectural comparisons in this guide derive from specific files in the chiphuyen/aie-book repository:

• chapter-summaries.md (lines 134-138): Explains the fundamental trade-offs between term-based and embedding-based retrievers
• resources.md (lines 213-218): Lists academic papers on hybrid and vector search algorithms
• study-notes.md (line 5): Quick reference to the RAG and Agents chapter
• ToC.md: Indicates the RAG section resides in Chapter 6 of the book structure

Summary

• BM25 and term-based methods excel in prototyping and exact-match scenarios with minimal infrastructure requirements
• Dense embedding retrievers capture semantic nuance but require significant compute resources and ANN indexes like FAISS or Milvus
• Hybrid approaches typically deliver the highest end-to-end RAG performance by combining lexical precision with semantic recall
• Cross-encoder rerankers provide quality improvements at the cost of additional latency and GPU requirements
• The chiphuyen/aie-book source code confirms that retriever selection should balance computational constraints against the semantic complexity of your domain

Frequently Asked Questions

What is the main difference between sparse and dense retrieval in RAG?

Sparse retrieval uses inverted indexes based on lexical tokens (like BM25), matching exact words between queries and documents. Dense retrieval encodes text into vector embeddings using neural networks, allowing it to match semantically similar content even when wording differs. According to the source repository, sparse methods are lighter to implement while dense methods offer higher performance potential at greater computational cost.

When should I use a hybrid retriever instead of a pure sparse or dense approach?

Use hybrid retrieval when you need both exact keyword matching and semantic understanding. Hybrid approaches are particularly effective for domain-specific applications like legal or medical question-answering, where you must catch exact terminology (sparse strength) while also understanding conceptual variations (dense strength). The combination typically outperforms either method alone, though it requires maintaining two indexes.

How does a reranker improve RAG performance after initial retrieval?

A reranker—typically a cross-encoder model—re-scores the top-k candidates returned by your primary retriever to refine relevance ordering. While initial retrieval prioritizes recall (finding all potentially relevant documents), reranking optimizes precision (selecting the most relevant). This two-stage approach reduces the context window noise sent to the LLM, improving answer quality at the cost of additional inference latency.

What infrastructure do I need for each retriever type?

Term-based retrievers require traditional search engines like Elasticsearch or Apache Solr. Dense retrievers need vector databases such as FAISS, Milvus, or Pinecone plus GPU resources for embedding generation. Hybrid systems combine both infrastructure stacks, while reranker pipelines add transformer serving capabilities (often via HuggingFace or Triton) for the cross-encoder model.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}