# How to Implement RAG with Dense and Sparse Embeddings: A Complete Guide

> Implement RAG with dense and sparse embeddings. Combine semantic search and BM25, fuse results with RRF, and rerank with cross-encoders for enhanced AI generation.

- Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book)
- Tags: how-to-guide
- Published: 2026-07-26

---

**You can implement a hybrid Retrieval-Augmented Generation (RAG) system by combining dense vector embeddings (semantic search) with sparse lexical embeddings (BM25), fusing results via Reciprocal Rank Fusion (RRF) and optionally reranking with a cross-encoder model.**

The open-source repository `bojieli/ai-agent-book` provides a production-ready implementation of this architecture in its Week 3 projects. By implementing RAG with dense and sparse embeddings together, you create a retrieval pipeline that captures both conceptual meaning and exact keyword matches.

## Why Combine Dense and Sparse Embeddings?

Dense embeddings excel at capturing semantic similarity and cross-lingual paraphrases, while sparse embeddings provide deterministic exact-term matching for rare keywords and proper nouns. When you implement RAG with dense and sparse embeddings, the dense service (BGE-M3) retrieves conceptually related passages even when terminology differs, while the sparse service (BM25) ensures high precision for specific technical terms or identifiers. The hybrid approach uses Reciprocal Rank Fusion to combine these complementary signals without requiring retraining.

## Architecture Overview

The system consists of three independent microservices that communicate via HTTP APIs, as structured in the `projects/week3/` directory.

### Dense Embedding Service

Located in [`projects/week3/dense-embedding/main.py`](https://github.com/bojieli/ai-agent-book/blob/main/projects/week3/dense-embedding/main.py), this service encodes text using the **BGE-M3** model to produce 1024-dimensional vectors. It stores these vectors in an in-memory ANNOY or HNSW index, configurable via [`projects/week3/dense-embedding/config.py`](https://github.com/bojieli/ai-agent-book/blob/main/projects/week3/dense-embedding/config.py). The service exposes endpoints for indexing new documents and performing similarity search using cosine distance on port 4240.

### Sparse Embedding Service

Implemented in [`projects/week3/sparse-embedding/main.py`](https://github.com/bojieli/ai-agent-book/blob/main/projects/week3/sparse-embedding/main.py), this service builds a **BM25** inverted index over tokenized documents. It computes term frequencies and inverse document frequencies to score documents based on lexical overlap with the query. This service handles exact keyword matching efficiently on port 4241.

### Retrieval Pipeline and Fusion

The orchestration layer in [`projects/week3/retrieval-pipeline/main.py`](https://github.com/bojieli/ai-agent-book/blob/main/projects/week3/retrieval-pipeline/main.py) coordinates parallel retrieval from both services. It implements **Reciprocal Rank Fusion (RRF)** to merge results using the formula `Σ 1/(k + rank_i)` where `k=60` by default. This fusion method boosts documents that rank highly in both dense and sparse results while preserving unique hits from either source. The pipeline optionally applies a **bge-reranker-v2** cross-encoder via [`projects/week3/retrieval-pipeline/reranker.py`](https://github.com/bojieli/ai-agent-book/blob/main/projects/week3/retrieval-pipeline/reranker.py) for final relevance scoring before returning results.

## Implementation Details

### Indexing Documents

To populate the hybrid retrieval system, send documents to both embedding services simultaneously. Each document requires a unique `doc_id` and raw text content.

```python
import requests
import pathlib

DOCS = pathlib.Path("data/docs")
DENSE_URL = "http://localhost:4240/index"
SPARSE_URL = "http://localhost:4241/index"

for file_path in DOCS.glob("*.txt"):
    text = file_path.read_text()
    payload = {"doc_id": file_path.stem, "text": text}
    
    # Index in dense vector store

    requests.post(DENSE_URL, json=payload).raise_for_status()
    
    # Index in sparse BM25 store

    requests.post(SPARSE_URL, json=payload).raise_for_status()

```

### Querying with Hybrid Retrieval

The retrieval pipeline accepts a query string and forwards it to both services concurrently. The client utility in [`projects/week3/retrieval-pipeline/retrieval_client.py`](https://github.com/bojieli/ai-agent-book/blob/main/projects/week3/retrieval-pipeline/retrieval_client.py) handles these parallel requests and timeout management.

```python
import requests

QUERY = "multilingual text encoding techniques"
PIPELINE_URL = "http://localhost:4242/query"

response = requests.post(
    PIPELINE_URL,
    json={"query": QUERY, "top_k": 5, "mode": "hybrid"}
)
results = response.json()

# Access individual rankings

dense_hits = results["dense_results"]
sparse_hits = results["sparse_results"]
fused_hits = results["rrf_results"]
final_hits = results["reranked_results"]

```

## Code Examples

### Running the Complete Pipeline

The following example from [`projects/week3/retrieval-pipeline/demo.py`](https://github.com/bojieli/ai-agent-book/blob/main/projects/week3/retrieval-pipeline/demo.py) demonstrates end-to-end indexing and querying:

```python
import requests
import json

def index_corpus(documents):
    """Index documents to both dense and sparse stores."""
    for doc_id, text in documents.items():
        payload = {"doc_id": doc_id, "text": text}
        requests.post("http://localhost:4240/index", json=payload)
        requests.post("http://localhost:4241/index", json=payload)

def hybrid_search(query, top_k=10):
    """Search using both embedding types with RRF fusion."""
    resp = requests.post(
        "http://localhost:4242/query",
        json={"query": query, "top_k": top_k, "mode": "hybrid"}
    )
    return resp.json()

# Example usage

if __name__ == "__main__":
    docs = {
        "doc1": "BGE-M3 supports multilingual dense retrieval.",
        "doc2": "Sparse vectors use BM25 for lexical matching."
    }
    index_corpus(docs)
    results = hybrid_search("multilingual retrieval methods")
    print(json.dumps(results, indent=2))

```

### Switching Retrieval Modes

You can restrict queries to a single retrieval method by changing the `mode` parameter:

```bash

# Dense-only semantic search

curl -X POST http://localhost:4242/query \
  -H "Content-Type: application/json" \
  -d '{"query":"neural network architectures","mode":"dense","top_k":5}'

# Sparse-only lexical search  

curl -X POST http://localhost:4242/query \
  -H "Content-Type: application/json" \
  -d '{"query":"neural network architectures","mode":"sparse","top_k":5}'

```

## Key Files in the Repository

- **[`projects/week3/dense-embedding/main.py`](https://github.com/bojieli/ai-agent-book/blob/main/projects/week3/dense-embedding/main.py)** – HTTP service wrapping BGE-M3 encoding and vector storage (port 4240)
- **[`projects/week3/dense-embedding/config.py`](https://github.com/bojieli/ai-agent-book/blob/main/projects/week3/dense-embedding/config.py)** – Configuration for vector dimension (1024) and index backend (ANNOY/HNSW)
- **[`projects/week3/sparse-embedding/main.py`](https://github.com/bojieli/ai-agent-book/blob/main/projects/week3/sparse-embedding/main.py)** – BM25 indexing and search service (port 4241)
- **[`projects/week3/retrieval-pipeline/main.py`](https://github.com/bojieli/ai-agent-book/blob/main/projects/week3/retrieval-pipeline/main.py)** – Orchestration service handling RRF fusion and reranking (port 4242)
- **[`projects/week3/retrieval-pipeline/retrieval_client.py`](https://github.com/bojieli/ai-agent-book/blob/main/projects/week3/retrieval-pipeline/retrieval_client.py)** – Client utilities for parallel dense and sparse queries
- **[`projects/week3/retrieval-pipeline/reranker.py`](https://github.com/bojieli/ai-agent-book/blob/main/projects/week3/retrieval-pipeline/reranker.py)** – Cross-encoder loader for bge-reranker-v2
- **[`projects/week3/retrieval-pipeline/demo.py`](https://github.com/bojieli/ai-agent-book/blob/main/projects/week3/retrieval-pipeline/demo.py)** – End-to-end demonstration script
- **[`projects/week3/retrieval-pipeline/evaluate.py`](https://github.com/bojieli/ai-agent-book/blob/main/projects/week3/retrieval-pipeline/evaluate.py)** – Benchmarking tool for comparing dense, sparse, and hybrid retrieval performance

## Summary

- **Hybrid retrieval** combines dense embeddings (BGE-M3, 1024-dim) for semantic search with sparse BM25 indices for lexical matching.
- **Reciprocal Rank Fusion (RRF)** merges results using `Σ 1/(60 + rank)`, ensuring documents ranking highly in both systems surface to the top.
- **Three microservices** operate independently: dense embedding (port 4240), sparse embedding (port 4241), and the retrieval pipeline (port 4242) that orchestrates fusion.
- **Optional reranking** with `bge-reranker-v2` provides final relevance scoring after fusion.
- The `bojieli/ai-agent-book` repository provides complete working implementations in `projects/week3/` with configurable backends and evaluation tools.

## Frequently Asked Questions

### What is the difference between dense and sparse embeddings in RAG?

Dense embeddings are continuous vector representations generated by neural networks like BGE-M3 that capture semantic meaning and context, enabling cross-lingual and paraphrase-tolerant search. Sparse embeddings are high-dimensional binary or count vectors (typically BM25) that represent exact term occurrences, excelling at precise keyword matching and handling rare technical terminology. According to the `bojieli/ai-agent-book` source code, dense vectors use cosine similarity while sparse vectors rely on term frequency-inverse document frequency scoring.

### How does Reciprocal Rank Fusion (RRF) combine dense and sparse results?

RRF aggregates rankings by assigning each document a score equal to the sum of reciprocal ranks across all retrieval methods, calculated as `Σ 1/(k + rank_i)` where `k` is a constant (default 60 in this implementation). As implemented in [`projects/week3/retrieval-pipeline/main.py`](https://github.com/bojieli/ai-agent-book/blob/main/projects/week3/retrieval-pipeline/main.py), this formula ensures that documents appearing in both dense and sparse results receive higher scores than those appearing in only one list, while still preserving unique relevant documents from either source.

### Why use BGE-M3 specifically for the dense embedding service?

BGE-M3 (BAAI General Embedding Multilingual) generates 1024-dimensional vectors that support multilingual semantic retrieval and long-context encoding up to 8192 tokens. The [`projects/week3/dense-embedding/main.py`](https://github.com/bojieli/ai-agent-book/blob/main/projects/week3/dense-embedding/main.py) implementation utilizes this model because it provides state-of-the-art performance on cross-lingual benchmarks while maintaining efficient inference speeds suitable for real-time RAG applications.

### Can I deploy only the dense or only the sparse retrieval component?

Yes, the retrieval pipeline supports mode selection via the `mode` parameter (`"dense"`, `"sparse"`, or `"hybrid"`). When set to `"dense"` or `"sparse"`, the system queries only the respective service on port 4240 or 4241, skipping the fusion step. This modularity allows you to deploy lightweight configurations when semantic or lexical search alone satisfies your use case, as demonstrated in the [`evaluate.py`](https://github.com/bojieli/ai-agent-book/blob/main/evaluate.py) benchmarking script.