# ai‑memory Retrieval Pipeline: FTS5 + Entity‑Match RRF + Graph‑Neighbor RRF Explained

> Explore the ai-memory retrieval pipeline combining FTS5, entity-match RRF, and graph-neighbor RRF. Discover how this powerful system surfaces your most relevant wiki pages.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: deep-dive
- Published: 2026-08-30

---

**The ai‑memory retrieval pipeline combines full‑text search (FTS5), entity‑match ranking, and graph‑neighbor proximity via Reciprocal Rank Fusion (RRF) to surface the most relevant wiki pages.**

ai‑memory, the open‑source AI‑native knowledge base, implements a multi‑stream retrieval system that merges independent search signals into a single ranked result list. This article breaks down exactly how the FTS5, entity‑match, and graph‑neighbor streams work together—and where vector search fits in.

## The Three Core Retrieval Streams

The default pipeline runs three complementary search strategies in parallel. Each stream captures a different semantic relationship between query and content.

### FTS5 Full‑Text Search

The **FTS5 stream** performs traditional keyword search against SQLite's FTS5 virtual table.

- Targets the `pages_fts` table indexed over page bodies and front‑matter
- Returns results ranked by BM25 relevance scoring
- Implemented in `crates/ai‑memory‑store/src/fts_query.rs` via `prepare_fts5_query`

```bash
ai-memory search "how to reset a project"

```

This command triggers the full pipeline, with FTS5 as the primary signal for exact and stemmed term matches.

### Entity‑Match RRF

The **entity‑match stream** expands recall by matching normalized entity tokens.

- Query is split into lower‑cased alphanumeric words (entity tokens)
- Searches the `pages_entity` table for pages containing overlapping entities
- Captures semantic similarity even when exact wording differs

In `crates/ai‑memory‑store/src/reader.rs`, the `entity_match_tokens` helper (lines 401‑410) builds token lists, and `entity_match_query` executes the SQL lookup. This stream is particularly effective for queries that name concepts without using the exact phrasing from source documents.

### Graph‑Neighbor RRF

The **graph‑neighbor stream** leverages the wiki link structure for contextual relevance.

- Traverses the page link graph one hop outward from initial hits
- Ranks neighbors by how many of their linked pages appeared in prior streams
- Converts this structural proximity into RRF scores

Implementation resides in `crates/ai‑memory‑store/src/reader.rs` at `graph_neighbour_ranking` (lines 530‑560). This captures the intuition that pages linked from relevant pages are often relevant themselves—a core principle of graph‑based retrieval.

## The RRF Fusion Step

All streams converge through **Reciprocal Rank Fusion**. The merge logic appears in `crates/ai‑memory‑store/src/reader.rs` (comment at lines 126‑130).

The RRF formula: `score = Σ 1 / (k + rank)` where `k` is the rank offset constant (typically 60).

Processing sequence:

1. Execute FTS5, entity‑match, and graph‑neighbor streams
2. Collect per‑stream rank positions for every page
3. Compute combined RRF score across all contributing streams
4. Sort by final score and return top results

This fusion approach prevents any single stream from dominating results while preserving each stream's relative ordering information.

## Optional Vector RRF Stream

When an embedding provider is configured, ai‑memory adds a **fourth vector similarity stream**.

- Query embedding generated via `ai‑memory‑llm/src/embedding.rs` (`embed` method)
- Similarity search against `pages_vector` table
- Results merged into the same RRF computation

Configure via `ai‑memory.toml` to enable:

```toml
[llm]
embedding_provider = "openai"

```

Once enabled, the same `search` call automatically incorporates vector RRF without code changes.

## Programmatic Usage

Access the pipeline directly through the `Reader` API:

```rust
use ai_memory_store::Reader;
use ai_memory_core::ProjectId;

let reader = Reader::new(pool.clone())?;
let project = ProjectId::new("my‑project");

// Runs FTS5 + entity‑match + graph‑neighbor (+ vector if configured)
let results = reader.search(project, "reset project", /*limit=*/20).await?;

for hit in results {
    println!("{} (score {:.3})", hit.page_path, hit.score);
}

```

The `search` method orchestrates all streams and returns pages with their combined RRF scores.

## Key Implementation Files

| File | Responsibility |
|------|--------------|
| `crates/ai‑memory‑store/src/fts_query.rs` | FTS5 query construction (`prepare_fts5_query`) |
| `crates/ai‑memory‑store/src/reader.rs` | Core streams (FTS5, entity‑match, graph‑neighbor) and RRF fusion |
| `crates/ai‑memory‑llm/src/embedding.rs` | Optional embedding generation and vector retrieval |
| `crates/ai‑memory‑mcp/src/server.rs` | HTTP `/search` endpoint (documented at line 1822) |

## Summary

- **FTS5** provides fast, exact‑match retrieval with BM25 scoring
- **Entity‑match** expands recall through normalized token overlap
- **Graph‑neighbor** injects link‑structure proximity into rankings
- **RRF fusion** combines these heterogeneous signals into unified scores
- **Vector search** optionally adds semantic similarity when embeddings are available

The pipeline is designed to balance precision, recall, and computational efficiency—no single technique dominates, and each compensates for the others' limitations.

## Frequently Asked Questions

### What is RRF and why does ai‑memory use it?

**RRF (Reciprocal Rank Fusion)** is a score‑free method for combining ranked lists from multiple retrieval systems. ai‑memory uses it because it requires no score normalization between streams—FTS5 BM25 scores, entity counts, and graph proximity measures operate on incomparable scales. RRF only needs rank positions, making it robust and stream‑agnostic.

### Can I disable individual streams in the retrieval pipeline?

The standard `search` method runs all three core streams automatically. For custom behavior, you would need to fork the [`reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/reader.rs) implementation or construct queries directly against the underlying tables (`pages_fts`, `pages_entity`, and the graph link table).

### How does graph‑neighbor ranking handle pages with no outbound links?

Pages without links contribute no neighbors to the stream but can still appear as neighbors themselves if linked from high‑ranking pages. The `graph_neighbour_ranking` function tracks in‑link counts from the initial result set, so isolated but frequently referenced pages still receive RRF scores through incoming citations.

### What embedding models work with the vector RRF stream?

Any provider implementing the embedding trait in `crates/ai‑memory‑llm/src/embedding.rs` is supported. The repository includes OpenAI and local model integrations; custom providers require implementing the `embed` method to return query vectors compatible with the `pages_vector` table schema.