# How FTS5, Entity-Match RRF, and Vector Similarity Work Together in ai-memory's memory_query

> Discover how FTS5, entity-match RRF, and vector similarity combine in ai-memory's memory_query for powerful hybrid retrieval and unified relevance scoring.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: deep-dive
- Published: 2026-08-25

---

**The `memory_query` function in `akitaonrails/ai-memory` performs hybrid retrieval by fusing FTS5 full-text search, entity-match tokenization, graph-neighbor expansion, and optional vector similarity through Reciprocal Rank Fusion (RRF), summing their contributions as `1/(60 + rank)` to produce a unified relevance score.**

The `memory_query` tool implements a sophisticated multi-stream retrieval system designed to maximize recall and relevance when searching stored memories. This hybrid approach combines traditional lexical matching with modern semantic search capabilities. Understanding the relationship between **FTS5**, **entity-match RRF**, and **vector similarity** requires examining how these independent ranking streams converge through the Reciprocal Rank Fusion algorithm.

## The Four RRF Streams in memory_query

According to [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs) (lines 49-52), the hybrid search aggregates four distinct retrieval streams that operate independently before fusion:

- **FTS5**: SQLite full-text index search for lexical matching against page content
- **Entity-Match**: Token-based matching against normalized terms extracted from pages and stored in the entity table
- **Graph-Neighbour**: Link-graph expansion following page connections to include related content
- **Vector Similarity**: Cosine similarity between embedded queries and stored page embeddings (optional)

Each stream generates its own ranked list of candidate pages. These rankings feed into a unified RRF pipeline where contributions from all active streams are mathematically combined.

## FTS5 and Entity-Match RRF Components

These two streams handle lexical matching through complementary mechanisms that capture different aspects of text relevance.

### FTS5 Full-Text Search

The **FTS5** stream queries the SQLite FTS5 index directly, performing classic full-text search using SQLite's built-in ranking algorithms. As implemented in the `hybrid_search` function around lines 3600-3655 of [`reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/reader.rs), this stream produces a rank `r₁` based on term frequency, proximity, and other lexical signals inherent to the FTS5 engine.

### Entity-Match Tokenization

The **entity-match** stream operates differently by tokenizing queries into lower-cased, alphanumeric tokens (as documented in comments within [`reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/reader.rs)). These normalized tokens match against the `entity` table, which stores terms extracted during page indexing. This approach captures exact lexical matches and normalized entities that FTS5's stemming or tokenization rules might handle differently, generating a separate rank `r₂`.

## Vector Similarity Search

When an embedder is configured, the **vector similarity** stream introduces semantic retrieval capabilities to the hybrid system. The implementation embeds the incoming query (`query_vec`) and computes cosine similarity against stored page embeddings identified by their `(provider, model, dim)` tuples.

This stream generates rank `r₄` based strictly on semantic distance rather than lexical overlap. As noted in [`crates/ai-memory-mcp/src/server.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-mcp/src/server.rs) (line 1810), if no embedder is configured, this stream is omitted entirely and the system falls back to the three-stream version (FTS5 + entity-match + graph).

## How RRF Fuses the Streams

The **Reciprocal Rank Fusion** algorithm combines these independent rankings into a single relevance score. As defined in [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs) (lines 63-66), the system uses the canonical RRF constant **k = 60**.

The fused score calculates as:

```

fused_score = Σ 1/(60 + rank_i)

```

For example, a page ranked #1 in FTS5 contributes `1/61 ≈ 0.0164` to the sum, while a page ranked #3 contributes `1/63 ≈ 0.0159`. Lower ranks (better positions) contribute more heavily, but signals from all streams are strictly additive. A page that scores well on both entity-match and vector similarity receives a higher fused score than a page that excels on only one stream.

After RRF fusion, `PageAuthority::adjust_rank` in [`reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/reader.rs) applies a page-authority multiplier to the fused scores. The final list sorts by these adjusted scores and truncates to the requested `limit`.

## Configuring Hybrid Search in Practice

You control which streams participate through the `memory_query` tool configuration. The following examples demonstrate how to toggle different retrieval mechanisms.

Basic three-stream retrieval (FTS5 + entity-match + graph) without vector search:

```rust
let resp = client
    .call_tool("memory_query", json!({
        "query": "how to reset ai-memory",
        "limit": 10
    }))
    .await?;

```

Enabling the fourth stream by configuring an embedder:

```rust
let resp = client
    .call_tool("memory_query", json!({
        "query": "reset ai‑memory workflow",
        "limit": 10,
        "embedder": {
            "provider": "openai",
            "model": "text-embedding-3-small",
            "dim": 1536
        }
    }))
    .await?;

```

Inspecting individual RRF contributions using the optional reranker:

```rust
let resp = client
    .call_tool("memory_query", json!({
        "query": "reset ai‑memory workflow",
        "limit": 5,
        "embedder": { "provider": "openai", "model": "text-embedding-3-small", "dim": 1536 },
        "reranker": "llm"
    }))
    .await?;
// Response includes explain entries showing:
//   "vector": { "rank": 1, "rrf": 0.017 },
//   "fts5": { "rank": 2, "rrf": 0.016 },
//   "entity_match": { "rank": 4, "rrf": 0.012 }

```

## Summary

- **RRF unifies four streams**: FTS5, entity-match, graph-neighbor, and optional vector similarity each contribute `1/(60 + rank)` to the fused score.
- **Complementary retrieval**: Entity-match captures exact lexical tokens while vector similarity captures semantic meaning; neither stream replaces the other.
- **Additive scoring architecture**: Pages scoring well on multiple streams receive higher fused scores than single-stream high performers.
- **Configurable vector search**: Vector similarity requires explicit embedder configuration; otherwise, the system operates as a three-stream hybrid.
- **Authority adjustment**: RRF scores undergo `PageAuthority::adjust_rank` multiplication before final sorting and truncation to the requested limit.

## Frequently Asked Questions

### What is the RRF constant k in ai-memory?

The RRF constant **k = 60** is hardcoded in [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs) (lines 63-64). This canonical value ensures that lower-ranked (better positioned) results contribute more heavily to the fused score while still allowing signals from higher-ranked items in other streams to influence the final ordering.

### Does vector similarity replace the FTS5 and entity-match streams?

No. The **vector similarity** stream operates in parallel with the lexical streams. According to the source code in [`reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/reader.rs), these are additive components within the RRF pipeline. While vector similarity captures semantic relationships, entity-match ensures exact lexical tokens are recognized, and FTS5 provides traditional full-text relevance scoring.

### How does memory_query handle cases where no embedder is configured?

When no embedder is provided, the vector similarity stream is omitted entirely. As noted in [`crates/ai-memory-mcp/src/server.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-mcp/src/server.rs) (line 1810), the system falls back to a three-stream hybrid combining only FTS5, entity-match, and graph-neighbor RRF without semantic search capabilities.

### What is the role of PageAuthority in the ranking process?

After RRF fusion calculates the initial fused score, `PageAuthority::adjust_rank` (implemented in [`reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/reader.rs)) applies a multiplier based on page authority metrics. This adjusts the fused scores before the final sort and truncation to the requested `limit`, ensuring authoritative pages receive appropriate ranking boosts independent of the retrieval streams' initial rankings.