# How ai-memory's Retrieval Architecture Combines FTS5, Entity Matching, and Vector Embeddings

> Discover how ai-memory's retrieval architecture integrates FTS5, entity matching, and vector embeddings with RRF for powerful semantic search. Learn about its robust design.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: architecture
- Published: 2026-09-12

---

**ai-memory's retrieval architecture fuses four independent relevance streams—FTS5 full-text search, entity matching, vector embeddings, and link-graph expansion—into a single ranked result list using Reciprocal Rank Fusion (RRF) with a constant k=60, enabling robust semantic search that degrades gracefully when individual components are unavailable.**

The `akitaonrails/ai-memory` project implements a sophisticated multi-modal retrieval system that combines deterministic and semantic search techniques. Its **hybrid architecture** processes queries through four parallel streams, aggregating results via rank-based fusion rather than score normalization. This design, centered in [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs), allows the system to leverage exact keyword matching, structured entity lookups, dense vector similarity, and graph-based context simultaneously.

## The Four Relevance Streams

The retrieval engine executes four independent searches in parallel, each producing a list of `(page_id, raw_score, rank)` tuples. These streams operate against distinct data structures within the SQLite backend.

### FTS5 Full-Text Search

The **FTS5** stream executes SQLite `MATCH` queries over the `pages_fts` and `observations_fts` tables. Query normalization—handling tokenization, punctuation, and hyphen preservation—occurs in [`crates/ai-memory-store/src/fts_query.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/fts_query.rs). This stream handles exact keyword matches and phrase queries with deterministic ranking.

### Entity Matching

The **entity matching** stream looks up pre-extracted entity rows (e.g., "GitHub", "Rust", identifiers) in the `entities` table. Population happens during ingestion via [`crates/ai-memory-consolidate/src/lint.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/lint.rs), while querying occurs in [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs). This bridges terminology gaps by matching canonical entity names regardless of surface form variations.

### Vector Embeddings

When the query includes a dense vector (`query_vec`), the engine computes **cosine similarity** against stored embeddings in the `embeddings` table. Embeddings are keyed by `(provider, model, dim)` triples and back-filled via [`crates/ai-memory-consolidate/src/embeddings.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/embeddings.rs). The similarity calculation occurs in [`reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/reader.rs), capturing semantic relationships beyond lexical overlap.

### Link-Graph Expansion

Starting from seed pages returned by the other streams, a **breadth-first walk** follows edges derived from markdown `[[link]]` syntax. Each neighbor receives a small rank boost, ensuring contextually related pages surface even when they lack direct query terms. This logic resides in [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs).

## Reciprocal Rank Fusion Implementation

The fusion logic lives in [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs) inside the private method `hybrid_search_inner`. The implementation applies **Reciprocal Rank Fusion (RRF)** to combine the four streams without requiring score calibration across different modalities.

The RRF formula applies a constant `k = 60` (as implemented in lines 45-89 of the source):

```rust
// Conceptual representation of the RRF calculation
let rrf_score = 1.0 / (60.0 + rank as f64);

```

The method normalizes each stream's output to an RRF contribution, sums these contributions per page to yield a **fused score**, then applies a bounded page-authority adjustment for recently authored or pinned pages before returning the top `limit` results (default 60).

## Graceful Degradation and Stream Independence

Because each stream degrades independently, callers can omit any component without breaking the query pipeline:

- **No `query_vec` provided** disables the vector embedding stream entirely
- **Empty `entities` table** results in zero contribution from the entity stream
- The **link-graph walk** always executes from whatever seeds the active streams produce

This design ensures the system remains functional across different deployment configurations, from lightweight keyword-only setups to full semantic search with embeddings.

## Practical Usage Examples

The `Store::hybrid_search` API in [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs) exposes this architecture through a unified interface. The following examples demonstrate different retrieval modes:

```rust
use ai_memory_store::Store;
use ai_memory_store::types::{WorkspaceId, ProjectId};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let store = Store::open("path/to/ai-memory.db").await?;

    // Pure keyword search - FTS5 only
    let fts_hits = store
        .hybrid_search(
            WorkspaceId::default(),
            ProjectId::default(),
            "rust async".to_string(),
            None,               // No query_vec disables embedding stream
            "".to_string(),
            "".to_string(),
            0,
            20,
            None,
        )
        .await?;

    // Semantic search with embeddings
    let embed_vec = my_embedding_provider.encode("how to write async code in rust")?;
    let semantic_hits = store
        .hybrid_search(
            WorkspaceId::default(),
            ProjectId::default(),
            "async rust".to_string(),
            Some(embed_vec),    // Activates cosine similarity stream
            "openai".to_string(),
            "text-embedding-ada-002".to_string(),
            1536,
            20,
            None,
        )
        .await?;

    // Debug view with per-stream rankings
    let explained = store
        .hybrid_search_explained(
            WorkspaceId::default(),
            ProjectId::default(),
            "rust".to_string(),
            None,
            "".to_string(),
            "".to_string(),
            0,
            10,
            None,
        )
        .await?;
    
    for (hit, explain) in explained {
        println!("Page {} – fused score {:.3}", hit.path, hit.score);
        println!("  – FTS rank: {:?}, Entity rank: {:?}, Vector rank: {:?}, Graph rank: {:?}",
                 explain.fts_rank, explain.entity_rank, explain.vector_rank, explain.graph_rank);
    }

    Ok(())
}

```

The HTTP endpoint in [`crates/ai-memory-web/src/routes/api.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-web/src/routes/api.rs) exposes this functionality via the `/search` route, making the hybrid retrieval available to web clients.

## Summary

- **ai-memory's retrieval architecture** combines four parallel streams: FTS5 full-text search, entity matching, vector embeddings, and link-graph expansion.
- **Reciprocal Rank Fusion** in [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs) aggregates ranks using `score = 1 / (60 + rank)`, eliminating the need for cross-modal score normalization.
- The system **degrades gracefully** when embeddings or entities are unavailable, falling back to available streams without code changes.
- **Link-graph expansion** boosts contextually related pages through breadth-first traversal of markdown link syntax.
- The `Store::hybrid_search` API provides unified access to all retrieval modes with optional explanation metadata for debugging.

## Frequently Asked Questions

### What is Reciprocal Rank Fusion (RRF) and why does ai-memory use it?

Reciprocal Rank Fusion is a rank-aggregation technique that combines multiple ordered result lists by assigning each item a score of `1 / (k + rank)`, where `k` is a constant (typically 60) that dampens the impact of lower-ranked items. ai-memory uses RRF in [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs) because it handles missing streams gracefully and does not require normalizing scores across different modalities (text match scores vs. cosine similarity vs. graph distance).

### Can I use ai-memory's retrieval system without vector embeddings?

Yes. The retrieval architecture degrades independently by stream. If you omit the `query_vec` parameter or provide `None` to `Store::hybrid_search`, the vector embedding stream is disabled and the system relies solely on FTS5, entity matching, and link-graph expansion. This allows lightweight deployments without embedding providers.

### How does the link-graph expansion improve search results?

The link-graph expansion performs a breadth-first walk starting from pages returned by other streams, following edges created from markdown `[[link]]` syntax. Each neighboring page receives a small rank boost through the RRF aggregation. This surfaces **contextually related pages** that may not contain the query terms but are linked to highly relevant pages, improving recall for exploratory search scenarios.

### Where is the hybrid search logic implemented in the codebase?

The core hybrid search implementation resides in [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs), specifically within the private method `hybrid_search_inner`. Supporting components include [`crates/ai-memory-store/src/fts_query.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/fts_query.rs) for query normalization, [`crates/ai-memory-consolidate/src/lint.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/lint.rs) for entity extraction, and [`crates/ai-memory-consolidate/src/embeddings.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/src/embeddings.rs) for vector management. The web API endpoint calling this logic is located in [`crates/ai-memory-web/src/routes/api.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-web/src/routes/api.rs).