How ai-memory's Retrieval Architecture Combines FTS5, Entity Matching, and Vector Embeddings

ai-memory's retrieval architecture fuses four independent relevance streams—FTS5 full-text search, entity matching, vector embeddings, and link-graph expansion—into a single ranked result list using Reciprocal Rank Fusion (RRF) with a constant k=60, enabling robust semantic search that degrades gracefully when individual components are unavailable.

The akitaonrails/ai-memory project implements a sophisticated multi-modal retrieval system that combines deterministic and semantic search techniques. Its hybrid architecture processes queries through four parallel streams, aggregating results via rank-based fusion rather than score normalization. This design, centered in crates/ai-memory-store/src/reader.rs, allows the system to leverage exact keyword matching, structured entity lookups, dense vector similarity, and graph-based context simultaneously.

The Four Relevance Streams

The retrieval engine executes four independent searches in parallel, each producing a list of (page_id, raw_score, rank) tuples. These streams operate against distinct data structures within the SQLite backend.

FTS5 Full-Text Search

The FTS5 stream executes SQLite MATCH queries over the pages_fts and observations_fts tables. Query normalization—handling tokenization, punctuation, and hyphen preservation—occurs in crates/ai-memory-store/src/fts_query.rs. This stream handles exact keyword matches and phrase queries with deterministic ranking.

Entity Matching

The entity matching stream looks up pre-extracted entity rows (e.g., "GitHub", "Rust", identifiers) in the entities table. Population happens during ingestion via crates/ai-memory-consolidate/src/lint.rs, while querying occurs in crates/ai-memory-store/src/reader.rs. This bridges terminology gaps by matching canonical entity names regardless of surface form variations.

Vector Embeddings

When the query includes a dense vector (query_vec), the engine computes cosine similarity against stored embeddings in the embeddings table. Embeddings are keyed by (provider, model, dim) triples and back-filled via crates/ai-memory-consolidate/src/embeddings.rs. The similarity calculation occurs in reader.rs, capturing semantic relationships beyond lexical overlap.

Link-Graph Expansion

Starting from seed pages returned by the other streams, a breadth-first walk follows edges derived from markdown [[link]] syntax. Each neighbor receives a small rank boost, ensuring contextually related pages surface even when they lack direct query terms. This logic resides in crates/ai-memory-store/src/reader.rs.

Reciprocal Rank Fusion Implementation

The fusion logic lives in crates/ai-memory-store/src/reader.rs inside the private method hybrid_search_inner. The implementation applies Reciprocal Rank Fusion (RRF) to combine the four streams without requiring score calibration across different modalities.

The RRF formula applies a constant k = 60 (as implemented in lines 45-89 of the source):

// Conceptual representation of the RRF calculation
let rrf_score = 1.0 / (60.0 + rank as f64);

The method normalizes each stream's output to an RRF contribution, sums these contributions per page to yield a fused score, then applies a bounded page-authority adjustment for recently authored or pinned pages before returning the top limit results (default 60).

Graceful Degradation and Stream Independence

Because each stream degrades independently, callers can omit any component without breaking the query pipeline:

• No query_vec provided disables the vector embedding stream entirely
• Empty entities table results in zero contribution from the entity stream
• The link-graph walk always executes from whatever seeds the active streams produce

This design ensures the system remains functional across different deployment configurations, from lightweight keyword-only setups to full semantic search with embeddings.

Practical Usage Examples

The Store::hybrid_search API in crates/ai-memory-store/src/reader.rs exposes this architecture through a unified interface. The following examples demonstrate different retrieval modes:

use ai_memory_store::Store;
use ai_memory_store::types::{WorkspaceId, ProjectId};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let store = Store::open("path/to/ai-memory.db").await?;

    // Pure keyword search - FTS5 only
    let fts_hits = store
        .hybrid_search(
            WorkspaceId::default(),
            ProjectId::default(),
            "rust async".to_string(),
            None,               // No query_vec disables embedding stream
            "".to_string(),
            "".to_string(),
            0,
            20,
            None,
        )
        .await?;

    // Semantic search with embeddings
    let embed_vec = my_embedding_provider.encode("how to write async code in rust")?;
    let semantic_hits = store
        .hybrid_search(
            WorkspaceId::default(),
            ProjectId::default(),
            "async rust".to_string(),
            Some(embed_vec),    // Activates cosine similarity stream
            "openai".to_string(),
            "text-embedding-ada-002".to_string(),
            1536,
            20,
            None,
        )
        .await?;

    // Debug view with per-stream rankings
    let explained = store
        .hybrid_search_explained(
            WorkspaceId::default(),
            ProjectId::default(),
            "rust".to_string(),
            None,
            "".to_string(),
            "".to_string(),
            0,
            10,
            None,
        )
        .await?;
    
    for (hit, explain) in explained {
        println!("Page {} – fused score {:.3}", hit.path, hit.score);
        println!("  – FTS rank: {:?}, Entity rank: {:?}, Vector rank: {:?}, Graph rank: {:?}",
                 explain.fts_rank, explain.entity_rank, explain.vector_rank, explain.graph_rank);
    }

    Ok(())
}

The HTTP endpoint in crates/ai-memory-web/src/routes/api.rs exposes this functionality via the /search route, making the hybrid retrieval available to web clients.

Summary

• ai-memory's retrieval architecture combines four parallel streams: FTS5 full-text search, entity matching, vector embeddings, and link-graph expansion.
• Reciprocal Rank Fusion in crates/ai-memory-store/src/reader.rs aggregates ranks using score = 1 / (60 + rank), eliminating the need for cross-modal score normalization.
• The system degrades gracefully when embeddings or entities are unavailable, falling back to available streams without code changes.
• Link-graph expansion boosts contextually related pages through breadth-first traversal of markdown link syntax.
• The Store::hybrid_search API provides unified access to all retrieval modes with optional explanation metadata for debugging.

Frequently Asked Questions

What is Reciprocal Rank Fusion (RRF) and why does ai-memory use it?

Reciprocal Rank Fusion is a rank-aggregation technique that combines multiple ordered result lists by assigning each item a score of 1 / (k + rank), where k is a constant (typically 60) that dampens the impact of lower-ranked items. ai-memory uses RRF in crates/ai-memory-store/src/reader.rs because it handles missing streams gracefully and does not require normalizing scores across different modalities (text match scores vs. cosine similarity vs. graph distance).

Can I use ai-memory's retrieval system without vector embeddings?

Yes. The retrieval architecture degrades independently by stream. If you omit the query_vec parameter or provide None to Store::hybrid_search, the vector embedding stream is disabled and the system relies solely on FTS5, entity matching, and link-graph expansion. This allows lightweight deployments without embedding providers.

How does the link-graph expansion improve search results?

The link-graph expansion performs a breadth-first walk starting from pages returned by other streams, following edges created from markdown [[link]] syntax. Each neighboring page receives a small rank boost through the RRF aggregation. This surfaces contextually related pages that may not contain the query terms but are linked to highly relevant pages, improving recall for exploratory search scenarios.

Where is the hybrid search logic implemented in the codebase?

The core hybrid search implementation resides in crates/ai-memory-store/src/reader.rs, specifically within the private method hybrid_search_inner. Supporting components include crates/ai-memory-store/src/fts_query.rs for query normalization, crates/ai-memory-consolidate/src/lint.rs for entity extraction, and crates/ai-memory-consolidate/src/embeddings.rs for vector management. The web API endpoint calling this logic is located in crates/ai-memory-web/src/routes/api.rs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}