How ai-memory’s FTS5 Full-Text Search Ranking Works Internally

ai-memory normalizes free-text queries into safe SQLite FTS5 syntax, retrieves pages using the built-in FTS5 rank column, adjusts scores with a page-authority multiplier, and fuses multiple retrieval streams via Reciprocal Rank Fusion to produce the final search results.

The ai-memory project implements a multi-stage ranking pipeline inside the ai-memory-store crate. When a user submits a full-text query, the system does not rely solely on SQLite’s FTS5 relevance score. Instead, it sanitizes the input, fetches candidates from external-content virtual tables, and re-ranks them using authority signals and optional vector or graph evidence.

Query Normalization in prepare_fts5_query

Before any ranking occurs, raw user input passes through prepare_fts5_query in crates/ai-memory-store/src/fts_query.rs. This function is pure—it performs no I/O—and returns an empty string when the input is empty or malformed.

Stripping Column Prefixes

The function uses a lazy static regex, COLUMN_PREFIX_RE, to detect and remove prefixes like title:, body:, or path_search:.

Regex::new(r"(?i)\b(title|body|path_search):(?P<term>[^\s]+)")

Because the underlying tables are external-content tables that already contain the indexed columns, these prefixes are rewritten to the bare term. This also prevents injection of arbitrary column names.

Whitespace Collapse and Validation

Consecutive whitespace is collapsed to a single space. Then a second regex validates that no stray colons remain:

Regex::new(r"^[^:]*$(?s)")

If validation fails, the function returns an empty string, causing the caller to skip the FTS5 filter entirely.

FTS5 Candidate Retrieval and the Built-in rank

The search_pages function in crates/ai-memory-store/src/reader.rs executes a prepared SQL statement against the pages_fts virtual table. The query selects pages_fts.rank directly from SQLite:

let mut stmt = conn.prepare_cached(
    "SELECT pages_fts.rowid, pages.rowid, pages.title, pages.path, \
     snippet(pages_fts, 1, '<mark>', '</mark>', '…', 24) AS snip, \
     pages_fts.rank, pages.tier, pages.pinned \
     FROM pages_fts \
     JOIN pages ON pages.rowid = pages_fts.rowid \
     WHERE pages_fts MATCH ?1 AND pages.is_latest = 1 \
     ORDER BY pages_fts.rank LIMIT ?2",
)?;

SQLite computes pages_fts.rank using its default FTS5 ranking function (BM25). In this pipeline, a lower numerical value means higher relevance.

Adjusting Relevance with PageAuthority

After retrieval, each PageHit is paired with a PageAuthority struct that encodes signals such as pin status and tier. The adjust_rank method applies a multiplicative factor:

impl PageAuthority {
    pub fn adjust_rank(&self, rank: f64) -> f64 {
        if rank <= 0.0 {
            rank * self.factor
        } else {
            rank / self.factor
        }
    }
}

Highly trusted pages receive a factor greater than 1.0. Because the method preserves the sign, it respects the FTS5 invariant that lower rank values indicate higher relevance regardless of whether the raw score is negative or positive.

Reciprocal Rank Fusion Across Streams

When multiple evidence sources are available—FTS5, vector similarity, graph proximity, or entity matches—ai-memory merges them using Reciprocal Rank Fusion (RRF). The helper rrf_fuse in crates/ai-memory-store/src/reader.rs implements the core formula:

fn rrf_fuse(k: f64, ranks: impl Iterator<Item = usize>) -> f64 {
    ranks.map(|r| 1.0 / (k + r as f64)).sum()
}

The constant k is set to 60.0. Each stream produces a 1-based ranking for a given page, and that page receives a contribution of 1 / (k + rank). These contributions are summed across all streams in fuse_streams:

let k = 60.0;
// ...
for (rank, hit) in fts_hits.iter().enumerate() {
    let contrib = 1.0 / (k + (rank + 1) as f64);
    *contributions.entry(hit.id).or_default() += contrib;
}

Higher-ranked pages in any stream accumulate larger fused scores, which are then converted back to a sortable rank.

End-to-End Flow in search_pages

The complete pipeline inside search_pages follows these steps:

1. Normalize the query via normalize_fts_query, which delegates to prepare_fts5_query in fts_query.rs.
2. Guard against empty input—if the normalized query is empty, return an empty result set immediately.
3. Execute the FTS5 MATCH query to fetch an initial candidate list ordered by pages_fts.rank.
4. Map each row into a PageHit and compute its PageAuthority.
5. Adjust each hit’s rank using PageAuthority::adjust_rank.
6. Sort and truncate the hits by the adjusted rank and crop to the requested limit.

If vector or graph streams are present, the system inserts an RRF fusion step between candidate collection and final truncation.

A minimal usage example looks like this:

use ai_memory_store::store::Connection;
use ai_memory_store::reader::search_pages;

let conn: Connection = /* obtain connection */;
let hits = search_pages(&conn, "memory handoff", 10)?;

if let Some(top) = hits.first() {
    println!("Title: {}", top.title);
    println!("Snippet: {}", top.snippet);
    println!("Adjusted rank: {:.4}", top.rank);
}

Summary

• prepare_fts5_query in fts_query.rs strips column prefixes and validates syntax before any database access occurs.
• search_pages in reader.rs retrieves candidates from the pages_fts external-content table using SQLite’s native rank.
• PageAuthority::adjust_rank scales the raw score by a trust factor, keeping lower values more relevant.
• rrf_fuse and fuse_streams blend FTS5 results with vector, graph, and entity signals using Reciprocal Rank Fusion with k = 60.0.
• The final output is a deduplicated, re-ranked Vec<PageHit> cropped to the caller’s limit.

Frequently Asked Questions

What SQLite FTS5 rank function does ai-memory use?

ai-memory relies on the default FTS5 rank provided by SQLite, which is based on BM25. The search_pages function selects pages_fts.rank directly and orders by it, interpreting lower values as higher relevance.

How does ai-memory prevent SQL injection in full-text queries?

The prepare_fts5_query function neutralizes column-prefix syntax and rejects any query containing stray colons or unbalanced quotes. If validation fails, it returns an empty string, causing the caller to omit the MATCH clause entirely rather than risk a malformed expression.

What is Reciprocal Rank Fusion and why does ai-memory use it?

Reciprocal Rank Fusion is a rank-based aggregation method that converts each stream’s ordinal position into a score of 1 / (k + rank) and sums them. ai-memory uses it because it is parameter-light, does not require normalized score ranges, and fairly balances evidence from full-text, vector, graph, and entity retrieval streams.

How does page authority affect the final search ranking?

Page authority is encoded as a multiplicative factor. PageAuthority::adjust_rank multiplies negative raw ranks by the factor and divides positive raw ranks by it. This nudges highly trusted or pinned pages toward the top without breaking the ordering invariant that lower numeric rank equals higher relevance.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}