# How ai-memory’s FTS5 Full-Text Search Ranking Works Internally

> Discover how ai-memory’s FTS5 full-text search ranking works internally. Learn about query normalization, score adjustment, and Reciprocal Rank Fusion for precise search results.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: internals
- Published: 2026-08-19

---

**ai-memory normalizes free-text queries into safe SQLite FTS5 syntax, retrieves pages using the built-in FTS5 `rank` column, adjusts scores with a page-authority multiplier, and fuses multiple retrieval streams via Reciprocal Rank Fusion to produce the final search results.**

The `ai-memory` project implements a multi-stage ranking pipeline inside the `ai-memory-store` crate. When a user submits a full-text query, the system does not rely solely on SQLite’s FTS5 relevance score. Instead, it sanitizes the input, fetches candidates from external-content virtual tables, and re-ranks them using authority signals and optional vector or graph evidence.

## Query Normalization in `prepare_fts5_query`

Before any ranking occurs, raw user input passes through `prepare_fts5_query` in [`crates/ai-memory-store/src/fts_query.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/fts_query.rs). This function is pure—it performs no I/O—and returns an empty string when the input is empty or malformed.

### Stripping Column Prefixes

The function uses a lazy static regex, `COLUMN_PREFIX_RE`, to detect and remove prefixes like `title:`, `body:`, or `path_search:`.

```rust
Regex::new(r"(?i)\b(title|body|path_search):(?P<term>[^\s]+)")

```

Because the underlying tables are external-content tables that already contain the indexed columns, these prefixes are rewritten to the bare term. This also prevents injection of arbitrary column names.

### Whitespace Collapse and Validation

Consecutive whitespace is collapsed to a single space. Then a second regex validates that no stray colons remain:

```rust
Regex::new(r"^[^:]*$(?s)")

```

If validation fails, the function returns an empty string, causing the caller to skip the FTS5 filter entirely.

## FTS5 Candidate Retrieval and the Built-in `rank`

The `search_pages` function in [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs) executes a prepared SQL statement against the `pages_fts` virtual table. The query selects `pages_fts.rank` directly from SQLite:

```rust
let mut stmt = conn.prepare_cached(
    "SELECT pages_fts.rowid, pages.rowid, pages.title, pages.path, \
     snippet(pages_fts, 1, '<mark>', '</mark>', '…', 24) AS snip, \
     pages_fts.rank, pages.tier, pages.pinned \
     FROM pages_fts \
     JOIN pages ON pages.rowid = pages_fts.rowid \
     WHERE pages_fts MATCH ?1 AND pages.is_latest = 1 \
     ORDER BY pages_fts.rank LIMIT ?2",
)?;

```

SQLite computes `pages_fts.rank` using its default FTS5 ranking function (BM25). In this pipeline, a lower numerical value means higher relevance.

## Adjusting Relevance with `PageAuthority`

After retrieval, each `PageHit` is paired with a `PageAuthority` struct that encodes signals such as pin status and tier. The `adjust_rank` method applies a multiplicative factor:

```rust
impl PageAuthority {
    pub fn adjust_rank(&self, rank: f64) -> f64 {
        if rank <= 0.0 {
            rank * self.factor
        } else {
            rank / self.factor
        }
    }
}

```

Highly trusted pages receive a factor greater than 1.0. Because the method preserves the sign, it respects the FTS5 invariant that lower rank values indicate higher relevance regardless of whether the raw score is negative or positive.

## Reciprocal Rank Fusion Across Streams

When multiple evidence sources are available—FTS5, vector similarity, graph proximity, or entity matches—ai-memory merges them using **Reciprocal Rank Fusion (RRF)**. The helper `rrf_fuse` in [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs) implements the core formula:

```rust
fn rrf_fuse(k: f64, ranks: impl Iterator<Item = usize>) -> f64 {
    ranks.map(|r| 1.0 / (k + r as f64)).sum()
}

```

The constant `k` is set to `60.0`. Each stream produces a 1-based ranking for a given page, and that page receives a contribution of `1 / (k + rank)`. These contributions are summed across all streams in `fuse_streams`:

```rust
let k = 60.0;
// ...
for (rank, hit) in fts_hits.iter().enumerate() {
    let contrib = 1.0 / (k + (rank + 1) as f64);
    *contributions.entry(hit.id).or_default() += contrib;
}

```

Higher-ranked pages in any stream accumulate larger fused scores, which are then converted back to a sortable rank.

## End-to-End Flow in `search_pages`

The complete pipeline inside `search_pages` follows these steps:

1. **Normalize the query** via `normalize_fts_query`, which delegates to `prepare_fts5_query` in [`fts_query.rs`](https://github.com/akitaonrails/ai-memory/blob/main/fts_query.rs).
2. **Guard against empty input**—if the normalized query is empty, return an empty result set immediately.
3. **Execute the FTS5 `MATCH` query** to fetch an initial candidate list ordered by `pages_fts.rank`.
4. **Map each row** into a `PageHit` and compute its `PageAuthority`.
5. **Adjust each hit’s rank** using `PageAuthority::adjust_rank`.
6. **Sort and truncate** the hits by the adjusted rank and crop to the requested limit.

If vector or graph streams are present, the system inserts an RRF fusion step between candidate collection and final truncation.

A minimal usage example looks like this:

```rust
use ai_memory_store::store::Connection;
use ai_memory_store::reader::search_pages;

let conn: Connection = /* obtain connection */;
let hits = search_pages(&conn, "memory handoff", 10)?;

if let Some(top) = hits.first() {
    println!("Title: {}", top.title);
    println!("Snippet: {}", top.snippet);
    println!("Adjusted rank: {:.4}", top.rank);
}

```

## Summary

- `prepare_fts5_query` in [`fts_query.rs`](https://github.com/akitaonrails/ai-memory/blob/main/fts_query.rs) strips column prefixes and validates syntax before any database access occurs.
- `search_pages` in [`reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/reader.rs) retrieves candidates from the `pages_fts` external-content table using SQLite’s native `rank`.
- `PageAuthority::adjust_rank` scales the raw score by a trust factor, keeping lower values more relevant.
- `rrf_fuse` and `fuse_streams` blend FTS5 results with vector, graph, and entity signals using Reciprocal Rank Fusion with `k = 60.0`.
- The final output is a deduplicated, re-ranked `Vec<PageHit>` cropped to the caller’s limit.

## Frequently Asked Questions

### What SQLite FTS5 rank function does ai-memory use?

ai-memory relies on the default FTS5 rank provided by SQLite, which is based on BM25. The `search_pages` function selects `pages_fts.rank` directly and orders by it, interpreting lower values as higher relevance.

### How does ai-memory prevent SQL injection in full-text queries?

The `prepare_fts5_query` function neutralizes column-prefix syntax and rejects any query containing stray colons or unbalanced quotes. If validation fails, it returns an empty string, causing the caller to omit the `MATCH` clause entirely rather than risk a malformed expression.

### What is Reciprocal Rank Fusion and why does ai-memory use it?

Reciprocal Rank Fusion is a rank-based aggregation method that converts each stream’s ordinal position into a score of `1 / (k + rank)` and sums them. ai-memory uses it because it is parameter-light, does not require normalized score ranges, and fairly balances evidence from full-text, vector, graph, and entity retrieval streams.

### How does page authority affect the final search ranking?

Page authority is encoded as a multiplicative factor. `PageAuthority::adjust_rank` multiplies negative raw ranks by the factor and divides positive raw ranks by it. This nudges highly trusted or pinned pages toward the top without breaking the ordering invariant that lower numeric rank equals higher relevance.