# How the ai-memory-consolidate Recall-Eval Framework Scores Retrieval Relevance

> Discover how the ai-memory-consolidate framework scores retrieval relevance using Recall@5. Compare FTS5 keyword search against hybrid search with Reciprocal Rank Fusion.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: deep-dive
- Published: 2026-08-30

---

**The `ai-memory-consolidate` recall-eval framework scores retrieval relevance using a `Recall@5` metric that measures whether ground-truth pages appear in the top 5 results of pure FTS5 keyword search versus hybrid search with Reciprocal Rank Fusion.**

The `ai-memory-consolidate` crate in the `akitaonrails/ai-memory` repository provides a lightweight, CI-integrated evaluation harness for validating retrieval quality. This article explains how the recall-eval framework quantifies relevance, enforces regression guards, and ensures hybrid search improvements never degrade baseline keyword performance.

---

## What the Recall-Eval Framework Measures

The framework implements a **Recall@5** metric that serves as a continuous integration (CI) safety net. Instead of complex relevance grading, it uses a binary hit/miss evaluation: does the expected page appear in the top 5 results?

The scoring formula is straightforward:

```rust
Recall@5 = # hits / total probes

```

This simplicity makes the test fast, deterministic, and easy to interpret in CI logs. The framework compares two search modes side-by-side:

- **Pure FTS5**: keyword-only search via `store.reader.search_pages`
- **Hybrid search**: keyword + embedding similarity fused via **Reciprocal Rank Fusion (RRF)** via `store.reader.hybrid_search`

---

## Test Corpus and Probe Design

The evaluation uses a hand-crafted wiki corpus defined in [`crates/ai-memory-consolidate/tests/recall_eval.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/tests/recall_eval.rs). The `CORPUS` constant contains structured pages with varying content types, while `PROBES` defines query/ground-truth pairs that exercise different retrieval paths.

This design intentionally uses a small, controlled dataset rather than production-scale data. Small corpora make failures reproducible and prevent flaky tests caused by embedding model drift or indexing noise.

---

## How Scoring Works in `measure_recall`

The core scoring logic resides in the `measure_recall` function. Here's the implementation from [`crates/ai-memory-consolidate/tests/recall_eval.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/tests/recall_eval.rs):

```rust
async fn measure_recall(
    store: &Store,
    ws: WorkspaceId,
    proj: ProjectId,
    embedder: Option<Arc<dyn Embedder>>,
) -> f64 {
    let mut hits = 0usize;
    for (query, expected) in PROBES {
        let results = if let Some(emb) = &embedder {
            // Hybrid path: embed query, then fuse with RRF
            let qv = emb.embed_query(query).await.expect("embed query");
            store.reader.hybrid_search(
                ws, proj, query.to_string(),
                Some(qv), emb.provider().to_string(),
                emb.model().to_string(), emb.dim(), 5, None
            ).await.expect("hybrid")
        } else {
            // Pure FTS5 path: keyword-only
            store.reader.search_pages(query.to_string(), 5).await.expect("search")
        };
        if results.iter().any(|r| r.path.as_str() == *expected) {
            hits += 1;
        }
    }
    hits as f64 / PROBES.len() as f64
}

```

The function accepts an optional `embedder`. When `None`, it benchmarks pure keyword retrieval. When `Some(emb)`, it exercises the full hybrid pipeline including vector similarity, entity streams, and graph expansion.

---

## Reciprocal Rank Fusion in Hybrid Search

The hybrid search path in [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs) combines four ranked result streams using **RRF**:

| Stream | Source |
|--------|--------|
| FTS5 scores | Keyword relevance from SQLite FTS5 |
| Entity stream | Pages matching front-matter `entities` |
| Graph-neighbor expansion | Pages linked via wiki `[[links]]` |
| Vector similarity | Cosine similarity over stored embeddings |

RRF assigns each candidate a score based on its rank in each stream: `score = Σ(1 / (k + rank))` where `k` is a constant (typically 60). These scores are summed across streams to produce the final ordering. This method requires no score normalization between heterogeneous sources.

---

## Enforcing the Recall Floor

The framework enforces a **minimum Recall@5 of 0.70** via the `RECALL_FLOOR` constant at line 7 of [`recall_eval.rs`](https://github.com/akitaonrails/ai-memory/blob/main/recall_eval.rs):

```rust
// crates/ai-memory-consolidate/tests/recall_eval.rs (line 7)
const RECALL_FLOOR: f64 = 0.70;

```

The baseline test `recall_at_5_baseline` asserts both search modes meet this threshold:

```rust
#[tokio::test]
async fn recall_at_5_baseline() {
    // ... setup temporary store, write corpus ...
    let fts_recall = measure_recall(&store, ws, proj, None).await;
    let hybrid_recall = measure_recall(&store, ws, proj, Some(embedder.clone())).await;
    
    eprintln!("recall_eval: FTS5={fts_recall:.3}, hybrid={hybrid_recall:.3}");
    
    assert!(fts_recall >= RECALL_FLOOR);
    assert!(hybrid_recall >= RECALL_FLOOR);
}

```

The `eprintln!` output surfaces metrics in CI logs for trend monitoring. A passing test guarantees that hybrid search **improves or maintains** pure FTS5 performance—never degrades it.

---

## Auxiliary Regression Tests

Beyond the main recall metric, three focused tests verify that individual hybrid components can rescue failures:

- **`graph_neighbor_expansion_recovers_linked_page`**: Validates that wiki-link traversal retrieves related pages missed by keyword search
- **`entity_stream_recovers_a_probe_fts_and_graph_both_miss`**: Confirms front-matter entity matching finds semantically related content
- **`raw_observation_fallback_recovers_detail_when_wiki_misses`**: Ensures raw observation fallback handles edge cases

These tests provide granular failure signals when specific retrieval paths break.

---

## Why Recall@5 Was Chosen

The `ai-memory-consolidate` authors selected Recall@5 over precision-oriented metrics for three reasons:

1. **User experience alignment**: In memory systems, users expect relevant results in the first few positions—exact rank beyond top-5 matters less
2. **Regression sensitivity**: Binary hit/miss detection catches catastrophic failures immediately
3. **CI speed**: Simple counting avoids expensive judgment aggregation or model-based evaluation

The metric deliberately ignores result ordering within the top 5. This trades granularity for stability—small embedding model updates that shuffle ranks without changing hit status won't trigger false failures.

---

## Summary

- **Recall-eval framework** uses `Recall@5 = hits / total probes` as its core metric
- Compares **pure FTS5** (`search_pages`) against **hybrid RRF** (`hybrid_search`) with optional embeddings
- Enforces **0.70 minimum recall** via `RECALL_FLOOR` constant in [`recall_eval.rs`](https://github.com/akitaonrails/ai-memory/blob/main/recall_eval.rs)
- Employs **Reciprocal Rank Fusion** to combine FTS5, entity, graph, and vector similarity streams
- Includes **three auxiliary tests** for component-specific regression detection
- Designed as **CI guard** ensuring hybrid enhancements never degrade keyword baseline

---

## Frequently Asked Questions

### What is Recall@5 and why does ai-memory-consolidate use it?

**Recall@5 measures the proportion of queries where the correct answer appears in the top 5 results.** The `ai-memory-consolidate` framework uses this metric because it directly tests whether users can find expected content quickly, without requiring complex relevance scoring or human judgment. It also produces stable, fast tests suitable for continuous integration.

### How does hybrid search differ from pure FTS5 in the evaluation?

**Pure FTS5 runs keyword-only search via `store.reader.search_pages`, while hybrid search fuses multiple signals via `store.reader.hybrid_search`.** The hybrid path embeds the query, then combines FTS5 scores, entity matches, graph neighbors, and vector similarity using Reciprocal Rank Fusion. The evaluation requires both paths to meet the 0.70 recall floor.

### Where is the recall floor defined and what happens if it's breached?

**The `RECALL_FLOOR` constant is defined at line 7 of [`crates/ai-memory-consolidate/tests/recall_eval.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/tests/recall_eval.rs) and set to 0.70.** If either pure FTS5 or hybrid search scores below this threshold, the `recall_at_5_baseline` test fails and blocks the CI pipeline, preventing regression merges.

### What auxiliary tests support the main recall evaluation?

**Three additional tests verify individual retrieval components:** `graph_neighbor_expansion_recovers_linked_page` tests wiki-link traversal; `entity_stream_recovers_a_probe_fts_and_graph_both_miss` tests front-matter entity matching; and `raw_observation_fallback_recovers_detail_when_wiki_misses` tests raw content fallback. These isolate failures to specific hybrid subsystems.