How the ai-memory-consolidate Recall-Eval Framework Scores Retrieval Relevance

The ai-memory-consolidate recall-eval framework scores retrieval relevance using a Recall@5 metric that measures whether ground-truth pages appear in the top 5 results of pure FTS5 keyword search versus hybrid search with Reciprocal Rank Fusion.

The ai-memory-consolidate crate in the akitaonrails/ai-memory repository provides a lightweight, CI-integrated evaluation harness for validating retrieval quality. This article explains how the recall-eval framework quantifies relevance, enforces regression guards, and ensures hybrid search improvements never degrade baseline keyword performance.

---

What the Recall-Eval Framework Measures

The framework implements a Recall@5 metric that serves as a continuous integration (CI) safety net. Instead of complex relevance grading, it uses a binary hit/miss evaluation: does the expected page appear in the top 5 results?

The scoring formula is straightforward:

Recall@5 = # hits / total probes

This simplicity makes the test fast, deterministic, and easy to interpret in CI logs. The framework compares two search modes side-by-side:

• Pure FTS5: keyword-only search via store.reader.search_pages
• Hybrid search: keyword + embedding similarity fused via Reciprocal Rank Fusion (RRF) via store.reader.hybrid_search

---

Test Corpus and Probe Design

The evaluation uses a hand-crafted wiki corpus defined in crates/ai-memory-consolidate/tests/recall_eval.rs. The CORPUS constant contains structured pages with varying content types, while PROBES defines query/ground-truth pairs that exercise different retrieval paths.

This design intentionally uses a small, controlled dataset rather than production-scale data. Small corpora make failures reproducible and prevent flaky tests caused by embedding model drift or indexing noise.

---

How Scoring Works in measure_recall

The core scoring logic resides in the measure_recall function. Here's the implementation from crates/ai-memory-consolidate/tests/recall_eval.rs:

async fn measure_recall(
    store: &Store,
    ws: WorkspaceId,
    proj: ProjectId,
    embedder: Option<Arc<dyn Embedder>>,
) -> f64 {
    let mut hits = 0usize;
    for (query, expected) in PROBES {
        let results = if let Some(emb) = &embedder {
            // Hybrid path: embed query, then fuse with RRF
            let qv = emb.embed_query(query).await.expect("embed query");
            store.reader.hybrid_search(
                ws, proj, query.to_string(),
                Some(qv), emb.provider().to_string(),
                emb.model().to_string(), emb.dim(), 5, None
            ).await.expect("hybrid")
        } else {
            // Pure FTS5 path: keyword-only
            store.reader.search_pages(query.to_string(), 5).await.expect("search")
        };
        if results.iter().any(|r| r.path.as_str() == *expected) {
            hits += 1;
        }
    }
    hits as f64 / PROBES.len() as f64
}

The function accepts an optional embedder. When None, it benchmarks pure keyword retrieval. When Some(emb), it exercises the full hybrid pipeline including vector similarity, entity streams, and graph expansion.

---

Reciprocal Rank Fusion in Hybrid Search

The hybrid search path in crates/ai-memory-store/src/reader.rs combines four ranked result streams using RRF:

Stream	Source
FTS5 scores	Keyword relevance from SQLite FTS5
Entity stream	Pages matching front-matter entities
Graph-neighbor expansion	Pages linked via wiki [[links]]
Vector similarity	Cosine similarity over stored embeddings

RRF assigns each candidate a score based on its rank in each stream: score = Σ(1 / (k + rank)) where k is a constant (typically 60). These scores are summed across streams to produce the final ordering. This method requires no score normalization between heterogeneous sources.

---

Enforcing the Recall Floor

The framework enforces a minimum Recall@5 of 0.70 via the RECALL_FLOOR constant at line 7 of recall_eval.rs:

// crates/ai-memory-consolidate/tests/recall_eval.rs (line 7)
const RECALL_FLOOR: f64 = 0.70;

The baseline test recall_at_5_baseline asserts both search modes meet this threshold:

#[tokio::test]
async fn recall_at_5_baseline() {
    // ... setup temporary store, write corpus ...
    let fts_recall = measure_recall(&store, ws, proj, None).await;
    let hybrid_recall = measure_recall(&store, ws, proj, Some(embedder.clone())).await;
    
    eprintln!("recall_eval: FTS5={fts_recall:.3}, hybrid={hybrid_recall:.3}");
    
    assert!(fts_recall >= RECALL_FLOOR);
    assert!(hybrid_recall >= RECALL_FLOOR);
}

The eprintln! output surfaces metrics in CI logs for trend monitoring. A passing test guarantees that hybrid search improves or maintains pure FTS5 performance—never degrades it.

---

Auxiliary Regression Tests

Beyond the main recall metric, three focused tests verify that individual hybrid components can rescue failures:

• graph_neighbor_expansion_recovers_linked_page: Validates that wiki-link traversal retrieves related pages missed by keyword search
• entity_stream_recovers_a_probe_fts_and_graph_both_miss: Confirms front-matter entity matching finds semantically related content
• raw_observation_fallback_recovers_detail_when_wiki_misses: Ensures raw observation fallback handles edge cases

These tests provide granular failure signals when specific retrieval paths break.

---

Why Recall@5 Was Chosen

The ai-memory-consolidate authors selected Recall@5 over precision-oriented metrics for three reasons:

1. User experience alignment: In memory systems, users expect relevant results in the first few positions—exact rank beyond top-5 matters less
2. Regression sensitivity: Binary hit/miss detection catches catastrophic failures immediately
3. CI speed: Simple counting avoids expensive judgment aggregation or model-based evaluation

The metric deliberately ignores result ordering within the top 5. This trades granularity for stability—small embedding model updates that shuffle ranks without changing hit status won't trigger false failures.

---

Summary

• Recall-eval framework uses Recall@5 = hits / total probes as its core metric
• Compares pure FTS5 (search_pages) against hybrid RRF (hybrid_search) with optional embeddings
• Enforces 0.70 minimum recall via RECALL_FLOOR constant in recall_eval.rs
• Employs Reciprocal Rank Fusion to combine FTS5, entity, graph, and vector similarity streams
• Includes three auxiliary tests for component-specific regression detection
• Designed as CI guard ensuring hybrid enhancements never degrade keyword baseline

---

Frequently Asked Questions

What is Recall@5 and why does ai-memory-consolidate use it?

Recall@5 measures the proportion of queries where the correct answer appears in the top 5 results. The ai-memory-consolidate framework uses this metric because it directly tests whether users can find expected content quickly, without requiring complex relevance scoring or human judgment. It also produces stable, fast tests suitable for continuous integration.

How does hybrid search differ from pure FTS5 in the evaluation?

Pure FTS5 runs keyword-only search via store.reader.search_pages, while hybrid search fuses multiple signals via store.reader.hybrid_search. The hybrid path embeds the query, then combines FTS5 scores, entity matches, graph neighbors, and vector similarity using Reciprocal Rank Fusion. The evaluation requires both paths to meet the 0.70 recall floor.

Where is the recall floor defined and what happens if it's breached?

The RECALL_FLOOR constant is defined at line 7 of crates/ai-memory-consolidate/tests/recall_eval.rs and set to 0.70. If either pure FTS5 or hybrid search scores below this threshold, the recall_at_5_baseline test fails and blocks the CI pipeline, preventing regression merges.

What auxiliary tests support the main recall evaluation?

Three additional tests verify individual retrieval components: graph_neighbor_expansion_recovers_linked_page tests wiki-link traversal; entity_stream_recovers_a_probe_fts_and_graph_both_miss tests front-matter entity matching; and raw_observation_fallback_recovers_detail_when_wiki_misses tests raw content fallback. These isolate failures to specific hybrid subsystems.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}