# How to Run the Recall-Eval Framework to Benchmark Retrieval Quality in AI-Memory

> Benchmark retrieval quality with the recall-eval framework. Install Rust, clone the AI-Memory repo, and run the command to test FTS5 and hybrid pipelines.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: how-to-guide
- Published: 2026-08-27

---

**Run the recall-eval benchmark by installing Rust 1.95, cloning the akitaonrails/ai-memory repository, and executing `cargo test -p ai-memory-consolidate --test recall_eval -- --nocapture` to measure recall@5 for both pure-FTS5 and hybrid retrieval pipelines.**

The recall-eval harness is the reference implementation of the "Real LongMemEval-S" benchmark found in the architecture documentation of the **akitaonrails/ai-memory** repository. This lightweight integration test constructs a synthetic wiki, executes targeted queries, and asserts that retrieval quality meets the minimum recall threshold defined in the source code. Running this framework provides immediate verification that the SQLite-based memory stack performs within acceptable parameters, especially when comparing keyword-only search against the hybrid vector-enhanced path.

## Prerequisites: Install Rust 1.95

The project pins a specific Rust toolchain to ensure deterministic builds.

Execute the following commands to install and activate Rust 1.95 as specified in [`rust-toolchain.toml`](https://github.com/akitaonrails/ai-memory/blob/main/rust-toolchain.toml):

```bash
rustup toolchain install 1.95
rustup default 1.95

```

Verify the installation with `rustc --version` before proceeding.

## Running the Recall-Eval Benchmark

The test harness lives in [`crates/ai-memory-consolidate/tests/recall_eval.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/tests/recall_eval.rs) and executes against a temporary SQLite database containing a synthetic corpus.

### Basic Execution

Clone the repository and run the evaluation with this command:

```bash
git clone https://github.com/akitaonrails/ai-memory.git
cd ai-memory
cargo test -p ai-memory-consolidate --test recall_eval -- --nocapture

```

The flags achieve the following:
- `-p ai-memory-consolidate` selects the specific crate containing the test
- `--test recall_eval` targets the exact test file instead of running the full suite
- `--nocapture` prints the `eprintln!` summary showing recall scores to stdout

The output displays a line similar to:

```

recall_eval: FTS5=0.78, hybrid=0.81

```

If either value falls below `0.70`, the test panics and fails the assertion check defined at line 107 of [`recates/ai-memory-consolidate/tests/recall_eval.rs`](https://github.com/akitaonrails/ai-memory/blob/main/recates/ai-memory-consolidate/tests/recall_eval.rs).

### Configuring Real Embedding Providers

By default, the harness uses a deterministic synthetic embedder that requires no external API keys. To benchmark the hybrid path with production-quality vectors, export the embedding provider environment variables recognized by [`crates/ai-memory-llm/src/factory.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-llm/src/factory.rs):

```bash
export AI_MEMORY_EMBEDDING_PROVIDER=openai
export OPENAI_API_KEY=sk-yourkeyhere
export AI_MEMORY_EMBEDDING_MODEL=text-embedding-3-small

```

With these variables set, the hybrid pipeline enables the **vector RRF** (Reciprocal Rank Fusion) component alongside existing entity and graph neighbors. This configuration reveals the true performance delta between pure keyword search and the full retrieval stack.

## Understanding the Test Output

The `RECALL_FLOOR` constant in [`recall_eval.rs`](https://github.com/akitaonrails/ai-memory/blob/main/recall_eval.rs) (line 107) establishes a hard minimum of **0.70** for both retrieval paths. The test calculates **recall@5** by comparing the top-5 results against a known ground-truth set for each probe query.

Interpret the printed scores as follows:
- **FTS5**: Pure full-text search using SQLite's FTS5 extension
- **Hybrid**: Combined ranking of FTS5, entity relationships, graph neighbors, and vector similarity (when configured)

A higher hybrid score indicates that the vector and knowledge-graph components successfully surface relevant documents that keyword search misses. If the harness reports scores approaching the 0.70 floor, investigate recent changes to the ranking logic in [`crates/ai-memory-core/src/ranking.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-core/src/ranking.rs) or the embedding generation pipeline.

## How the Harness Works

The evaluation follows a strict six-step procedure implemented in [`recall_eval.rs`](https://github.com/akitaonrails/ai-memory/blob/main/recall_eval.rs):

1. **Create a temporary SQLite store** using `TempDir` and `Store::open` to ensure test isolation
2. **Populate the synthetic wiki** with ten hand-crafted pages defined in the `CORPUS` constant
3. **Insert a synthetic session** containing raw observations to test the "raw-observation fallback" probe
4. **Execute dual retrieval pipelines** for each probe query:
   - Pure-FTS5 path using keyword search only
   - Hybrid path combining keyword results with entity RRF, graph-neighbor RRF, and optional vector RRF
5. **Calculate recall@5** by checking for ground-truth hits in the top-5 returned results
6. **Assert against the floor** value and print diagnostic scores via `eprintln!`

This architecture requires no external data files, making the harness completely self-contained and suitable for rapid iteration.

## Integrating with CI/CD

The recall-eval framework executes in less than one second, making it ideal for continuous integration pipelines. Add the cargo test command to your GitHub Actions workflow or similar CI system to catch retrieval regressions automatically:

```yaml
- name: Run recall-eval benchmark
  run: cargo test -p ai-memory-consolidate --test recall_eval -- --nocapture
  env:
    AI_MEMORY_EMBEDDING_PROVIDER: ${{ secrets.EMBEDDING_PROVIDER }}
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

```

The test will fail the build immediately if the refactoring of query planners or index structures degrades recall below the acceptable threshold.

## Summary

- The recall-eval framework benchmarks **recall@5** for both FTS5-only and hybrid retrieval in the **akitaonrails/ai-memory** repository
- Execute the test with `cargo test -p ai-memory-consolidate --test recall_eval -- --nocapture` after installing Rust 1.95
- The `RECALL_FLOOR` constant at line 107 of [`recall_eval.rs`](https://github.com/akitaonrails/ai-memory/blob/main/recall_eval.rs) enforces a minimum recall of **0.70**; violations trigger test failures
- Configure `AI_MEMORY_EMBEDDING_*` environment variables to enable real vector embeddings instead of synthetic data
- The harness builds a self-contained synthetic wiki requiring no external files, suitable for CI regression testing

## Frequently Asked Questions

### What does the recall-eval framework actually measure?

The framework measures **recall@5**—the proportion of ground-truth relevant documents that appear in the top-5 search results for a predefined set of probe queries. It compares two retrieval strategies: pure FTS5 keyword search and a hybrid approach combining FTS5 with entity extraction, graph traversal, and vector similarity (when configured).

### Why does the test fail with a recall floor error?

The test fails when either the pure-FTS5 or hybrid path produces a recall score below the `RECALL_FLOOR` of **0.70** defined in [`crates/ai-memory-consolidate/tests/recall_eval.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/tests/recall_eval.rs). This indicates a regression in the retrieval stack, such as broken ranking logic, missing index entries, or malformed query parsing that fails to surface relevant documents.

### Can I run the benchmark without an OpenAI API key?

Yes. The harness defaults to a **deterministic synthetic embedder** that generates consistent vector representations without network calls. Only set `AI_MEMORY_EMBEDDING_PROVIDER` and related variables if you specifically want to benchmark the hybrid path using production embedding models like OpenAI's `text-embedding-3-small`.

### Where is the synthetic test data defined?

The test corpus is hard-coded in the `CORPUS` constant within [`crates/ai-memory-consolidate/tests/recall_eval.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-consolidate/tests/recall_eval.rs). This array contains ten hand-crafted wiki pages covering fictional topics, explicitly designed to test edge cases in entity extraction and vector similarity without requiring external data dependencies.