How to Run the Recall-Eval Framework to Benchmark Retrieval Quality in AI-Memory

Run the recall-eval benchmark by installing Rust 1.95, cloning the akitaonrails/ai-memory repository, and executing cargo test -p ai-memory-consolidate --test recall_eval -- --nocapture to measure recall@5 for both pure-FTS5 and hybrid retrieval pipelines.

The recall-eval harness is the reference implementation of the "Real LongMemEval-S" benchmark found in the architecture documentation of the akitaonrails/ai-memory repository. This lightweight integration test constructs a synthetic wiki, executes targeted queries, and asserts that retrieval quality meets the minimum recall threshold defined in the source code. Running this framework provides immediate verification that the SQLite-based memory stack performs within acceptable parameters, especially when comparing keyword-only search against the hybrid vector-enhanced path.

Prerequisites: Install Rust 1.95

The project pins a specific Rust toolchain to ensure deterministic builds.

Execute the following commands to install and activate Rust 1.95 as specified in rust-toolchain.toml:

rustup toolchain install 1.95
rustup default 1.95

Verify the installation with rustc --version before proceeding.

Running the Recall-Eval Benchmark

The test harness lives in crates/ai-memory-consolidate/tests/recall_eval.rs and executes against a temporary SQLite database containing a synthetic corpus.

Basic Execution

Clone the repository and run the evaluation with this command:

git clone https://github.com/akitaonrails/ai-memory.git
cd ai-memory
cargo test -p ai-memory-consolidate --test recall_eval -- --nocapture

The flags achieve the following:

• -p ai-memory-consolidate selects the specific crate containing the test
• --test recall_eval targets the exact test file instead of running the full suite
• --nocapture prints the eprintln! summary showing recall scores to stdout

The output displays a line similar to:

recall_eval: FTS5=0.78, hybrid=0.81

If either value falls below 0.70, the test panics and fails the assertion check defined at line 107 of recates/ai-memory-consolidate/tests/recall_eval.rs.

Configuring Real Embedding Providers

By default, the harness uses a deterministic synthetic embedder that requires no external API keys. To benchmark the hybrid path with production-quality vectors, export the embedding provider environment variables recognized by crates/ai-memory-llm/src/factory.rs:

export AI_MEMORY_EMBEDDING_PROVIDER=openai
export OPENAI_API_KEY=sk-yourkeyhere
export AI_MEMORY_EMBEDDING_MODEL=text-embedding-3-small

With these variables set, the hybrid pipeline enables the vector RRF (Reciprocal Rank Fusion) component alongside existing entity and graph neighbors. This configuration reveals the true performance delta between pure keyword search and the full retrieval stack.

Understanding the Test Output

The RECALL_FLOOR constant in recall_eval.rs (line 107) establishes a hard minimum of 0.70 for both retrieval paths. The test calculates recall@5 by comparing the top-5 results against a known ground-truth set for each probe query.

Interpret the printed scores as follows:

• FTS5: Pure full-text search using SQLite's FTS5 extension
• Hybrid: Combined ranking of FTS5, entity relationships, graph neighbors, and vector similarity (when configured)

A higher hybrid score indicates that the vector and knowledge-graph components successfully surface relevant documents that keyword search misses. If the harness reports scores approaching the 0.70 floor, investigate recent changes to the ranking logic in crates/ai-memory-core/src/ranking.rs or the embedding generation pipeline.

How the Harness Works

The evaluation follows a strict six-step procedure implemented in recall_eval.rs:

1. Create a temporary SQLite store using TempDir and Store::open to ensure test isolation
2. Populate the synthetic wiki with ten hand-crafted pages defined in the CORPUS constant
3. Insert a synthetic session containing raw observations to test the "raw-observation fallback" probe
4. Execute dual retrieval pipelines for each probe query:
   • Pure-FTS5 path using keyword search only
   • Hybrid path combining keyword results with entity RRF, graph-neighbor RRF, and optional vector RRF
5. Calculate recall@5 by checking for ground-truth hits in the top-5 returned results
6. Assert against the floor value and print diagnostic scores via eprintln!

This architecture requires no external data files, making the harness completely self-contained and suitable for rapid iteration.

Integrating with CI/CD

The recall-eval framework executes in less than one second, making it ideal for continuous integration pipelines. Add the cargo test command to your GitHub Actions workflow or similar CI system to catch retrieval regressions automatically:

- name: Run recall-eval benchmark
  run: cargo test -p ai-memory-consolidate --test recall_eval -- --nocapture
  env:
    AI_MEMORY_EMBEDDING_PROVIDER: ${{ secrets.EMBEDDING_PROVIDER }}
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

The test will fail the build immediately if the refactoring of query planners or index structures degrades recall below the acceptable threshold.

Summary

• The recall-eval framework benchmarks recall@5 for both FTS5-only and hybrid retrieval in the akitaonrails/ai-memory repository
• Execute the test with cargo test -p ai-memory-consolidate --test recall_eval -- --nocapture after installing Rust 1.95
• The RECALL_FLOOR constant at line 107 of recall_eval.rs enforces a minimum recall of 0.70; violations trigger test failures
• Configure AI_MEMORY_EMBEDDING_* environment variables to enable real vector embeddings instead of synthetic data
• The harness builds a self-contained synthetic wiki requiring no external files, suitable for CI regression testing

Frequently Asked Questions

What does the recall-eval framework actually measure?

The framework measures recall@5—the proportion of ground-truth relevant documents that appear in the top-5 search results for a predefined set of probe queries. It compares two retrieval strategies: pure FTS5 keyword search and a hybrid approach combining FTS5 with entity extraction, graph traversal, and vector similarity (when configured).

Why does the test fail with a recall floor error?

The test fails when either the pure-FTS5 or hybrid path produces a recall score below the RECALL_FLOOR of 0.70 defined in crates/ai-memory-consolidate/tests/recall_eval.rs. This indicates a regression in the retrieval stack, such as broken ranking logic, missing index entries, or malformed query parsing that fails to surface relevant documents.

Can I run the benchmark without an OpenAI API key?

Yes. The harness defaults to a deterministic synthetic embedder that generates consistent vector representations without network calls. Only set AI_MEMORY_EMBEDDING_PROVIDER and related variables if you specifically want to benchmark the hybrid path using production embedding models like OpenAI's text-embedding-3-small.

Where is the synthetic test data defined?

The test corpus is hard-coded in the CORPUS constant within crates/ai-memory-consolidate/tests/recall_eval.rs. This array contains ten hand-crafted wiki pages covering fictional topics, explicitly designed to test edge cases in entity extraction and vector similarity without requiring external data dependencies.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}