# How the LLM Reranker Integrates with ai-memory’s Retrieval Pipeline

> Discover how the LLM reranker integrates into akitaonrails/ai-memory's retrieval pipeline as a final relevance scoring layer to semantically reorder results for improved accuracy.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: internals
- Published: 2026-08-25

---

**The LLM reranker integrates as a final relevance-scoring layer after reciprocal rank fusion (RRF), operating on an over-fetched candidate pool to semantically reorder results before returning them to the client.**

The `ai-memory` retrieval system employs a multi-stage pipeline to surface relevant pages from a knowledge graph. Understanding how the **LLM reranker** fits into this flow requires examining the precise handoff between candidate generation, fusion algorithms, and final result ordering in the `akitaonrails/ai-memory` codebase.

## Multi-Stage Retrieval Architecture

The pipeline processes queries through distinct phases before reaching the optional reranking stage.

### Candidate Generation Streams

Initially, the `memory_query` RPC gathers hits from four independent streams, as documented in [`docs/ARCHITECTURE.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/ARCHITECTURE.md) at line 353:

- **Full-text search (FTS5)** against indexed content
- **Entity-match tokens** extracted from the query
- **Graph-expansion neighbours** traversed from matched entities
- **Optional vector similarity** when embeddings are configured

Each stream produces a ranked list of `PageHit` objects. These lists remain separate until the fusion stage.

### RRF Fusion and Authority Adjustment

In [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs) (lines 305-320), the `rerank_page_hits` function merges the independent streams using **reciprocal rank fusion (RRF)** scores. The implementation adjusts ranks according to page authority signals—including pinned status, content tier, and tags—then truncates the fused list to the caller-requested limit (default ≤ 30 results).

## LLM Reranker Integration Point

The LLM reranker operates as a post-processing pass after RRF fusion completes but before the final response is serialized.

### Configuration and Activation

Integration is gated by the environment variable `AI_MEMORY_RERANKER=llm`. When set and a valid LLM provider is configured, the server injects one additional relevance pass over the fused candidates. Notably, global-scope queries (`global=true`) bypass the LLM reranker entirely, preserving their pure-FTS ranking according to the architecture documentation at [`docs/ARCHITECTURE.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/ARCHITECTURE.md) line 353.

### The Reranker Trait Interface

The contract is defined in [`crates/ai-memory-llm/src/reranker.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-llm/src/reranker.rs) (lines 53-66) by the `Reranker` trait:

```rust
pub trait Reranker: Send + Sync {
    fn name(&self) -> &str;
    fn model(&self) -> &str;
    async fn rerank(&self, query: &str, candidates: &[Candidate]) -> Result<Vec<RerankScore>>;
}

```

The built-in `LlmReranker` implementation (lines 71-89) packs the query and up to 30 candidate titles/snippets into a bounded JSON prompt (approximately 600 bytes per snippet) and transmits it to the configured LLM provider. Implementations return scores ∈ [0, 1] for each candidate.

### Server-Side Implementation Details

In [`crates/ai-memory-mcp/src/server.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-mcp/src/server.rs) (lines 1624-1641), the `rerank_hits` function orchestrates the integration:

1. **Over-fetching**: The candidate set is enlarged by a configurable factor (default ≈ 3× the final limit) so the LLM can promote hits that fell just below the cutoff (lines 1606-1614).
2. **Concurrency control**: A semaphore (`RERANK_MAX_IN_FLIGHT`) limits simultaneous LLM calls to prevent resource exhaustion.
3. **Timeout protection**: A hard `RERANK_TIMEOUT` prevents slow LLM responses from blocking the query pipeline.

If the LLM call fails, times out, or returns malformed scores, the server falls back to the pre-rerank order and logs a diagnostic entry (lines 1669-1687). When successful, the server reorders hits by the returned `rerank_score` values and includes these scores in the `explain` output (lines 1740-1750).

## Enabling and Using the LLM Reranker

Activate the integration via environment variables:

```bash

# .env or shell export

AI_MEMORY_RERANKER=llm
AI_MEMORY_LLM_PROVIDER=openai
AI_MEMORY_LLM_MODEL=gpt-4

```

Start the server with reranking enabled:

```bash
ai-memory serve

```

Query with explanation to observe rerank scores:

```bash
ai-memory query --query "consensus algorithm implementation" --limit 5 --explain

```

Programmatic usage via the MCP client:

```rust
use ai_memory_mcp::client::MemoryClient;

#[tokio::main]
async fn main() {
    let client = MemoryClient::new("http://127.0.0.1:49374");
    
    let resp = client
        .memory_query()
        .query("how does the consensus algorithm work?")
        .limit(10)
        .explain(true)
        .await
        .unwrap();

    for hit in resp.hits {
        println!("{} (score {:.2})", hit.path, hit.score);
        if let Some(rerank) = hit.explain.rerank_score {
            println!("  LLM rerank: {:.2}", rerank);
        }
    }
}

```

## Summary

- The **LLM reranker** sits after RRF fusion and authority scoring but before final result serialization.
- It evaluates an **over-fetched candidate pool** (3× the result limit) to allow recovery of high-relevance items initially ranked lower.
- Integration is controlled by the `AI_MEMORY_RERANKER=llm` environment variable and defined by the `Reranker` trait in `ai-memory-llm`.
- The implementation in `ai-memory-mcp` includes **concurrency limits** (`RERANK_MAX_IN_FLIGHT`) and **timeouts** (`RERANK_TIMEOUT`) to maintain pipeline responsiveness.
- **Graceful degradation** ensures that LLM failures do not break retrieval; the system falls back to the RRF-ordered results.

## Frequently Asked Questions

### Where exactly does the LLM reranker sit in the processing order?

The reranker executes after the four candidate streams undergo RRF fusion and authority adjustment in `ai-memory-store`, but before the final JSON response is constructed in `ai-memory-mcp`. It receives the truncated candidate list, expands it via over-fetching, and reorders based on semantic relevance scores returned by the LLM.

### What happens if the LLM reranker fails or times out?

According to the server implementation in [`crates/ai-memory-mcp/src/server.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-mcp/src/server.rs) (lines 1669-1687), any failure—including timeouts, network errors, or malformed JSON responses—triggers a fallback to the original RRF-ordered results. The system logs the failure for diagnostics but returns valid results to the client without interruption.

### How many candidates does the LLM reranker evaluate?

By default, the reranker evaluates up to 30 candidates, though the exact number depends on the query's `limit` parameter multiplied by an over-fetch factor (approximately 3×). The `LlmReranker` implementation packs these into a prompt with roughly 600 bytes allocated per snippet to stay within context window constraints.

### Can I use the LLM reranker with global scope queries?

No. As specified in [`docs/ARCHITECTURE.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/ARCHITECTURE.md) line 353, queries executed with `global=true` bypass the LLM reranker entirely. Global queries rely solely on full-text search ranking without the secondary semantic reordering pass to ensure deterministic performance across large datasets.