# What Is AI_MEMORY_RERANKER=llm? Optional LLM Relevance Reranking in ai-memory

> Discover the purpose of AI_MEMORY_RERANKER=llm. This optional pass uses LLM relevance reranking to reorder search results for improved accuracy in ai-memory.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: deep-dive
- Published: 2026-09-12

---

**`AI_MEMORY_RERANKER=llm` enables a single-call LLM-based relevance reranking pass that reorders project-scope search results by sending the query and up to 30 candidate snippets to a configured provider for final scoring.**

The `ai-memory` CLI and server support an optional quality enhancement for local knowledge base searches. When enabled, this environment variable triggers a secondary relevance pass using a Large Language Model (LLM) to intelligently reorder search hits before they are returned to the user. This feature targets project-scope queries specifically, providing semantic refinement on top of the standard full-text and vector search pipeline.

## How AI_MEMORY_RERANKER=llm Works

When `AI_MEMORY_RERANKER` is set to `llm`, the server continues to perform its standard retrieval pipeline: **FTS5** full-text search, entity matching, graph-based RRF (Reciprocal Rank Fusion), and optionally **vector cosine similarity**. After assembling the initial candidate pool, the system initiates a specialized reranking phase.

According to the architecture documentation in [`docs/ARCHITECTURE.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/ARCHITECTURE.md) (lines 126-130), the server constructs a bounded payload containing the original query and **up to 30 bounded titles/snippets** from the candidate results. This JSON-encoded payload is sent to the configured LLM provider in a single request. The LLM evaluates the semantic relevance between the query and each candidate, returning a rerank score that the server uses to reorder the final result set.

## Key Characteristics of the LLM Reranker

The implementation in [`crates/ai-memory-mcp/src/commands/serve.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-mcp/src/commands/serve.rs) enforces strict constraints to maintain system stability:

- **Single Call Per Query**: Each search query triggers exactly one LLM inference call, regardless of result count.
- **Concurrency Limiting**: The system allows at most **four concurrent LLM reranking calls** to prevent provider rate limiting or local resource exhaustion.
- **Graceful Fallback**: If the LLM request is invalid, fails, times out, or hits the concurrency ceiling, the server immediately falls back to the original ordering from the FTS5/graph pipeline without erroring.
- **Scope-Specific Operation**: The reranker only processes **project-scope** and sibling-scope results. Global-scope searches (`global=true`) and supplemental global-preference hits bypass this step entirely.
- **Privacy-Preserving Bounds**: Both the query text and candidate snippets are strictly bounded and JSON-encoded before transmission, ensuring controlled token usage and data privacy.

## Enabling and Configuring the Reranker

To activate LLM reranking, export the environment variable before starting the server. You must also configure a valid LLM provider via `AI_MEMORY_LLM_PROVIDER`.

```bash

# Enable LLM reranking (requires an LLM provider to be configured)

export AI_MEMORY_RERANKER=llm
export AI_MEMORY_LLM_PROVIDER=anthropic

# Start the server

ai-memory serve

```

As documented in [`docs/usage.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/usage.md) (lines 101-102), the variable accepts `llm` as its value to trigger the behavior described above. The configuration parser in [`crates/ai-memory-cli/src/config.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-cli/src/config.rs) validates that a provider is actually configured when this flag is set.

The following Rust example demonstrates a client request that would trigger the reranker (when enabled server-side):

```rust
// Example client request (project-scope query)
let resp = client
    .memory_query()
    .query("how to backup ai-memory")
    .scopes(true)               // project/scopes search
    .send()
    .await?;

// With AI_MEMORY_RERANKER=llm the response contains
// hits reordered by the LLM relevance score
println!("{:?}", resp.hits);

```

To disable the reranker and return to pure FTS5 ordering:

```bash
unset AI_MEMORY_RERANKER
ai-memory serve

```

## Implementation Details

The reranking logic is implemented across three critical components:

1. **[`docs/ARCHITECTURE.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/ARCHITECTURE.md)**: Defines the conceptual flow and constraints of the optional reranker pass, including the 30-candidate limit and the single-call design (lines 126-130).

2. **[`crates/ai-memory-cli/src/config.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-cli/src/config.rs)**: Parses the `AI_MEMORY_RERANKER` environment variable and validates that a compatible LLM provider configuration exists at startup.

3. **[`crates/ai-memory-mcp/src/commands/serve.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-mcp/src/commands/serve.rs)**: Contains the runtime orchestration logic that intercepts project-scope search results, constructs the LLM prompt, manages the concurrency semaphore (max 4), and applies the returned scores to reorder hits.

## Summary

- **`AI_MEMORY_RERANKER=llm`** adds an optional LLM-based relevance pass to project-scope searches in `ai-memory`.
- It sends **one bounded request** per query containing up to 30 candidates to your configured LLM provider.
- The feature includes **strict concurrency limits** (4 max) and **automatic fallback** to prevent service disruption.
- It **only affects project/sibling scopes**; global searches remain unaffected.
- Configuration is handled in [`crates/ai-memory-cli/src/config.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-cli/src/config.rs) and execution in [`crates/ai-memory-mcp/src/commands/serve.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-mcp/src/commands/serve.rs).

## Frequently Asked Questions

### Does AI_MEMORY_RERANKER=llm work with global searches?

No. According to the source implementation, the reranker specifically targets **project-scope** and sibling-scope results. When you query with `global=true`, the system bypasses the LLM reranker entirely and returns results ordered by the standard FTS5, entity match, and graph RRF pipeline.

### What happens if the LLM provider times out?

The system implements **graceful degradation**. If the LLM request fails, times out, returns an invalid response, or exceeds the four-concurrent-call limit, `ai-memory` immediately returns the search results in their original order from the primary search pipeline. This ensures search availability even during LLM outages.

### How many search results does the LLM rerank at once?

The reranker processes **up to 30 candidates** per query. As detailed in [`docs/ARCHITECTURE.md`](https://github.com/akitaonrails/ai-memory/blob/main/docs/ARCHITECTURE.md), the server bounds both the query text and the candidate titles/snippets, JSON-encodes them, and sends this limited set to the LLM. This design controls token costs and latency while focusing the LLM on the most promising candidates from the initial retrieval phase.

### Can I use AI_MEMORY_RERANKER without configuring an LLM provider?

No. The configuration validator in [`crates/ai-memory-cli/src/config.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-cli/src/config.rs) requires a valid `AI_MEMORY_LLM_PROVIDER` to be set when `AI_MEMORY_RERANKER=llm` is enabled. The server will fail to start if the reranker flag is set without a compatible provider configuration.