# How to Enable LLM Reranking for ai-memory Queries: Complete Configuration Guide

> Easily enable LLM reranking for ai-memory queries by setting environment variables. This complete guide shows you how to configure your LLM provider for automatic activation without code changes.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: how-to-guide
- Published: 2026-08-26

---

**To enable LLM reranking for ai-memory queries, set the environment variable `AI_MEMORY_RERANKER=llm` and configure a compatible LLM provider via `AI_MEMORY_LLM_PROVIDER`; the system requires no code changes and activates automatically on server startup.**

The ai-memory repository provides an optional **LLM-based reranking** layer that improves search relevance by reordering candidate results through a relevance scoring pass. This feature integrates seamlessly with existing FTS, lexical-entity, graph, and vector search pipelines entirely through configuration variables, requiring no modifications to application code.

## Prerequisites and Configuration

The reranking system validates its dependencies during server initialization. According to the source code in [`crates/ai-memory-cli/src/config.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-cli/src/config.rs) (lines 200-210), the system checks for the `AI_MEMORY_RERANKER` flag when building the query service. If present, it injects a `Reranker` implementation into the query handler at [`crates/ai-memory-mcp/src/server.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-mcp/src/server.rs) (lines 387-390).

### Environment Variable Setup

Set the following variables before starting the server:

```bash
export AI_MEMORY_RERANKER=llm               # Required: enables the feature

export AI_MEMORY_LLM_PROVIDER=openai         # Required: selects the provider

export AI_MEMORY_LLM_MODEL=gpt-4o-mini       # Optional: specifies the model

ai-memory serve                               # Start with reranking active

```

The server validates that both the reranker and provider are present during startup. In [`crates/ai-memory-cli/src/commands/serve.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-cli/src/commands/serve.rs) (lines 1470-1474), the initialization logic aborts with an informative error if the configuration is inconsistent.

### TOML Configuration File

Alternatively, enable reranking via the configuration template:

```toml

# crates/ai-memory-cli/templates/config.default.toml

reranker = "llm"           # Uncomment to activate LLM reranking

llm_provider = "openai"
llm_model = "gpt-4o-mini"

```

## How the Reranking Pipeline Works

Once enabled, the query path proceeds through several controlled stages designed to balance relevance improvements with latency constraints.

### Candidate Pool Retrieval

The system retrieves an initial **candidate pool** sized at `limit × RERANK_OVERFETCH` (defaulting to 30 candidates). This overfetching ensures the LLM has sufficient context to identify the most relevant items even when the initial retrieval order is imperfect.

### Concurrency Control and Scoring

Before invoking the LLM, the server acquires a permit from a bounded semaphore controlled by `RERANK_MAX_IN_FLIGHT` (default 4) to limit concurrent calls. The `Reranker::rerank` method—defined in [`crates/ai-memory-llm/src/reranker.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-llm/src/reranker.rs) and invoked at [`crates/ai-memory-mcp/src/server.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-mcp/src/server.rs) (lines 1640-1660)—accepts the original query and candidate snippets, returning relevance scores that reorder the final results.

A timeout (`RERANK_TIMEOUT`, default 2 seconds) ensures queries remain responsive even when the LLM service experiences latency.

### Graceful Degradation

If the LLM call fails, times out, or the concurrency limit is reached, the system preserves the original pre-rerank order. This fallback logic is implemented in [`crates/ai-memory-mcp/src/server.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-mcp/src/server.rs) (lines 1665-1680), ensuring that reranking failures never result in empty result sets.

## Accessing Rerank Scores in Results

Each hit processed by the reranker receives an additional `rerank_score` field within its `SearchExplain` structure. This field makes the LLM's relevance contribution visible to callers requesting detailed explanations, as implemented in [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs) (lines 532-536).

To inspect these scores programmatically:

```rust
use ai_memory_mcp::client::MemoryClient;

#[tokio::main]
async fn main() {
    let client = MemoryClient::new("http://localhost:49374");
    let response = client
        .query("how to reset a password")
        .with_explain(true)      // Required to receive rerank_score
        .await
        .expect("query failed");
    
    for hit in response.hits {
        let score = hit.explain.as_ref()
            .and_then(|e| e.rerank_score)
            .unwrap_or(0.0);
        println!("→ {} (rerank score: {:.2})", hit.title, score);
    }
}

```

## Performance Tuning

Three configuration knobs control the reranking behavior:

- **`RERANK_OVERFETCH`** (default: 30): Multiplier determining how many extra candidates to fetch for the LLM to consider. Higher values improve reranking quality at the cost of increased initial retrieval time.
- **`RERANK_MAX_IN_FLIGHT`** (default: 4): Maximum concurrent LLM calls permitted. Prevents overwhelming the provider API during high query volume.
- **`RERANK_TIMEOUT`** (default: 2s): Hard deadline for LLM scoring operations. Exceeding this triggers the fallback to lexical ordering.

## Summary

- LLM reranking requires setting `AI_MEMORY_RERANKER=llm` and configuring a valid `AI_MEMORY_LLM_PROVIDER`.
- The feature activates automatically during server startup with validation occurring in [`crates/ai-memory-cli/src/commands/serve.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-cli/src/commands/serve.rs).
- The system fetches up to 30× the result limit, scores candidates with bounded concurrency (max 4 calls), and applies a 2-second timeout.
- Failed or slow reranking operations gracefully fall back to the original retrieval order.
- Enable `.with_explain(true)` in client queries to access the `rerank_score` field added in [`crates/ai-memory-store/src/reader.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-store/src/reader.rs).

## Frequently Asked Questions

### Is LLM reranking enabled by default?

No. The feature is disabled by default and must be explicitly activated by setting the `AI_MEMORY_RERANKER` environment variable to `llm` or uncommenting the `reranker` line in the configuration file.

### Which LLM providers support reranking?

Any provider compatible with ai-memory's LLM client interface—including OpenAI, Anthropic, and other OpenAI-compatible endpoints—can power the reranker. The system validates provider availability at startup in [`crates/ai-memory-cli/src/commands/serve.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-cli/src/commands/serve.rs).

### What happens if the LLM service is unavailable?

The query pipeline preserves the original search results without modification. As implemented in [`crates/ai-memory-mcp/src/server.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-mcp/src/server.rs) (lines 1665-1680), timeouts and API failures trigger an immediate fallback to the pre-rerank ordering, ensuring no query failures occur due to LLM issues.

### How can I verify that reranking is active?

Check that your queries return `rerank_score` values in the `SearchExplain` structure by calling `.with_explain(true)` on your query. If reranking is disabled, this field will be `None`. Additionally, server logs confirm the `Reranker` injection during startup when `AI_MEMORY_RERANKER=llm` is set.