How to Enable LLM Reranking for ai-memory Queries: Complete Configuration Guide

To enable LLM reranking for ai-memory queries, set the environment variable AI_MEMORY_RERANKER=llm and configure a compatible LLM provider via AI_MEMORY_LLM_PROVIDER; the system requires no code changes and activates automatically on server startup.

The ai-memory repository provides an optional LLM-based reranking layer that improves search relevance by reordering candidate results through a relevance scoring pass. This feature integrates seamlessly with existing FTS, lexical-entity, graph, and vector search pipelines entirely through configuration variables, requiring no modifications to application code.

Prerequisites and Configuration

The reranking system validates its dependencies during server initialization. According to the source code in crates/ai-memory-cli/src/config.rs (lines 200-210), the system checks for the AI_MEMORY_RERANKER flag when building the query service. If present, it injects a Reranker implementation into the query handler at crates/ai-memory-mcp/src/server.rs (lines 387-390).

Environment Variable Setup

Set the following variables before starting the server:

export AI_MEMORY_RERANKER=llm               # Required: enables the feature

export AI_MEMORY_LLM_PROVIDER=openai         # Required: selects the provider

export AI_MEMORY_LLM_MODEL=gpt-4o-mini       # Optional: specifies the model

ai-memory serve                               # Start with reranking active

The server validates that both the reranker and provider are present during startup. In crates/ai-memory-cli/src/commands/serve.rs (lines 1470-1474), the initialization logic aborts with an informative error if the configuration is inconsistent.

TOML Configuration File

Alternatively, enable reranking via the configuration template:

# crates/ai-memory-cli/templates/config.default.toml

reranker = "llm"           # Uncomment to activate LLM reranking

llm_provider = "openai"
llm_model = "gpt-4o-mini"

How the Reranking Pipeline Works

Once enabled, the query path proceeds through several controlled stages designed to balance relevance improvements with latency constraints.

Candidate Pool Retrieval

The system retrieves an initial candidate pool sized at limit × RERANK_OVERFETCH (defaulting to 30 candidates). This overfetching ensures the LLM has sufficient context to identify the most relevant items even when the initial retrieval order is imperfect.

Concurrency Control and Scoring

Before invoking the LLM, the server acquires a permit from a bounded semaphore controlled by RERANK_MAX_IN_FLIGHT (default 4) to limit concurrent calls. The Reranker::rerank method—defined in crates/ai-memory-llm/src/reranker.rs and invoked at crates/ai-memory-mcp/src/server.rs (lines 1640-1660)—accepts the original query and candidate snippets, returning relevance scores that reorder the final results.

A timeout (RERANK_TIMEOUT, default 2 seconds) ensures queries remain responsive even when the LLM service experiences latency.

Graceful Degradation

If the LLM call fails, times out, or the concurrency limit is reached, the system preserves the original pre-rerank order. This fallback logic is implemented in crates/ai-memory-mcp/src/server.rs (lines 1665-1680), ensuring that reranking failures never result in empty result sets.

Accessing Rerank Scores in Results

Each hit processed by the reranker receives an additional rerank_score field within its SearchExplain structure. This field makes the LLM's relevance contribution visible to callers requesting detailed explanations, as implemented in crates/ai-memory-store/src/reader.rs (lines 532-536).

To inspect these scores programmatically:

use ai_memory_mcp::client::MemoryClient;

#[tokio::main]
async fn main() {
    let client = MemoryClient::new("http://localhost:49374");
    let response = client
        .query("how to reset a password")
        .with_explain(true)      // Required to receive rerank_score
        .await
        .expect("query failed");
    
    for hit in response.hits {
        let score = hit.explain.as_ref()
            .and_then(|e| e.rerank_score)
            .unwrap_or(0.0);
        println!("→ {} (rerank score: {:.2})", hit.title, score);
    }
}

Performance Tuning

Three configuration knobs control the reranking behavior:

• RERANK_OVERFETCH (default: 30): Multiplier determining how many extra candidates to fetch for the LLM to consider. Higher values improve reranking quality at the cost of increased initial retrieval time.
• RERANK_MAX_IN_FLIGHT (default: 4): Maximum concurrent LLM calls permitted. Prevents overwhelming the provider API during high query volume.
• RERANK_TIMEOUT (default: 2s): Hard deadline for LLM scoring operations. Exceeding this triggers the fallback to lexical ordering.

Summary

• LLM reranking requires setting AI_MEMORY_RERANKER=llm and configuring a valid AI_MEMORY_LLM_PROVIDER.
• The feature activates automatically during server startup with validation occurring in crates/ai-memory-cli/src/commands/serve.rs.
• The system fetches up to 30× the result limit, scores candidates with bounded concurrency (max 4 calls), and applies a 2-second timeout.
• Failed or slow reranking operations gracefully fall back to the original retrieval order.
• Enable .with_explain(true) in client queries to access the rerank_score field added in crates/ai-memory-store/src/reader.rs.

Frequently Asked Questions

Is LLM reranking enabled by default?

No. The feature is disabled by default and must be explicitly activated by setting the AI_MEMORY_RERANKER environment variable to llm or uncommenting the reranker line in the configuration file.

Which LLM providers support reranking?

Any provider compatible with ai-memory's LLM client interface—including OpenAI, Anthropic, and other OpenAI-compatible endpoints—can power the reranker. The system validates provider availability at startup in crates/ai-memory-cli/src/commands/serve.rs.

What happens if the LLM service is unavailable?

The query pipeline preserves the original search results without modification. As implemented in crates/ai-memory-mcp/src/server.rs (lines 1665-1680), timeouts and API failures trigger an immediate fallback to the pre-rerank ordering, ensuring no query failures occur due to LLM issues.

How can I verify that reranking is active?

Check that your queries return rerank_score values in the SearchExplain structure by calling .with_explain(true) on your query. If reranking is disabled, this field will be None. Additionally, server logs confirm the Reranker injection during startup when AI_MEMORY_RERANKER=llm is set.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}