What Is AI_MEMORY_RERANKER=llm? Optional LLM Relevance Reranking in ai-memory

AI_MEMORY_RERANKER=llm enables a single-call LLM-based relevance reranking pass that reorders project-scope search results by sending the query and up to 30 candidate snippets to a configured provider for final scoring.

The ai-memory CLI and server support an optional quality enhancement for local knowledge base searches. When enabled, this environment variable triggers a secondary relevance pass using a Large Language Model (LLM) to intelligently reorder search hits before they are returned to the user. This feature targets project-scope queries specifically, providing semantic refinement on top of the standard full-text and vector search pipeline.

How AI_MEMORY_RERANKER=llm Works

When AI_MEMORY_RERANKER is set to llm, the server continues to perform its standard retrieval pipeline: FTS5 full-text search, entity matching, graph-based RRF (Reciprocal Rank Fusion), and optionally vector cosine similarity. After assembling the initial candidate pool, the system initiates a specialized reranking phase.

According to the architecture documentation in docs/ARCHITECTURE.md (lines 126-130), the server constructs a bounded payload containing the original query and up to 30 bounded titles/snippets from the candidate results. This JSON-encoded payload is sent to the configured LLM provider in a single request. The LLM evaluates the semantic relevance between the query and each candidate, returning a rerank score that the server uses to reorder the final result set.

Key Characteristics of the LLM Reranker

The implementation in crates/ai-memory-mcp/src/commands/serve.rs enforces strict constraints to maintain system stability:

• Single Call Per Query: Each search query triggers exactly one LLM inference call, regardless of result count.
• Concurrency Limiting: The system allows at most four concurrent LLM reranking calls to prevent provider rate limiting or local resource exhaustion.
• Graceful Fallback: If the LLM request is invalid, fails, times out, or hits the concurrency ceiling, the server immediately falls back to the original ordering from the FTS5/graph pipeline without erroring.
• Scope-Specific Operation: The reranker only processes project-scope and sibling-scope results. Global-scope searches (global=true) and supplemental global-preference hits bypass this step entirely.
• Privacy-Preserving Bounds: Both the query text and candidate snippets are strictly bounded and JSON-encoded before transmission, ensuring controlled token usage and data privacy.

Enabling and Configuring the Reranker

To activate LLM reranking, export the environment variable before starting the server. You must also configure a valid LLM provider via AI_MEMORY_LLM_PROVIDER.

# Enable LLM reranking (requires an LLM provider to be configured)

export AI_MEMORY_RERANKER=llm
export AI_MEMORY_LLM_PROVIDER=anthropic

# Start the server

ai-memory serve

As documented in docs/usage.md (lines 101-102), the variable accepts llm as its value to trigger the behavior described above. The configuration parser in crates/ai-memory-cli/src/config.rs validates that a provider is actually configured when this flag is set.

The following Rust example demonstrates a client request that would trigger the reranker (when enabled server-side):

// Example client request (project-scope query)
let resp = client
    .memory_query()
    .query("how to backup ai-memory")
    .scopes(true)               // project/scopes search
    .send()
    .await?;

// With AI_MEMORY_RERANKER=llm the response contains
// hits reordered by the LLM relevance score
println!("{:?}", resp.hits);

To disable the reranker and return to pure FTS5 ordering:

unset AI_MEMORY_RERANKER
ai-memory serve

Implementation Details

The reranking logic is implemented across three critical components:

1. docs/ARCHITECTURE.md: Defines the conceptual flow and constraints of the optional reranker pass, including the 30-candidate limit and the single-call design (lines 126-130).

2. crates/ai-memory-cli/src/config.rs: Parses the AI_MEMORY_RERANKER environment variable and validates that a compatible LLM provider configuration exists at startup.

3. crates/ai-memory-mcp/src/commands/serve.rs: Contains the runtime orchestration logic that intercepts project-scope search results, constructs the LLM prompt, manages the concurrency semaphore (max 4), and applies the returned scores to reorder hits.

Summary

• AI_MEMORY_RERANKER=llm adds an optional LLM-based relevance pass to project-scope searches in ai-memory.
• It sends one bounded request per query containing up to 30 candidates to your configured LLM provider.
• The feature includes strict concurrency limits (4 max) and automatic fallback to prevent service disruption.
• It only affects project/sibling scopes; global searches remain unaffected.
• Configuration is handled in crates/ai-memory-cli/src/config.rs and execution in crates/ai-memory-mcp/src/commands/serve.rs.

Frequently Asked Questions

Does AI_MEMORY_RERANKER=llm work with global searches?

No. According to the source implementation, the reranker specifically targets project-scope and sibling-scope results. When you query with global=true, the system bypasses the LLM reranker entirely and returns results ordered by the standard FTS5, entity match, and graph RRF pipeline.

What happens if the LLM provider times out?

The system implements graceful degradation. If the LLM request fails, times out, returns an invalid response, or exceeds the four-concurrent-call limit, ai-memory immediately returns the search results in their original order from the primary search pipeline. This ensures search availability even during LLM outages.

How many search results does the LLM rerank at once?

The reranker processes up to 30 candidates per query. As detailed in docs/ARCHITECTURE.md, the server bounds both the query text and the candidate titles/snippets, JSON-encodes them, and sends this limited set to the LLM. This design controls token costs and latency while focusing the LLM on the most promising candidates from the initial retrieval phase.

Can I use AI_MEMORY_RERANKER without configuring an LLM provider?

No. The configuration validator in crates/ai-memory-cli/src/config.rs requires a valid AI_MEMORY_LLM_PROVIDER to be set when AI_MEMORY_RERANKER=llm is enabled. The server will fail to start if the reranker flag is set without a compatible provider configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}