How the Memory System in OmniRoute Persists Conversational Context: Extraction, Injection, Retrieval, and Summarization

The memory system in OmniRoute persists conversational context by extracting factual statements from LLM responses, storing them in SQLite, retrieving relevant memories via hybrid full-text and vector search, injecting them into subsequent prompts, and automatically summarizing older entries to enforce token limits.

OmniRoute, the open-source AI routing gateway by diegosouzapw, implements a full-stack persistent memory layer that maintains state across stateless chat completions. This system operates through four distinct stages—extraction, injection, retrieval, and summarization—ensuring that long-running conversations retain critical context without overwhelming the LLM's context window.

The Four Stages of OmniRoute's Memory System

Extraction: Capturing Facts from LLM Responses

The extraction stage identifies and preserves factual information from every LLM response. In src/lib/memory/extraction.ts, the async extractFacts() pipeline processes incoming text through extractFactsFromText(), generating ExtractedFact objects that represent discrete pieces of information.

Each extracted fact becomes a persistent record with MemoryType.FACTUAL (or SEMANTIC for embedding vectors) written directly to SQLite. This stage runs automatically after the response streams back to the client, ensuring zero latency impact on the chat experience.

Key implementation details:

• Facts are extracted based on grammatical patterns and confidence scoring
• Metadata includes session ID, API key association, and timestamps
• Records link to the specific conversation thread for precise retrieval

Injection: Inserting Context into Prompts

Injection determines whether and how to insert stored memories into new requests. The src/lib/memory/injection.ts module defines shouldInjectMemory(), which checks for the x-omniroute-no-memory header or the noMemory request flag to conditionally bypass memory loading.

When injection is enabled, injectMemory() executes retrieveMemories() to fetch relevant context, then formats the results using formatMemoryContext(). These memories are concatenated into the system message or appended to the user prompt before the request routes to the upstream provider.

The system supports flexible scoping:

• Session-based: Retrieves memories from the current sessionId
• API-key based: Shares memory across all sessions using the same key
• Custom query: Filters by specific content patterns or memory types

Retrieval: Hybrid Search for Relevant Memories

Retrieval locates the most pertinent historical context using a hybrid approach defined in src/lib/memory/retrieval.ts. The retrieveMemories() function combines SQLite's native FTS5 full-text search with optional vector similarity searches through the built-in vectorStore.ts implementation.

Relevance scoring occurs in retrieval/scoring.ts via getRelevanceScore(), which ranks results by semantic proximity and recency. The system calculates similarity scores against the current conversation embedding, ensuring that injected memories remain contextually appropriate rather than merely keyword-matched.

The retrieval layer supports preview functionality through retrievePreview(), allowing administrators to audit which memories would be injected for a given query without executing the full pipeline.

Summarization: Compressing Historical Context

Summarization prevents memory bloat by collapsing older or voluminous memory piles into dense summaries. The src/lib/memory/summarization.ts module provides summarizeMemories() and summarizeMemoriesOlderThan(), which batch-process stale entries through the LLM using specialized condensation prompts.

When triggered, the function:

1. Selects memories exceeding the age or count thresholds defined in src/lib/memory/settings.ts (controlled by MEMORY_MAX_SIZE and MEMORY_TYPED_DECAY_* environment variables)
2. Sends the batch to the LLM with a summarization prompt
3. Stores the resulting compact text as a new memory with MemoryType.SUMMARY
4. Archives or removes the original detailed entries

This process maintains conversational coherence while keeping token usage bounded for the injection stage.

How the Memory Pipeline Works in Practice

The complete request-response cycle demonstrates how these stages interact:

1. Request arrival: Middleware checks shouldInjectMemory() for the incoming /api/v1/chat/completions request
2. Context loading: If enabled, retrieveMemories() fetches relevant facts using the session ID and query parameters, scored by getRelevanceScore()
3. Prompt assembly: formatMemoryContext() converts memories into formatted strings merged into the payload
4. Upstream routing: The enriched request routes to the selected provider (OpenAI, Anthropic, etc.)
5. Response processing: After streaming the response to the client, extractFacts() parses the LLM output via extractFactsFromText()
6. Persistence: Each extracted fact writes to the memories table as MemoryType.FACTUAL
7. Maintenance: Background jobs periodically execute summarizeMemoriesOlderThan() to compress entries older than the configured threshold

Configuration and Storage Architecture

All memory operations persist to SQLite with optional Qdrant vector store backing for high-performance semantic search. The src/lib/memory/settings.ts file centralized configuration through environment variables:

• MEMORY_MAX_SIZE: Hard limit on total memory entries per scope
• MEMORY_TYPED_DECAY_*: Type-specific TTL configurations (factual, episodic, semantic)
• Vector store toggles for enabling cosine similarity vs. pure FTS5

The src/lib/memory/types.ts module defines the MemoryType enum categorizing records as FACTUAL, EPISODIC, SEMANTIC, PROCEDURAL, or SUMMARY, allowing the retrieval engine to weight different memory categories during scoring.

REST API Endpoints for Memory Management

OmniRoute exposes memory operations through authenticated REST endpoints:

List memories with filtering

curl "https://api.omniroute.dev/api/memory?sessionId=sess_abc123&type=FACTUAL" \
  -H "Authorization: Bearer <management-api-key>"

Create a manual memory entry

POST /api/memory HTTP/1.1
Content-Type: application/json
Authorization: Bearer <management-api-key>

{
  "content": "User prefers Python over JavaScript for data tasks",
  "type": "EPISODIC",
  "sessionId": "sess_abc123",
  "apiKeyId": "key_xyz"
}

Search with hybrid scoring

curl "https://api.omniroute.dev/api/memory/search?q=programming+preferences&limit=5"

Returns JSON containing memory IDs, content, and relevance scores generated by getRelevanceScore().

Trigger administrative summarization

POST /api/memory/summarize?olderThanDays=30 HTTP/1.1
Authorization: Bearer <admin-api-key>

This executes summarizeMemoriesOlderThan(), compressing memories older than 30 days into SUMMARY type records.

Disable memory for specific requests

POST /api/v1/chat/completions HTTP/1.1
x-omniroute-no-memory: true
Content-Type: application/json

{
  "model": "gpt-4o",
  "messages": [{"role": "user", "content": "Execute one-off command"}]
}

Setting the header bypasses retrieveMemories(), reducing latency and token consumption for stateless queries.

Summary

• Extraction captures factual statements from LLM responses in src/lib/memory/extraction.ts, storing them as MemoryType.FACTUAL records in SQLite immediately after streaming completes
• Injection inserts relevant context into prompts via src/lib/memory/injection.ts, controllable through the x-omniroute-no-memory header or noMemory flag
• Retrieval employs hybrid FTS5 and vector search in src/lib/memory/retrieval.ts, ranking results with getRelevanceScore() from retrieval/scoring.ts
• Summarization compresses historical memories using src/lib/memory/summarization.ts to enforce limits set by MEMORY_MAX_SIZE and decay settings
• All stages operate through the REST API surface documented in docs/reference/API_REFERENCE.md, supporting manual CRUD operations and administrative maintenance

Frequently Asked Questions

How does OmniRoute decide which memories to inject into a conversation?

OmniRoute uses shouldInjectMemory() in src/lib/memory/injection.ts to check for opt-out flags, then calls retrieveMemories() which ranks candidates using hybrid FTS5 full-text search and vector similarity from src/lib/memory/vectorStore.ts. The scoring function getRelevanceScore() weights results by semantic proximity to the current query and recency, returning only the top-ranked entries that fit within the configured token budget.

Can I disable memory persistence for specific API requests?

Yes. Send the header x-omniroute-no-memory: true or include "noMemory": true in the request body to bypass the entire injection pipeline. This prevents retrieveMemories() from executing, ensuring the request processes with zero historical context and reduced latency.

What is the difference between FACTUAL and SUMMARY memory types?

FACTUAL memories contain direct extracted statements from LLM responses, created by extractFacts() in src/lib/memory/extraction.ts. SUMMARY memories are compressed aggregations generated by summarizeMemories() in src/lib/memory/summarization.ts, which condense multiple older factual or episodic entries into dense narrative form to save token space while preserving conversational continuity.

How does the summarization process prevent memory from growing indefinitely?

The summarizeMemoriesOlderThan() function in src/lib/memory/summarization.ts runs periodically (or via administrative endpoint) to batch-process memories exceeding age thresholds defined by MEMORY_TYPED_DECAY_* environment variables. These batches are sent to the LLM with condensation prompts, and the resulting compact text replaces the original entries as new SUMMARY type records, enforcing the MEMORY_MAX_SIZE limits configured in src/lib/memory/settings.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}