How OmniRoute Manages Conversational Context: Extraction, Injection, Retrieval, and Summarization

OmniRoute implements a persistent four-stage memory pipeline that extracts facts from LLM responses, injects relevant context into prompts, retrieves memories via hybrid FTS5 and vector search, and summarizes aging memories to maintain token efficiency—all backed by SQLite.

OmniRoute is an open-source LLM gateway that maintains conversational continuity across requests through a sophisticated memory subsystem. Unlike simple prompt buffering, this system persists facts, summaries, and semantic embeddings in SQLite, enabling long-running conversations that exceed typical context windows. The architecture operates through four distinct stages—extraction, injection, retrieval, and summarization—each implemented as dedicated modules in the src/lib/memory/ directory.

Stage 1: Conversational Context Extraction

The extraction phase captures factual information from LLM responses for persistent storage. In src/lib/memory/extraction.ts, the pipeline defines the ExtractedFact interface and exposes extractFactsFromText() and the async extractFacts() functions.

Afterstreaming a response back to the client, OmniRoute parses the LLM output to identify factual statements. Each extracted fact becomes a new row in the memories table with MemoryType.FACTUAL. For vector-based semantic storage, the system also creates MemoryType.SEMANTIC records using the minimal SQLite-vector implementation found in src/lib/memory/vectorStore.ts.

// Conceptual flow in extraction.ts
interface ExtractedFact {
  content: string;
  confidence: number;
  timestamp: Date;
}

async function extractFacts(text: string): Promise<ExtractedFact[]>

This process ensures that user preferences, stated facts, and key details survive beyond the immediate request lifecycle.

Stage 2: Memory Injection into Prompts

Before sending requests upstream, OmniRoute evaluates whether to enrich the prompt with historical context. The src/lib/memory/injection.ts module contains shouldInjectMemory(), injectMemory(), and formatMemoryContext() to handle this logic.

When enabled, retrieveMemories() fetches relevant records by session ID, API key, or custom query, then formatMemoryContext() concatenates these into a system message or appends them to the user prompt. Developers can bypass this behavior entirely by including the x-omniroute-no-memory: true header or setting the noMemory request flag, which forces shouldInjectMemory() to return false and saves token budget.

POST /api/v1/chat/completions HTTP/1.1
x-omniroute-no-memory: true
Content-Type: application/json
Authorization: Bearer <user-api-key>

{
  "model": "gpt-4o",
  "messages": [{ "role": "user", "content": "What is the weather?" }]
}

This injection mechanism ensures the LLM receives pertinent background without manual conversation management.

Stage 3: Hybrid Memory Retrieval (FTS5 + Vector Search)

Retrieving the most relevant context requires balancing keyword precision with semantic similarity. The src/lib/memory/retrieval.ts module implements retrieveMemories(), retrievePreview(), and engineStatus() to orchestrate this hybrid approach.

The system combines SQLite FTS5 full-text search with optional vector similarity queries against the built-in vector store. Results are ranked using getRelevanceScore() from src/lib/memory/retrieval/scoring.ts, which balances exact match accuracy with embedding proximity. This dual strategy ensures that specific terms (like product names or dates) surface alongside conceptually related content (like "vacation plans" matching "trip to Japan").

curl "https://my.omniroute.dev/api/memory/search?q=travel&limit=5"

The JSON response includes id, content, and a relevance score generated by the scoring engine, allowing the injection layer to prioritize the most pertinent memories.

Stage 4: Memory Summarization for Token Efficiency

As conversations grow, raw memory accumulation threatens to exceed token limits. OmniRoute addresses this through src/lib/memory/summarization.ts, which provides summarizeMemories() and summarizeMemoriesOlderThan() to compress aging records.

When triggered—either periodically or via admin endpoint—the system sends batches of memories to the LLM with a specialized summarization prompt. The resulting condensed text is stored as a new memory record with MemoryType.SUMMARY. This compaction reduces the total token count while preserving essential context boundaries defined in src/lib/memory/settings.ts via MEMORY_MAX_SIZE and MEMORY_TYPED_DECAY_* environment variables.

POST /api/memory/summarize?olderThanDays=30 HTTP/1.1
Authorization: Bearer <admin-api-key>

The endpoint processes eligible memories, generates summaries, and writes them back to the database, maintaining long-term conversational coherence without unbounded growth.

Memory Types and Configuration

All memory records conform to the MemoryType enum defined in src/lib/memory/types.ts:

• FACTUAL: Explicit facts extracted from LLM outputs
• EPISODodic: Specific conversation events or interactions
• SEMANTIC: Vector embeddings for similarity search
• PROCEDURAL: System-level patterns or instructions
• SUMMARY: Compressed representations of memory batches

Configuration defaults and limits reside in src/lib/memory/settings.ts, allowing operators to tune retention policies, decay rates, and maximum storage thresholds without code changes.

REST API for Memory Operations

OmniRoute exposes the memory subsystem through REST endpoints documented in docs/reference/API_REFERENCE.md:

• List memories: GET /api/memory (filters: apiKeyId, type, sessionId, q)
• Create memory: POST /api/memory (validated by MemoryCreateInputSchema)
• Retrieve single memory: GET /api/memory/[id]
• Search: GET /api/memory/search (FTS5 + vector hybrid)
• Clear memories: POST /api/memory/clear
• Health check: GET /api/memory/health

Manual memory creation allows external systems to inject domain knowledge:

POST /api/memory HTTP/1.1
Content-Type: application/json
Authorization: Bearer <management-api-key>

{
  "content": "User said they love sushi.",
  "type": "EPISODIC",
  "sessionId": "sess_abc123",
  "apiKeyId": "key_xyz"
}

Summary

• Extraction occurs via src/lib/memory/extraction.ts, where extractFactsFromText() parses LLM responses into FACTUAL and SEMANTIC memory records stored in SQLite.
• Injection happens in src/lib/memory/injection.ts through injectMemory() and formatMemoryContext(), which can be disabled per-request using the x-omniroute-no-memory header.
• Retrieval combines FTS5 and vector search in src/lib/memory/retrieval.ts, scoring results with getRelevanceScore() to find relevant context.
• Summarization compresses older memories via summarizeMemoriesOlderThan() in src/lib/memory/summarization.ts, creating SUMMARY type records to bound token usage.
• The system supports five memory types defined in types.ts, with configurable limits in settings.ts and full CRUD access via REST endpoints.

Frequently Asked Questions

How does OmniRoute extract facts from LLM responses?

After streaming a response to the client, OmniRoute runs extractFacts() from src/lib/memory/extraction.ts, which calls extractFactsFromText() to identify factual statements. These are persisted as MemoryType.FACTUAL records in SQLite, while semantic embeddings are stored as MemoryType.SEMANTIC via the vector store.

Can I disable memory injection for specific requests?

Yes. Include the header x-omniroute-no-memory: true or set the noMemory flag in the request payload. This causes shouldInjectMemory() in src/lib/memory/injection.ts to skip the retrieval and formatting steps, preventing context from being added to that specific prompt.

What search methods does OmniRoute use for memory retrieval?

OmniRoute employs a hybrid approach in src/lib/memory/retrieval.ts that combines SQLite FTS5 for full-text search with optional vector similarity search through the built-in vectorStore.ts. The getRelevanceScore() function ranks results to surface the most pertinent memories for injection.

How does OmniRoute handle memory size limits?

The system uses summarizeMemories() and summarizeMemoriesOlderThan() from src/lib/memory/summarization.ts to compress batches of older memories into concise SUMMARY records. This process respects the MEMORY_MAX_SIZE and MEMORY_TYPED_DECAY_* environment variables defined in src/lib/memory/settings.ts, ensuring token usage remains bounded while preserving critical conversational context.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}