How OmniRoute's Memory System Manages Extraction, Injection, Retrieval, and Summarization

OmniRoute's memory subsystem is a modular TypeScript pipeline that extracts facts from LLM responses, injects them into a Qdrant vector store, retrieves relevant context using decay-aware scoring, and summarizes results before re-injection into prompts.

The memory system in OmniRoute (available at diegosouzapw/OmniRoute) transforms raw LLM output into persistent, searchable knowledge through a four-stage architecture. By isolating extraction, injection, retrieval, and summarization into distinct TypeScript modules, the system enables context-aware AI conversations while respecting per-session boundaries and configurable retention policies controlled via src/lib/memory/settings.ts.

The Four-Stage Memory Pipeline

OmniRoute implements a modular pipeline that processes natural language into retrievable vectors. Each stage handles a specific transformation from raw text to context-enriched prompts.

Stage 1: Extraction with Regex Patterns

Fact extraction begins in src/lib/memory/extraction.ts, where the extractFactsFromText function scans LLM responses for semantic patterns such as "I really prefer …" or "my favorite is …". Each match generates an ExtractedFact object that records the fact string, the originating apiKeyId, the sessionId, and a timestamp. The asynchronous extractFacts wrapper writes these facts to the memory store without blocking the request-handling thread.

Stage 2: Injection into Vector Storage

The injection layer in src/lib/memory/injection.ts receives ExtractedFact objects and persists them via createMemory. Facts are categorized by MemoryType (e.g., USER_PREFS, CHAT_CONTEXT) and embedded using providers defined in src/lib/memory/embedding/*.ts. The resulting vectors are stored in Qdrant (implemented in src/lib/memory/qdrant.ts), while auxiliary tables update retention policies and decay logic defined in src/lib/memory/typedDecay.ts.

Stage 3: Decay-Aware Retrieval

When new requests arrive, src/lib/memory/retrieval.ts builds query embeddings and executes similarity searches against the vector database. The scoring engine in src/lib/memory/retrieval/scoring.ts mixes cosine similarity with a time-decay factor from typedDecay.ts to favor recent memories. Results are deduplicated and bounded by the reindex step in reindex.ts, returning ranked MemoryItem objects ready for prompt enrichment.

Stage 4: Summarization for Token Efficiency

Before injection into outgoing prompts, src/lib/memory/summarization.ts compresses retrieved facts using a configurable summarizer (either a locally-run transformer or a remote LLM). This produces concise prose blocks that preserve semantic relevance while preventing token limit exhaustion. The summary then re-enters the injection pipeline to enrich the request payload sent to the LLM handler.

Orchestration via the Store API

Higher-level coordination occurs in src/lib/memory/store.ts, which exposes addFact, queryFacts, and clearMemory helpers used throughout the chat request pipeline. These methods enforce per-session isolation and respect per-API-key toggles configured in src/lib/memory/settings.ts, allowing fine-grained control over memory retention and decay policies.

Practical Implementation Examples

The following TypeScript snippets demonstrate the complete memory workflow.

Extract facts from an LLM response and inject them into the store:

import { extractFactsFromText } from '@/lib/memory/extraction';
import { addFact } from '@/lib/memory/store';

const llmResponse = "I really prefer dark mode and my favorite color is blue.";
const facts = extractFactsFromText(llmResponse);

for (const fact of facts) {
  await addFact({
    type: MemoryType.USER_PREFS,
    content: fact.fact,
    sessionId: 'session-123',
    apiKeyId: 'key-abc',
  });
}

Retrieve the most relevant facts for a new user message:

import { queryFacts } from '@/lib/memory/store';

const userMessage = "Can you set the UI to my usual style?";
const relevant = await queryFacts({
  query: userMessage,
  type: MemoryType.USER_PREFS,
  sessionId: 'session-123',
  limit: 5,
});

console.log('Relevant facts:', relevant.map(f => f.content));

Summarize retrieved facts before sending them to the LLM:

import { summarizeFacts } from '@/lib/memory/summarization';

const summary = await summarizeFacts(relevant);
const enrichedPrompt = `${summary}\n\n${userMessage}`;
// send `enrichedPrompt` to the LLM handler

Summary

• Extraction occurs in src/lib/memory/extraction.ts using regex patterns to create ExtractedFact objects with session metadata (apiKeyId, sessionId).
• Injection persists vectors via src/lib/memory/injection.ts using Qdrant storage and embedding providers, respecting decay policies from typedDecay.ts.
• Retrieval combines cosine similarity with time-decay scoring in src/lib/memory/retrieval/scoring.ts to rank recent memories highest.
• Summarization compresses facts in src/lib/memory/summarization.ts before re-injection, preventing token limit exhaustion.
• The store.ts API orchestrates all stages with addFact, queryFacts, and clearMemory, supporting per-API-key configuration via settings.ts.

Frequently Asked Questions

How does OmniRoute extract facts without blocking the request thread?

The extraction pipeline in src/lib/memory/extraction.ts runs asynchronously via the extractFacts function, which writes ExtractedFact objects to the store in a non-blocking manner while the main request-handling thread continues processing user requests.

Which vector database does OmniRoute use for memory storage?

According to the source code in src/lib/memory/qdrant.ts, OmniRoute uses Qdrant as its default vector store backend, though the embedding abstraction in src/lib/memory/embedding/*.ts allows for pluggable provider implementations.

How does the retrieval system prioritize recent memories over older ones?

The scoring engine in src/lib/memory/retrieval/scoring.ts mixes cosine similarity with a time-decay factor implemented in src/lib/memory/typedDecay.ts, exponentially reducing the relevance score of older facts while boosting recent entries in the final ranking.

Can memory features be disabled for specific API keys?

Yes. The src/lib/memory/settings.ts module supports per-API-key configuration toggles, allowing administrators to enable or disable memory extraction, injection, and retrieval for individual keys through the store.ts API.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}