How to Use the Memory System for Persistent Conversational Context in OmniRoute

OmniRoute provides a modular pipeline that extracts, embeds, stores, and retrieves conversation snippets using vector search to maintain context across disconnected sessions.

OmniRoute ships with a full-featured memory subsystem that persists conversational context across multiple requests through a durable vector store implementation. By leveraging Qdrant or SQLite backends combined with intelligent chunking in src/lib/memory/extraction.ts, the system remains stateless from the request handler perspective while enabling horizontal scaling and coherent multi-turn interactions.

Architectural Overview

The memory pipeline follows a strict lifecycle implemented across dedicated modules in src/lib/memory/. Each request flows through six phases: extraction, optional summarization, embedding, storage, retrieval, and injection.

Extraction and Chunking

In src/lib/memory/extraction.ts, incoming message payloads are parsed and sliced into discrete chunks. Each chunk receives metadata tags including a conversation-id and turn-index, creating traceable fragments ready for vectorization.

Vectorization and Storage

Chunks are converted to 1536-dimensional embeddings via the vectorize() function from the embeddings service. These vectors persist in either Qdrant (src/lib/memory/qdrant.ts) or SQLite (src/lib/memory/store.ts), depending on the configured provider. The store.ts module exposes storeChunk(), which orchestrates extraction, embedding, and persistence as a high-level façade.

Retrieval and Injection

When a new request arrives, src/lib/memory/retrieval.ts executes a similarity search using retrieveRelevantChunks(). The function embeds the current user prompt and performs nearest-neighbour lookup against the vector store. Retrieved snippets are then merged into the request payload by src/lib/memory/injection.ts, either prepended to the prompt or inserted as system messages.

Configuration and Initialization

Runtime behavior is controlled through src/lib/memory/settings.ts and validated by src/lib/memory/verify.ts using Zod schemas.

Environment Settings

Key toggles include MEMORY_ENABLED, MEMORY_MAX_TOKENS, and the vector store provider selection. Configure these via the settings API endpoint at src/app/api/settings/memory/route.ts:

POST /api/settings/memory
Content-Type: application/json

{
  "enabled": true,
  "maxTokens": 2048,
  "topK": 5,
  "provider": "qdrant"
}

This endpoint validates the payload and persists configuration to the database.

Storing Conversation Context

To persist a turn programmatically, import the storage façade from src/lib/memory/store.ts:

import { storeChunk } from "@/lib/memory/store";

// Persist a user message with session metadata
await storeChunk({
  conversationId: convId,
  messageId: msg.id,
  content: msg.content,
  metadata: { role: "user", timestamp: Date.now() },
});

Behind the scenes, this triggers extraction.ts to chunk the text, calls the embedding service, and commits the vector plus metadata to your configured backend.

Retrieving Context for New Requests

Access historical context using the retrieval module before forwarding requests to your LLM executor:

import { retrieveRelevantChunks } from "@/lib/memory/retrieval";

const context = await retrieveRelevantChunks({
  conversationId,
  prompt: newUserPrompt,
  topK: 5,
});

const enrichedPrompt = `${context.join("\n")}\n\n${newUserPrompt}`;

The function returns plain text snippets ranked by semantic similarity, ready for concatenation with new user input.

Integration in Request Pipeline

The complete lifecycle is orchestrated by src/app/api/memory/route.ts through its handleMemory function:

import { handleMemory } from "@/app/api/memory/route";

export async function POST(req: Request) {
  // Execute full pipeline: extract → store → retrieve → inject
  const enrichedReq = await handleMemory(req);
  
  // Forward enriched payload to target model
  const response = await executor.execute(enrichedReq);
  return response;
}

This design ensures the request handler remains stateless while the durable vector store maintains continuity across disconnected sessions.

Performance Optimization and Caching

The system implements an LRU cache in src/lib/memory/cache.ts to prevent redundant embedding API calls. When identical chunks are encountered, cached vectors are reused, reducing latency and API quota consumption.

For background maintenance, src/lib/memory/reindex.ts handles schema migrations and vector database synchronization without blocking live traffic.

Edge Cases and Fault Tolerance

Token Budget Enforcement: settings.ts caps injected context at MEMORY_MAX_TOKENS. When retrieved chunks exceed this limit, the system truncates from the oldest entries forward.

Privacy and Isolation: Memory scopes can be restricted per API key. Delete historical data via the settings route using DELETE /api/settings/memory.

Graceful Degradation: If the vector store becomes unavailable, verify.ts triggers a fallback to no-memory mode, allowing requests to proceed without context rather than failing entirely.

Summary

• OmniRoute's memory system uses a six-stage pipeline (extraction → summarization → embedding → storage → retrieval → injection) implemented in src/lib/memory/
• Stateless request handlers ensure horizontal scalability while the vector store maintains durability
• Configure via src/app/api/settings/memory/route.ts using environment variables like MEMORY_ENABLED and MEMORY_MAX_TOKENS
• Use storeChunk() from src/lib/memory/store.ts to persist turns and retrieveRelevantChunks() from src/lib/memory/retrieval.ts to fetch context
• LRU caching in src/lib/memory/cache.ts optimizes embedding costs by avoiding duplicate vector calculations

Frequently Asked Questions

What vector store backends does OmniRoute support?

OmniRoute ships with adapters for Qdrant (src/lib/memory/qdrant.ts) and SQLite (src/lib/memory/store.ts). You can extend support for alternatives like Pinecone by implementing the VectorStore interface used by the storage façade.

How does OmniRoute handle token limits when injecting memory?

The system enforces a MEMORY_MAX_TOKENS budget defined in src/lib/memory/settings.ts. Retrieved chunks are trimmed from the oldest entries when the cumulative token count exceeds this threshold, ensuring the final prompt stays within model context windows.

Can conversations be isolated between different API keys?

Yes. The memory system supports per-key scoping configured through src/lib/memory/settings.ts. You can purge data for specific keys via DELETE requests to /api/settings/memory, making it suitable for multi-tenant deployments.

What happens if the vector database is unavailable?

According to src/lib/memory/verify.ts, the pipeline implements graceful degradation. If the vector store connection fails, the system automatically falls back to no-memory mode, processing requests without historical context rather than returning errors.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}