How to Use OmniRoute's Memory System with Int8 Vector Quantization: Complete Implementation Guide

Enable int8 vector quantization in OmniRoute by setting MEMORY_VEC_QUANTIZATION=int8, which automatically converts 32-bit float embeddings to 8-bit integers using SQLite's vec_quantize_int8 function, reducing storage by ~75% while maintaining retrieval accuracy.

OmniRoute provides a hybrid SQLite-based vector store for persisting user memories with optional int8 vector quantization. This guide explains how to configure, use, and migrate the memory system to int8 format based on the v3.8.50 source code in the diegosouzapw/OmniRoute repository.

How Int8 Quantization Works in OmniRoute's Memory System

OmniRoute's memory system stores text chunks as vector embeddings in a virtual table called vec_memories. When int8 quantization is enabled, the system switches from the default FLOAT[dim] column type to a compact int8 representation.

The quantization pipeline follows three core steps in src/lib/memory/vectorStore.ts:

1. Embedding generation — Your configured provider returns a Float32Array (typically 384, 768, or 1536 dimensions)
2. Vector encoding — VectorStore.upsertVector() calls encodeVector() to produce a little-endian byte buffer
3. Quantization — For int8 tables, the buffer passes through SQLite's built-in vec_quantize_int8(?, 'unit') function (lines 79-81), which unit-normalizes and compresses each float32 to a single signed byte

The resulting table signature changes from memories:vec to memories:vec:int8, triggering automatic re-indexing of all existing embeddings.

Configuring Memory Vec Quantization via Environment Variables

OmniRoute reads quantization settings at runtime from environment variables. No code changes are required to enable int8 mode.

Variable	Values	Effect
MEMORY_VEC_QUANTIZATION	none (default), int8	Controls SQLite-vec column type
MEMORY_VECTOR_STORE	sqlite-vec, qdrant, auto	Selects backend implementation
QDRANT_QUANTIZATION	none, int8	Qdrant-specific scalar quantization

Set these in your .env file or process environment before initializing the vector store:

# Enable int8 quantization with SQLite-vec backend

MEMORY_VEC_QUANTIZATION=int8
MEMORY_VECTOR_STORE=sqlite-vec

When ensureReady() runs (lines 91-95 of vectorStore.ts), it calls requestedVecQuantization() to determine the active mode. The addInt8SuffixToSignature() helper (lines 105-108) bakes this into the table signature. Any signature mismatch forces a table reset and marks all rows with needs_reindex=1.

Inserting and Searching Quantized Vectors

Once configured, the int8 vector quantization is transparent to application code. The same API works for both float and int8 modes.

Storing a Memory with Automatic Quantization

import { embedText } from "@/lib/memory/embedding";
import { getVectorStore } from "@/lib/memory/vectorStore";

async function addMemory(id: string, text: string) {
  // Generate embedding using your configured provider
  const { vector } = await embedText({ 
    model: "all-MiniLM-L6-v2", 
    text 
  });
  
  const vec = await getVectorStore();
  
  // Automatically uses int8 quantization due to MEMORY_VEC_QUANTIZATION=int8
  await vec.upsertVector(id, vector);
}

The upsertVector() method detects the table signature and routes through vec_quantize_int8 when needed. The original float32 buffer is never stored—only the compressed int8 representation hits disk.

Semantic Search with Int8 Vectors

import { embedText } from "@/lib/memory/embedding";
import { getVectorStore } from "@/lib/memory/vectorStore";

async function semanticSearch(query: string, topK = 10) {
  const { vector } = await embedText({ 
    model: "all-MiniLM-L6-v2", 
    text: query 
  });
  
  const vec = await getVectorStore();

  // Pure K-NN: SQLite de-quantizes int8 vectors automatically during search
  const knn = await vec.searchVector(vector, topK);

  // Hybrid RRF: Combines vector similarity with FTS5 full-text ranking
  const hybrid = await vec.searchHybrid(vector, query, topK);
  
  return { knn, hybrid };
}

Both searchVector() and searchHybrid() accept float32 query vectors regardless of storage format. The SQLite vec_search virtual table handles de-quantization transparently during distance computation.

Using Int8 Quantization with Qdrant Backend

OmniRoute also supports Qdrant as an alternative vector store with equivalent int8 capabilities. The implementation lives in src/lib/memory/qdrant.ts (lines 9-14 define the QdrantQuantization enum).

When QDRANT_QUANTIZATION=int8 is set, the client builds a scalar quantization configuration:

// From qdrant.ts lines 38-41
{
  scalar: {
    type: "int8",
    always_ram: true,
    quantile: 0.99
  }
}

This tells Qdrant to:

• Store vectors as 8-bit scalars for memory efficiency
• Keep vectors in RAM for fast access (always_ram: true)
• Rescore the top results using original float vectors at the 99th percentile for accuracy

Qdrant Int8 Configuration Example

import { getVectorStore } from "@/lib/memory/vectorStore";

// Configure environment before first store access
process.env.MEMORY_VECTOR_STORE = "qdrant";
process.env.QDRANT_QUANTIZATION = "int8";
process.env.QDRANT_URL = "http://localhost:6333";

// Initializes Qdrant client with int8 scalar config
const vec = await getVectorStore();

Handling Quantization Changes and Re-indexing

Changing MEMORY_VEC_QUANTIZATION mid-lifecycle requires rebuilding all embeddings. OmniRoute handles this automatically through its re-indexing worker in src/lib/memory/reindex.ts.

When the table signature changes:

1. ensureReady() detects the mismatch
2. The virtual table is dropped and recreated with the new schema
3. All existing memories are marked needs_reindex=1
4. The background worker (line 92 of reindex.ts) regenerates embeddings using the current provider and re-inserts them through vec.upsertVector()

You can also trigger manual re-indexing after model changes:

import { triggerReindex } from "@/lib/memory/reindex";

// Force rebuild of all vectors with current quantization settings
await triggerReindex({ full: true });

Performance and Accuracy Characteristics

Storage reduction: int8 quantization compresses 4-byte floats to 1-byte integers—a 75% reduction in disk and memory bandwidth.

Accuracy: The vec_quantize_int8 function with 'unit' normalization preserves cosine similarity relationships sufficiently for most LLM retrieval tasks. Qdrant's rescoring step (enabled by default) further eliminates precision loss by re-ranking shortlists with full float vectors.

Latency: Compression reduces I/O overhead. The SQLite-vec extension performs de-quantization in native code during search, with minimal overhead compared to float storage.

Summary

• Set MEMORY_VEC_QUANTIZATION=int8 to enable 8-bit vector storage in OmniRoute's memory system
• The VectorStore class in src/lib/memory/vectorStore.ts handles quantization automatically via vec_quantize_int8()
• Table signatures encode the quantization mode; changes trigger automatic re-indexing via src/lib/memory/reindex.ts
• Qdrant backend supports equivalent int8 scalar quantization through QDRANT_QUANTIZATION=int8
• Search APIs remain unchanged—int8 de-quantization happens transparently during query execution

Frequently Asked Questions

How do I verify that int8 quantization is active?

Check the table signature in your SQLite database. An active int8 configuration shows memories:vec:int8 rather than memories:vec. You can also inspect logs during ensureReady() initialization, which logs the detected quantization mode and any signature mismatches that trigger re-indexing.

Can I switch from int8 back to float32 without data loss?

Switching quantization modes triggers a full re-index, not data loss. The original text content remains in the memories table. The needs_reindex flag forces regeneration of embeddings through your current provider, so you'll need that provider available. Back up your database before significant configuration changes.

Does int8 quantization affect hybrid search quality?

No. The hybrid RRF implementation in searchHybrid() combines vector distances with FTS5 text scores after de-quantization. The int8 format is transparent to the ranking fusion step. Qdrant users additionally benefit from rescoring, which re-evaluates top candidates with float precision.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}