# How to Use OmniRoute's Memory System with Int8 Vector Quantization: Complete Implementation Guide

> Implement int8 vector quantization in OmniRoute by setting MEMORY_VEC_QUANTIZATION=int8. Reduce storage by ~75% with SQLite vec_quantize_int8 while keeping retrieval accuracy.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: how-to-guide
- Published: 2026-08-15

---

**Enable int8 vector quantization in OmniRoute by setting `MEMORY_VEC_QUANTIZATION=int8`, which automatically converts 32-bit float embeddings to 8-bit integers using SQLite's `vec_quantize_int8` function, reducing storage by ~75% while maintaining retrieval accuracy.**

OmniRoute provides a hybrid SQLite-based vector store for persisting user memories with optional **int8 vector quantization**. This guide explains how to configure, use, and migrate the memory system to int8 format based on the v3.8.50 source code in the `diegosouzapw/OmniRoute` repository.

## How Int8 Quantization Works in OmniRoute's Memory System

OmniRoute's memory system stores text chunks as vector embeddings in a virtual table called `vec_memories`. When **int8 quantization** is enabled, the system switches from the default `FLOAT[dim]` column type to a compact int8 representation.

The quantization pipeline follows three core steps in [`src/lib/memory/vectorStore.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/memory/vectorStore.ts):

1. **Embedding generation** — Your configured provider returns a `Float32Array` (typically 384, 768, or 1536 dimensions)
2. **Vector encoding** — `VectorStore.upsertVector()` calls `encodeVector()` to produce a little-endian byte buffer
3. **Quantization** — For int8 tables, the buffer passes through SQLite's built-in `vec_quantize_int8(?, 'unit')` function (lines 79-81), which unit-normalizes and compresses each float32 to a single signed byte

The resulting table signature changes from `memories:vec` to `memories:vec:int8`, triggering automatic re-indexing of all existing embeddings.

## Configuring Memory Vec Quantization via Environment Variables

OmniRoute reads quantization settings at runtime from environment variables. No code changes are required to enable int8 mode.

| Variable | Values | Effect |
|----------|--------|--------|
| `MEMORY_VEC_QUANTIZATION` | `none` (default), `int8` | Controls SQLite-vec column type |
| `MEMORY_VECTOR_STORE` | `sqlite-vec`, `qdrant`, `auto` | Selects backend implementation |
| `QDRANT_QUANTIZATION` | `none`, `int8` | Qdrant-specific scalar quantization |

Set these in your `.env` file or process environment before initializing the vector store:

```bash

# Enable int8 quantization with SQLite-vec backend

MEMORY_VEC_QUANTIZATION=int8
MEMORY_VECTOR_STORE=sqlite-vec

```

When `ensureReady()` runs (lines 91-95 of [`vectorStore.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/vectorStore.ts)), it calls `requestedVecQuantization()` to determine the active mode. The `addInt8SuffixToSignature()` helper (lines 105-108) bakes this into the table signature. Any signature mismatch forces a table reset and marks all rows with `needs_reindex=1`.

## Inserting and Searching Quantized Vectors

Once configured, the **int8 vector quantization** is transparent to application code. The same API works for both float and int8 modes.

### Storing a Memory with Automatic Quantization

```typescript
import { embedText } from "@/lib/memory/embedding";
import { getVectorStore } from "@/lib/memory/vectorStore";

async function addMemory(id: string, text: string) {
  // Generate embedding using your configured provider
  const { vector } = await embedText({ 
    model: "all-MiniLM-L6-v2", 
    text 
  });
  
  const vec = await getVectorStore();
  
  // Automatically uses int8 quantization due to MEMORY_VEC_QUANTIZATION=int8
  await vec.upsertVector(id, vector);
}

```

The `upsertVector()` method detects the table signature and routes through `vec_quantize_int8` when needed. The original float32 buffer is never stored—only the compressed int8 representation hits disk.

### Semantic Search with Int8 Vectors

```typescript
import { embedText } from "@/lib/memory/embedding";
import { getVectorStore } from "@/lib/memory/vectorStore";

async function semanticSearch(query: string, topK = 10) {
  const { vector } = await embedText({ 
    model: "all-MiniLM-L6-v2", 
    text: query 
  });
  
  const vec = await getVectorStore();

  // Pure K-NN: SQLite de-quantizes int8 vectors automatically during search
  const knn = await vec.searchVector(vector, topK);

  // Hybrid RRF: Combines vector similarity with FTS5 full-text ranking
  const hybrid = await vec.searchHybrid(vector, query, topK);
  
  return { knn, hybrid };
}

```

Both `searchVector()` and `searchHybrid()` accept float32 query vectors regardless of storage format. The SQLite `vec_search` virtual table handles de-quantization transparently during distance computation.

## Using Int8 Quantization with Qdrant Backend

OmniRoute also supports **Qdrant** as an alternative vector store with equivalent int8 capabilities. The implementation lives in [`src/lib/memory/qdrant.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/memory/qdrant.ts) (lines 9-14 define the `QdrantQuantization` enum).

When `QDRANT_QUANTIZATION=int8` is set, the client builds a scalar quantization configuration:

```typescript
// From qdrant.ts lines 38-41
{
  scalar: {
    type: "int8",
    always_ram: true,
    quantile: 0.99
  }
}

```

This tells Qdrant to:
- Store vectors as 8-bit scalars for memory efficiency
- Keep vectors in RAM for fast access (`always_ram: true`)
- Rescore the top results using original float vectors at the 99th percentile for accuracy

### Qdrant Int8 Configuration Example

```typescript
import { getVectorStore } from "@/lib/memory/vectorStore";

// Configure environment before first store access
process.env.MEMORY_VECTOR_STORE = "qdrant";
process.env.QDRANT_QUANTIZATION = "int8";
process.env.QDRANT_URL = "http://localhost:6333";

// Initializes Qdrant client with int8 scalar config
const vec = await getVectorStore();

```

## Handling Quantization Changes and Re-indexing

Changing `MEMORY_VEC_QUANTIZATION` mid-lifecycle requires rebuilding all embeddings. OmniRoute handles this automatically through its **re-indexing worker** in [`src/lib/memory/reindex.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/memory/reindex.ts).

When the table signature changes:
1. `ensureReady()` detects the mismatch
2. The virtual table is dropped and recreated with the new schema
3. All existing memories are marked `needs_reindex=1`
4. The background worker (line 92 of [`reindex.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/reindex.ts)) regenerates embeddings using the current provider and re-inserts them through `vec.upsertVector()`

You can also trigger manual re-indexing after model changes:

```typescript
import { triggerReindex } from "@/lib/memory/reindex";

// Force rebuild of all vectors with current quantization settings
await triggerReindex({ full: true });

```

## Performance and Accuracy Characteristics

**Storage reduction:** int8 quantization compresses 4-byte floats to 1-byte integers—a **75% reduction** in disk and memory bandwidth.

**Accuracy:** The `vec_quantize_int8` function with `'unit'` normalization preserves cosine similarity relationships sufficiently for most LLM retrieval tasks. Qdrant's rescoring step (enabled by default) further eliminates precision loss by re-ranking shortlists with full float vectors.

**Latency:** Compression reduces I/O overhead. The SQLite-vec extension performs de-quantization in native code during search, with minimal overhead compared to float storage.

## Summary

- Set `MEMORY_VEC_QUANTIZATION=int8` to enable 8-bit vector storage in OmniRoute's memory system
- The `VectorStore` class in [`src/lib/memory/vectorStore.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/memory/vectorStore.ts) handles quantization automatically via `vec_quantize_int8()`
- Table signatures encode the quantization mode; changes trigger automatic re-indexing via [`src/lib/memory/reindex.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/memory/reindex.ts)
- Qdrant backend supports equivalent int8 scalar quantization through `QDRANT_QUANTIZATION=int8`
- Search APIs remain unchanged—int8 de-quantization happens transparently during query execution

## Frequently Asked Questions

### How do I verify that int8 quantization is active?

Check the table signature in your SQLite database. An active int8 configuration shows `memories:vec:int8` rather than `memories:vec`. You can also inspect logs during `ensureReady()` initialization, which logs the detected quantization mode and any signature mismatches that trigger re-indexing.

### Can I switch from int8 back to float32 without data loss?

Switching quantization modes triggers a full re-index, not data loss. The original text content remains in the `memories` table. The `needs_reindex` flag forces regeneration of embeddings through your current provider, so you'll need that provider available. Back up your database before significant configuration changes.

### Does int8 quantization affect hybrid search quality?

No. The hybrid RRF implementation in `searchHybrid()` combines vector distances with FTS5 text scores after de-quantization. The int8 format is transparent to the ranking fusion step. Qdrant users additionally benefit from rescoring, which re-evaluates top candidates with float precision.