# How OmniRoute’s Memory System Uses Vector Quantization to Reduce Memory Usage

> Discover how OmniRoute's memory system uses vector quantization to reduce RAM by 32x while maintaining search quality. Learn about on-the-fly dequantization and rescoring.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: deep-dive
- Published: 2026-08-10

---

**OmniRoute's memory system stores conversational embeddings in a pluggable backend—SQLite or Qdrant—and optionally applies vector quantization to reduce RAM by up to 32× while preserving search quality through on-the-fly dequantization and rescoring.**

The **vector quantization** capability in OmniRoute lets deployments trade storage efficiency for retrieval precision. This article explains how the system implements int8 and binary quantization across both SQLite and Qdrant backends, with specific reference to the source code in `diegosouzapw/OmniRoute`.

## SQLite Backend: Native int8 Quantization

The SQLite-based vector store ([`src/lib/memory/vectorStore.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/memory/vectorStore.ts)) supports **scalar int8 quantization** activated via the `MEMORY_VEC_QUANTIZATION` environment variable.

### Enabling and Applying Quantization

Set the environment variable before initialization:

```bash
export MEMORY_VEC_QUANTIZATION=int8

```

When active, the store transforms `float32` embeddings at the SQL layer. At line 80-88, the code documents the mode, and at lines 117-121, the insert path conditionally uses `vec_quantize_int8(?, 'unit')` as the placeholder:

```typescript
// From src/lib/memory/vectorStore.ts#L117-L121
const placeholder = quantizationMode === "int8"
  ? "vec_quantize_int8(?, 'unit')"
  : "?";

```

The `unit` parameter ensures vectors remain **unit-normalized** after quantization. The insert implementation (lines 167-219) folds this mode into the SQL signature—no schema migration is required; the table's column type determines whether conversion occurs.

### Storage and Retrieval Behavior

Quantized vectors occupy approximately **¼ of original memory**. During similarity search, SQLite returns the compressed representation and the application dequantizes on-the-fly, maintaining semantic compatibility with full-precision vectors.

## Qdrant Backend: Configurable int8 and Binary Quantization

For production deployments, OmniRoute integrates with **Qdrant** ([`src/lib/memory/qdrant.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/memory/qdrant.ts)) offering two quantization strategies:

| Mode | Memory Reduction | Characteristics |
|------|------------------|---------------|
| `int8` | ~4× | Scalar 8-bit; rescoring preserves ranking accuracy |
| `binary` | ~32× | Bit-level compression; more aggressive, slightly lossier |

### Configuration and Collection Setup

The quantization mode is read from `qdrantQuantization` in the global memory configuration (lines 107-111), defaulting to `"none"` if unset. The collection creation helper (lines 29-39) builds the appropriate `quantization_config` block:

```typescript
// From src/lib/memory/qdrant.ts#L29-L39
const quantizationConfig = mode === "int8"
  ? { scalar: { type: "int8", always_ram: true, quantile: 0.99 } }
  : mode === "binary"
    ? { binary: { always_ram: true } }
    : undefined;

```

### Search with Rescoring

Critical to quality preservation: quantized collections enable `rescore: true` (lines 49-57). This instructs Qdrant to retrieve candidates using compressed vectors, then re-rank the shortlist with original high-dimensional vectors:

```typescript
// From src/lib/memory/qdrant.ts#L49-L57
searchParams: {
  rescore: quantizationMode !== "none", // true for int8 or binary
  // ... other parameters
}

```

The test suite at [`src/lib/memory/__tests__/qdrant-wiring.test.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/memory/__tests__/qdrant-wiring.test.ts) (lines 85-110) validates this wiring, confirming that both `int8` and `binary` modes set `rescore: true` while invalid configurations fall back to no quantization.

## End-to-End Flow: From Embedding to Retrieval

The memory manager ([`src/lib/memory/manager.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/memory/manager.ts)) orchestrates quantization transparently:

1. **Generation** — The embedding module produces a `float32` vector from conversation text
2. **Routing** — The manager selects SQLite or Qdrant based on configuration
3. **Compression** — If enabled, the backend quantizes before persistence
4. **Search** — Queries use quantized storage but return results rescored against original precision

This pipeline ensures developers can switch quantization modes via configuration without changing application code.

## Practical Configuration Examples

Enable SQLite int8 quantization via environment:

```typescript
// Before any memory operations
process.env.MEMORY_VEC_QUANTIZATION = "int8";

await memoryManager.storeEmbedding({
  text: "User query about authentication...",
  embedding: new Float32Array(768).fill(0.1), // your embedding
  metadata: { conversationId: "conv-2024-001" },
});

```

Configure Qdrant for maximum memory efficiency:

```typescript
// Binary quantization for large-scale deployments
await memoryManager.configure({
  backend: "qdrant",
  qdrantUrl: process.env.QDRANT_URL,
  qdrantQuantization: "binary", // 32× RAM reduction
});

```

## Summary

- **OmniRoute supports vector quantization in both SQLite and Qdrant backends**, activated through environment variables or configuration
- **SQLite uses native `vec_quantize_int8` SQL functions** for ~4× memory reduction with automatic dequantization on retrieval
- **Qdrant offers `int8` and `binary` modes** with mandatory rescoring to preserve ranking quality
- **The memory manager abstracts backend differences**—embedding storage and retrieval APIs remain identical regardless of quantization settings
- **Quantization is opt-in and backward-compatible**; omitting configuration retains full `float32` precision

## Frequently Asked Questions

### How much memory does vector quantization actually save in OmniRoute?

SQLite's int8 mode reduces storage to approximately **25% of original size** (4× reduction). Qdrant's int8 mode achieves similar savings in RAM, while binary quantization yields **~3% of original size** (32× reduction). The trade-off is precision: binary quantization is more lossy and works best for high-dimensional embeddings where approximate similarity suffices.

### Does enabling quantization affect search accuracy in OmniRoute?

Search accuracy is preserved through **rescoring mechanisms**. SQLite dequantizes vectors on-the-fly during similarity computation. Qdrant retrieves candidates using quantized vectors but necessarily rescores the top results with original vectors when `rescore: true` is set. This dual-phase approach maintains ranking quality while benefiting from compressed storage.

### Which OmniRoute backend should I choose for vector quantization?

Choose **SQLite** for edge deployments, single-node setups, or when simplicity outweighs scale—quantification happens via built-in SQL functions with zero external dependencies. Choose **Qdrant** for distributed systems, millions of vectors, or when you need binary quantization's aggressive compression. Both support the same embedding API in [`src/lib/memory/manager.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/memory/manager.ts).

### How do I debug vector quantization issues in OmniRoute?

Verify the active mode by checking `MEMORY_VEC_QUANTIZATION` (SQLite) or `qdrantQuantization` configuration (Qdrant). Inspect the `quantization_config` in Qdrant's collection info, or confirm SQLite placeholder substitution in query logs. The test file [`src/lib/memory/__tests__/qdrant-wiring.test.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/memory/__tests__/qdrant-wiring.test.ts) demonstrates expected configuration shapes for validation.