# On-Disk vs In-Memory kNN Vector Indexes in OpenSearch: Storage Modes Explained

> Explore on-disk vs in-memory kNN vector indexes in OpenSearch. Discover how on-disk saves RAM up to 97% while in-memory offers sub-100ms queries for optimal performance.

- Repository: [AWS Samples/sample-for-amazon-opensearch-service-tutorials-101](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101)
- Tags: deep-dive
- Published: 2026-02-25

---

**On-disk mode stores compressed kNN graphs in memory and full-precision vectors on disk to reduce RAM usage by up to 97%, while in-memory mode keeps the entire graph resident in RAM for the fastest possible sub-100ms queries.**

Amazon OpenSearch Service offers two distinct storage strategies for k-nearest neighbor (kNN) vector search that trade memory consumption against query latency. According to the `aws-samples/sample-for-amazon-opensearch-service-tutorials-101` repository, these modes are implemented through specific mapping configurations in the `knn_vector` field definition and demonstrated in a React-based comparison UI.

## How Storage Architectures Differ

The fundamental difference lies in where the vector data resides during query execution.

**On-disk mode** keeps only the compressed HNSW graph structure in RAM (approximately 1% of the full dataset size), while storing the complete, full-precision vectors on disk. During a search, OpenSearch performs a two-phase lookup: first selecting candidates from the in-memory compressed index, then fetching the actual vectors from disk for rescoring. As noted in [`artifacts/opensearch-app-ui/src/pages/vector-search-page.tsx`](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101/blob/main/artifacts/opensearch-app-ui/src/pages/vector-search-page.tsx) (lines 52-56), this approach reduces memory consumption by up to **97%** compared to traditional methods.

**In-memory mode** loads the entire kNN graph—including all full-precision vectors—directly into RAM. This enables single-phase lookups that bypass disk I/O entirely, delivering results in tens of milliseconds rather than the low-hundreds of milliseconds typical of on-disk queries (lines 245-247).

## Performance, Memory, and Cost Trade-offs

When selecting between these modes, consider three primary factors:

- **Memory consumption**: On-disk mode requires significantly less RAM because it retains only the compressed graph representation. The sample documentation indicates potential memory savings of up to **97%**, making it viable for large corpora that would otherwise require 10–100 GB of RAM.
- **Query latency**: In-memory queries complete in tens of milliseconds, while on-disk queries typically incur **100–200ms** latency due to the second-phase disk retrieval.
- **Infrastructure cost**: By enabling smaller instance types, on-disk mode can reduce EC2 memory costs by **67–83%**, whereas in-memory mode requires larger instances to hold the full vector dataset.

## Configuring kNN Storage Modes

You define the storage strategy during index creation through the `knn_vector` field mapping.

### On-Disk Configuration

To enable on-disk storage, explicitly set `"mode": "on_disk"` and optionally specify a compression level. In [`artifacts/index_lambda/opensearch_index.py`](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101/blob/main/artifacts/index_lambda/opensearch_index.py) (lines 92-98), the tutorial implements this as follows:

```python
res = ops_client.indices.create(
    index="products_vector_on_disk",
    body={
        "settings": {
            "index": {
                "knn": True,
                "knn.algo_param.ef_search": 100
            }
        },
        "mappings": {
            "properties": {
                "vector_embedding": {
                    "type": "knn_vector",
                    "dimension": 1024,
                    "data_type": "float",
                    "mode": "on_disk",                # Enables on-disk storage

                    "compression_level": "32x",       # Optional 32× compression

                    "method": {
                        "name": "hnsw",
                        "engine": "faiss",
                        "space_type": "innerproduct",
                        "parameters": {"ef_construction": 128, "m": 24}
                    }
                }
            }
        }
    }
)

```

### In-Memory Configuration

The default behavior uses in-memory storage when the `mode` property is omitted. Lines 60-66 of the same file demonstrate this configuration:

```python
res = ops_client.indices.create(
    index="products_vector_in_memory",
    body={
        "settings": {"index": {"knn": True, "knn.algo_param.ef_search": 100}},
        "mappings": {
            "properties": {
                "vector_embedding": {
                    "type": "knn_vector",
                    "dimension": 1024,
                    "method": {
                        "name": "hnsw",
                        "engine": "faiss",
                        "space_type": "innerproduct",
                        "parameters": {"ef_construction": 128, "m": 24}
                    }
                }
            }
        }
    }
)

```

## Querying Vector Indexes

Regardless of storage mode, you query kNN indexes using the same DSL syntax. The sample application builds requests as shown in [`artifacts/opensearch-app-ui/src/pages/vector-search-page.tsx`](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101/blob/main/artifacts/opensearch-app-ui/src/pages/vector-search-page.tsx) (lines 81-87):

```json
{
  "query": {
    "knn": {
      "field": "vector_embedding",
      "vector": [0.15, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
      "k": 10
    }
  }
}

```

OpenSearch automatically handles the retrieval strategy based on the index mapping—fetching from disk for rescoring in on-disk mode, or returning directly from RAM in in-memory mode.

## Visualizing Mode Differences in the Sample UI

The tutorial repository includes a React application that demonstrates both modes side-by-side. In [`artifacts/opensearch-app-ui/src/pages/vector-search-page.tsx`](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101/blob/main/artifacts/opensearch-app-ui/src/pages/vector-search-page.tsx) (lines 24-28), the UI renders timing comparisons:

```tsx
{onDiskTime > 0 && <Header variant="h3">
  On-Disk Mode ({onDiskTime}ms) ({onDiskHits} items)
</Header>}
{inMemoryTime > 0 && <Header variant="h3">
  In-Memory Mode ({inMemoryTime}ms) ({inMemoryHits} items)
</Header>}

```

This implementation allows developers to observe the latency differential between the two storage strategies using identical query workloads.

## Summary

- **On-disk mode** trades approximately 100–200ms of additional latency for up to 97% memory savings and 67–83% cost reduction, making it ideal for large-scale deployments.
- **In-memory mode** provides the lowest possible latency (tens of milliseconds) by storing full vectors in RAM, suitable for small-to-medium datasets requiring maximum query speed.
- **Configuration** requires adding `"mode": "on_disk"` to the `knn_vector` mapping for disk-based storage, or omitting the property entirely for in-memory operation.
- **Implementation** examples are available in [`artifacts/index_lambda/opensearch_index.py`](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101/blob/main/artifacts/index_lambda/opensearch_index.py) and the comparison UI in [`artifacts/opensearch-app-ui/src/pages/vector-search-page.tsx`](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101/blob/main/artifacts/opensearch-app-ui/src/pages/vector-search-page.tsx).

## Frequently Asked Questions

### How do I enable on-disk mode for kNN indexes in OpenSearch?

Add `"mode": "on_disk"` to your `knn_vector` field mapping during index creation, as implemented in [`artifacts/index_lambda/opensearch_index.py`](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101/blob/main/artifacts/index_lambda/opensearch_index.py) (lines 92-98). You can optionally specify `"compression_level": "32x"` to further reduce the in-memory footprint of the graph structure.

### What are the specific latency differences between the two modes?

According to the sample application code in [`artifacts/opensearch-app-ui/src/pages/vector-search-page.tsx`](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101/blob/main/artifacts/opensearch-app-ui/src/pages/vector-search-page.tsx) (lines 245-247), in-memory queries typically complete in tens of milliseconds, while on-disk queries range from 100–200ms due to the additional disk I/O required for the second-phase vector retrieval.

### When should I choose in-memory mode over on-disk mode?

Select **in-memory mode** when your workload consists of small-to-medium collections that demand absolute lowest latency and you can provision sufficient RAM (often 10–100 GB for large corpora). Choose **on-disk mode** for large vector collections where memory constraints exist or when cost optimization outweighs the need for sub-100ms response times.

### Does on-disk mode affect kNN search accuracy?

No. On-disk mode performs exact rescoring of candidates after retrieving full-precision vectors from disk, ensuring the same recall and accuracy as in-memory searches. The compression applies only to the graph index structure used for candidate selection, not the final vector comparison.