How to Configure kNN Vector Fields in OpenSearch Index Settings

To enable kNN vector fields in OpenSearch, set "knn": true in the index settings, define the field with "type": "knn_vector" and specify the "dimension" to match your embedding size, optionally tuning "knn.algo_param.ef_search" for accuracy.

The aws-samples/sample-for-amazon-opensearch-service-tutorials-101 repository demonstrates production-ready patterns for implementing semantic search using kNN vector fields. This tutorial extracts the exact configuration patterns used in the Lambda indexing functions and React frontend components to show you how to properly configure vector fields for sub-millisecond similarity searches.

Prerequisites for kNN Vector Configuration

Before configuring kNN fields, ensure your OpenSearch domain runs a version that supports the kNN plugin (2.x or later). The tutorial repository uses 512-dimensional embeddings generated by the generate_product_images_vectors.py script, so the index mapping must match this dimension exactly.

Enable kNN at the Index Level

To use kNN vector fields, you must first enable the kNN plugin for the entire index. This is done in the index settings JSON payload.

In artifacts/index_lambda/opensearch_index.py (lines 259-260 and 326-327), the Lambda function constructs the index creation request:

{
  "settings": {
    "knn": true,
    "knn.algo_param.ef_search": 100
  }
}

Key settings explained:

• "knn": true – Enables the kNN plugin for this index. Without this, OpenSearch rejects any knn_vector field mappings.
• "knn.algo_param.ef_search": 100 – Controls the size of the dynamic candidate list during search. Higher values increase accuracy at the cost of latency. The tutorial uses 100 as a balanced default.

Map the kNN Vector Field

After enabling kNN at the index level, you must define the specific field that stores vectors. In artifacts/index_lambda/opensearch_index.py (lines 293 and 360), the mapping defines the image_vector field:

{
  "mappings": {
    "properties": {
      "image_vector": {
        "type": "knn_vector",
        "dimension": 512
      }
    }
  }
}

Required mapping parameters:

• "type": "knn_vector" – Declares this field as a dense vector suitable for approximate nearest neighbor search.
• "dimension": 512 – Must exactly match the dimensionality of your embedding model. The tutorial uses 512-dimensional vectors from the image embedding model.

The knn_vector type automatically sets "index": true, so you do not need to explicitly enable indexing for the field.

Complete Implementation Example

Here is the complete Python implementation using boto3 and the opensearch-py client, following the patterns from opensearch_index.py:

import boto3
from opensearchpy import OpenSearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth

# Initialize the OpenSearch client

host = 'your-domain-endpoint.us-east-1.es.amazonaws.com'
region = 'us-east-1'
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

client = OpenSearch(
    hosts=[{'host': host, 'port': 443}],
    http_auth=awsauth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)

# Define index configuration

index_name = 'products'
index_body = {
    "settings": {
        "knn": True,
        "knn.algo_param.ef_search": 100
    },
    "mappings": {
        "properties": {
            "product_id": {"type": "keyword"},
            "title": {"type": "text"},
            "image_vector": {
                "type": "knn_vector",
                "dimension": 512
            }
        }
    }
}

# Create the index

response = client.indices.create(index=index_name, body=index_body)
print(f"Index created: {response}")

Querying kNN Vector Fields

Once configured, you query the vector field using the kNN query syntax. In artifacts/search_lambda/opensearch_search.py, the Lambda constructs queries like this:

{
  "size": 10,
  "query": {
    "knn": {
      "image_vector": {
        "vector": [0.12, 0.07, -0.03, ...],
        "k": 10
      }
    }
  }
}

The vector parameter contains the query embedding, and k specifies how many nearest neighbors to return. OpenSearch uses the HNSW graph built during indexing to return results in sub-millisecond time.

Summary

• Enable kNN at the index level by setting "knn": true in the index settings, as implemented in artifacts/index_lambda/opensearch_index.py (lines 259 and 326).
• Tune search accuracy with "knn.algo_param.ef_search" (recommended value: 100) to balance between recall and latency.
• Map vector fields using "type": "knn_vector" and specify the exact "dimension" to match your embedding model (512 in the tutorial).
• Query using kNN syntax with the knn query type, passing the query vector and desired number of neighbors (k).

Frequently Asked Questions

What happens if I don't set "knn": true in the index settings?

OpenSearch will reject the index creation request or fail to build the HNSW graph for vector fields. Without this setting, the knn_vector field type is not activated, and any kNN queries against that field will return errors stating the field is not searchable via kNN.

How do I choose the right value for knn.algo_param.ef_search?

Higher values increase search accuracy (recall) at the cost of increased latency. For most image similarity and semantic search use cases, a value between 100 and 200 provides a good balance. The tutorial repository uses 100 as a default. If you need higher precision and can tolerate slower queries, increase this to 500 or higher.

Can I change the dimension of a kNN vector field after creating the index?

No, you cannot modify the dimension of an existing knn_vector field. The dimension is fixed at index creation time and determines the size of the HNSW graph structure. If your embedding model changes dimensions, you must create a new index with the updated mapping and reindex your data.

Does enabling kNN on an index affect storage costs?

Yes, enabling kNN increases storage requirements because OpenSearch builds an HNSW (Hierarchical Navigable Small World) graph in memory and on disk for each knn_vector field. The graph size scales with the number of vectors and the dimensionality. For the 512-dimensional vectors used in the tutorial, expect approximately 10-20% additional storage overhead compared to storing the raw vectors alone.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}