How Search Pipelines and Normalization Processors Function in OpenSearch

Search pipelines in OpenSearch execute configurable processors after query execution to normalize and combine relevance scores from hybrid searches, enabling fine-tuned ranking without modifying application code.

Amazon OpenSearch Service supports search pipelines as a mechanism to post-process query results before they return to the client. In the aws-samples/sample-for-amazon-opensearch-service-tutorials-101 repository, these pipelines leverage normalization processors to rescale and merge scores from hybrid keyword-and-vector searches. This article explains how search pipelines and normalization processors function based on the actual implementation in the sample project.

Understanding Search Pipelines in OpenSearch

A search pipeline is a series of processors that OpenSearch runs after executing the main query but before returning results to the client. Unlike ingest pipelines—which transform documents during indexing—search pipelines operate on the search response itself.

In the sample project, the pipeline serves a specific purpose: it normalizes the disparate score ranges produced by boolean keyword queries and k-NN vector similarity searches, then combines them using configurable weights. This decouples relevance tuning from query construction, allowing operators to adjust ranking behavior without redeploying application code.

How Normalization Processors Work

The normalization processor is a specialized search pipeline component that rescales raw relevance scores to a common scale before combining them. The sample project implements this in opensearch_index.py within the search_nlp() function.

Min-Max Normalization Technique

The processor uses the min_max normalization technique, which linearly rescales each sub-query's scores to the [0, 1] range. This ensures that a boolean query returning scores in the hundreds and a k-NN query returning cosine similarities between 0 and 1 contribute equally to the final ranking, preventing either algorithm from dominating due to raw score magnitude differences.

Score Combination Methods

After normalization, the processor combines scores using the arithmetic_mean technique. The sample project configures this with specific weights:

• 70% weight for the keyword (boolean) query results
• 30% weight for the k-NN vector query results

This weighting scheme prioritizes lexical matches while still incorporating semantic vector similarity.

Implementing Search Pipelines in the Sample Project

The repository demonstrates a complete implementation spanning pipeline creation and query-time attachment.

Creating the Pipeline Definition

In artifacts/index_lambda/opensearch_index.py, the search_nlp() function constructs the pipeline definition as a Python dictionary:

post_processor_search_pipleline = {
    "description": "Post processor for hybrid search",
    "phase_results_processors": [{
        "normalization-processor": {
            "normalization": {"technique": "min_max"},
            "combination": {
                "technique": "arithmetic_mean",
                "parameters": {"weights": [0.7, 0.3]}
            }
        }
    }]
}

The function then persists this configuration to OpenSearch via a PUT request to the /_search/pipeline/<pipeline_name> endpoint:

response = requests.put(
    f"https://{ENDPOINT}/_search/pipeline/{SEARCH_PIPELINE_NAME}",
    auth=awsauth,
    json=post_processor_search_pipleline,
    headers={'Content-Type': 'application/json'},
    verify=False
)

Attaching Pipelines to Queries

When executing searches in artifacts/search_lambda/opensearch_search.py, the application references the pipeline by including it in the search body:

search_body = {
    "size": 100,
    # query definition...

    "search_pipeline": SEARCH_PIPELINE_NAME
}

OpenSearch automatically applies the configured normalization processor to the query results before returning them to the Lambda function.

End-to-End Hybrid Search Flow

The sample project implements the following complete workflow:

1. Pipeline Creation: The search_nlp() function creates the search pipeline during the vector-indexing phase, defining the normalization and combination parameters.

2. Document Indexing: Products are indexed with both keyword fields (category, color, description) and vector embeddings (generated via Amazon Bedrock).

3. Hybrid Query Execution: The search Lambda constructs a hybrid query containing both a bool clause for keyword matching and a knn clause for vector similarity.

4. Score Normalization: OpenSearch executes both sub-queries, then the normalization-processor rescales the scores using min_max normalization.

5. Score Combination: The processor combines the normalized scores using arithmetic_mean with the configured weights (70% keyword, 30% vector).

6. Result Return: The final merged scores determine the hit ranking, which OpenSearch returns to the client.

This architecture enables operators to tune search relevance by adjusting the weights parameter or changing the technique values without modifying the application code that constructs the queries.

Summary

• Search pipelines in OpenSearch execute after query logic but before returning results, enabling post-processing of relevance scores.
• Normalization processors rescale disparate score ranges (such as boolean query scores versus k-NN similarities) to a common [0, 1] scale using techniques like min_max.
• The combination phase merges normalized scores using methods like arithmetic_mean with configurable weights, allowing fine-tuned control over hybrid search ranking.
• In the sample project, the pipeline is defined in opensearch_index.py by the search_nlp() function and attached to queries in opensearch_search.py via the search_pipeline parameter.

Frequently Asked Questions

What is the difference between an ingest pipeline and a search pipeline in OpenSearch?

An ingest pipeline processes documents during indexing, transforming or enriching data before it is stored in the index. A search pipeline processes query results after execution but before they are returned to the client, enabling operations like score normalization and result reranking. The sample project uses a search pipeline specifically to normalize and combine hybrid search scores without modifying the indexed documents.

When should I use min_max normalization versus other techniques?

Use min_max normalization when you need to rescale scores to a fixed [0, 1] range while preserving the relative ranking within each sub-query. This is ideal when combining boolean queries (which may produce unbounded scores) with k-NN searches (which typically return cosine similarities between 0 and 1). Other techniques like mean normalization might be preferable if you need to center scores around a statistical mean rather than bounding them to a specific range.

How do I update search pipeline weights without redeploying my application?

Update the pipeline configuration by sending a new PUT request to the /_search/pipeline/<pipeline_name> endpoint with modified weights parameters. For example, change the weights array in the JSON payload from [0.7, 0.3] to [0.5, 0.5] and re-run the curl or Python request. OpenSearch applies the new configuration immediately to all subsequent queries that reference that pipeline name, requiring no changes to the application code in opensearch_search.py.

Can I use search pipelines with non-hybrid queries?

Yes, search pipelines work with any query type, not just hybrid searches. While the sample project demonstrates normalization processors for combining boolean and k-NN scores, you can attach pipelines to standard keyword queries, geospatial searches, or aggregations. The processors will operate on the single result set, though normalization and combination logic provides the most value when merging disparate score types from multiple query clauses.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}