# How Search Pipelines and Normalization Processors Function in OpenSearch

> Discover how OpenSearch search pipelines and normalization processors enhance relevance scoring and ranking for hybrid searches without application code changes.

- Repository: [AWS Samples/sample-for-amazon-opensearch-service-tutorials-101](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101)
- Tags: deep-dive
- Published: 2026-02-25

---

**Search pipelines in OpenSearch execute configurable processors after query execution to normalize and combine relevance scores from hybrid searches, enabling fine-tuned ranking without modifying application code.**

Amazon OpenSearch Service supports **search pipelines** as a mechanism to post-process query results before they return to the client. In the `aws-samples/sample-for-amazon-opensearch-service-tutorials-101` repository, these pipelines leverage **normalization processors** to rescale and merge scores from hybrid keyword-and-vector searches. This article explains how search pipelines and normalization processors function based on the actual implementation in the sample project.

## Understanding Search Pipelines in OpenSearch

A **search pipeline** is a series of processors that OpenSearch runs after executing the main query but before returning results to the client. Unlike ingest pipelines—which transform documents during indexing—search pipelines operate on the search response itself.

In the sample project, the pipeline serves a specific purpose: it normalizes the disparate score ranges produced by boolean keyword queries and k-NN vector similarity searches, then combines them using configurable weights. This decouples relevance tuning from query construction, allowing operators to adjust ranking behavior without redeploying application code.

## How Normalization Processors Work

The **normalization processor** is a specialized search pipeline component that rescales raw relevance scores to a common scale before combining them. The sample project implements this in [`opensearch_index.py`](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101/blob/main/opensearch_index.py) within the `search_nlp()` function.

### Min-Max Normalization Technique

The processor uses the `min_max` normalization technique, which linearly rescales each sub-query's scores to the [0, 1] range. This ensures that a boolean query returning scores in the hundreds and a k-NN query returning cosine similarities between 0 and 1 contribute equally to the final ranking, preventing either algorithm from dominating due to raw score magnitude differences.

### Score Combination Methods

After normalization, the processor combines scores using the `arithmetic_mean` technique. The sample project configures this with specific weights:

- **70% weight** for the keyword (boolean) query results
- **30% weight** for the k-NN vector query results

This weighting scheme prioritizes lexical matches while still incorporating semantic vector similarity.

## Implementing Search Pipelines in the Sample Project

The repository demonstrates a complete implementation spanning pipeline creation and query-time attachment.

### Creating the Pipeline Definition

In [`artifacts/index_lambda/opensearch_index.py`](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101/blob/main/artifacts/index_lambda/opensearch_index.py), the `search_nlp()` function constructs the pipeline definition as a Python dictionary:

```python
post_processor_search_pipleline = {
    "description": "Post processor for hybrid search",
    "phase_results_processors": [{
        "normalization-processor": {
            "normalization": {"technique": "min_max"},
            "combination": {
                "technique": "arithmetic_mean",
                "parameters": {"weights": [0.7, 0.3]}
            }
        }
    }]
}

```

The function then persists this configuration to OpenSearch via a PUT request to the `/_search/pipeline/<pipeline_name>` endpoint:

```python
response = requests.put(
    f"https://{ENDPOINT}/_search/pipeline/{SEARCH_PIPELINE_NAME}",
    auth=awsauth,
    json=post_processor_search_pipleline,
    headers={'Content-Type': 'application/json'},
    verify=False
)

```

### Attaching Pipelines to Queries

When executing searches in [`artifacts/search_lambda/opensearch_search.py`](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101/blob/main/artifacts/search_lambda/opensearch_search.py), the application references the pipeline by including it in the search body:

```python
search_body = {
    "size": 100,
    # query definition...

    "search_pipeline": SEARCH_PIPELINE_NAME
}

```

OpenSearch automatically applies the configured normalization processor to the query results before returning them to the Lambda function.

## End-to-End Hybrid Search Flow

The sample project implements the following complete workflow:

1. **Pipeline Creation**: The `search_nlp()` function creates the search pipeline during the vector-indexing phase, defining the normalization and combination parameters.

2. **Document Indexing**: Products are indexed with both keyword fields (category, color, description) and vector embeddings (generated via Amazon Bedrock).

3. **Hybrid Query Execution**: The search Lambda constructs a `hybrid` query containing both a `bool` clause for keyword matching and a `knn` clause for vector similarity.

4. **Score Normalization**: OpenSearch executes both sub-queries, then the **normalization-processor** rescales the scores using `min_max` normalization.

5. **Score Combination**: The processor combines the normalized scores using `arithmetic_mean` with the configured weights (70% keyword, 30% vector).

6. **Result Return**: The final merged scores determine the hit ranking, which OpenSearch returns to the client.

This architecture enables operators to tune search relevance by adjusting the `weights` parameter or changing the `technique` values without modifying the application code that constructs the queries.

## Summary

- **Search pipelines** in OpenSearch execute after query logic but before returning results, enabling post-processing of relevance scores.
- **Normalization processors** rescale disparate score ranges (such as boolean query scores versus k-NN similarities) to a common [0, 1] scale using techniques like `min_max`.
- The **combination** phase merges normalized scores using methods like `arithmetic_mean` with configurable weights, allowing fine-tuned control over hybrid search ranking.
- In the sample project, the pipeline is defined in [`opensearch_index.py`](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101/blob/main/opensearch_index.py) by the `search_nlp()` function and attached to queries in [`opensearch_search.py`](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101/blob/main/opensearch_search.py) via the `search_pipeline` parameter.

## Frequently Asked Questions

### What is the difference between an ingest pipeline and a search pipeline in OpenSearch?

An **ingest pipeline** processes documents during indexing, transforming or enriching data before it is stored in the index. A **search pipeline** processes query results after execution but before they are returned to the client, enabling operations like score normalization and result reranking. The sample project uses a search pipeline specifically to normalize and combine hybrid search scores without modifying the indexed documents.

### When should I use min_max normalization versus other techniques?

Use **min_max** normalization when you need to rescale scores to a fixed [0, 1] range while preserving the relative ranking within each sub-query. This is ideal when combining boolean queries (which may produce unbounded scores) with k-NN searches (which typically return cosine similarities between 0 and 1). Other techniques like `mean` normalization might be preferable if you need to center scores around a statistical mean rather than bounding them to a specific range.

### How do I update search pipeline weights without redeploying my application?

Update the pipeline configuration by sending a new `PUT` request to the `/_search/pipeline/<pipeline_name>` endpoint with modified `weights` parameters. For example, change the `weights` array in the JSON payload from `[0.7, 0.3]` to `[0.5, 0.5]` and re-run the curl or Python request. OpenSearch applies the new configuration immediately to all subsequent queries that reference that pipeline name, requiring no changes to the application code in [`opensearch_search.py`](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101/blob/main/opensearch_search.py).

### Can I use search pipelines with non-hybrid queries?

Yes, search pipelines work with any query type, not just hybrid searches. While the sample project demonstrates normalization processors for combining boolean and k-NN scores, you can attach pipelines to standard keyword queries, geospatial searches, or aggregations. The processors will operate on the single result set, though normalization and combination logic provides the most value when merging disparate score types from multiple query clauses.