# OpenSearch Index Mapping for Hybrid Search: BM25 + Vector Implementation

> Discover the OpenSearch index mapping for hybrid search. Learn how to combine BM25 lexical search with vector similarity using HNSW and RRF for unified scoring in your applications.

- Repository: [jamwithai/production-agentic-rag-course](https://github.com/jamwithai/production-agentic-rag-course)
- Tags: how-to-guide
- Published: 2026-03-23

---

**The production-agentic-rag-course repository defines a strict OpenSearch index mapping named `arxiv-papers-chunks` that fuses BM25 lexical search with dense vector similarity using HNSW algorithm and Reciprocal Rank Fusion (RRF) for unified scoring.**

This **OpenSearch index mapping for hybrid search** powers the arXiv papers retrieval system in the `jamwithai/production-agentic-rag-course` repository. The configuration in [`src/services/opensearch/index_config_hybrid.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/opensearch/index_config_hybrid.py) implements a production-ready schema that combines traditional inverted-index search with approximate nearest neighbor (ANN) vector search, enabling semantic + lexical retrieval over scientific paper chunks.

## Hybrid Index Architecture

The index uses a strict schema enforcement strategy to prevent accidental field injection, ensuring production stability. This architecture supports two parallel search paths that are later fused into a single relevance-ranked result set.

### Index Settings and Configuration

According to [`src/services/opensearch/index_config_hybrid.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/opensearch/index_config_hybrid.py) (lines 11-22), the index settings optimize for development while enabling k-NN capabilities:

- **1 shard, 0 replicas** – Configured for single-node development environments to minimize overhead
- **index.knn enabled** – Activates OpenSearch's native vector search engine with `cosine` similarity as the default space type
- **Custom analyzers** – Two analyzers handle text processing:
  - `standard_analyzer`: Tokenizes with standard tokenizer plus stopword filtering
  - `text_analyzer`: Extends standard analysis with lowercase normalization and Snowball stemming for aggressive term reduction

These settings ensure that both keyword matching and vector similarity operate on optimally processed text.

### Strict Mapping Enforcement

The mapping declares `"dynamic": "strict"` (line 24), which rejects any documents containing fields not explicitly defined in the schema. This safeguard prevents schema drift in production pipelines and ensures that the hybrid search logic always operates against known field types.

## Field Definitions for Multi-Modal Search

The mapping defines three distinct field categories to support hybrid retrieval: identifier keywords, analyzed text, and dense vectors.

### Keyword and Text Fields

The schema includes **keyword fields** for exact-match identifiers like `chunk_id`, `arxiv_id`, and `paper_id`. **Text fields** use the custom analyzers defined in settings:

- `chunk_text`, `title`, and `abstract` use `text_analyzer` (standard + lowercase + stop + snowball)
- `authors` uses `standard_analyzer` for less aggressive tokenization
- Each text field exposes a `.keyword` sub-field for exact-match filtering without re-analyzing

This dual-field strategy supports both full-text relevance scoring and precise filtering in hybrid queries.

### Dense Vector Configuration

The **vector field** `embedding` (lines 38-44) stores 1024-dimensional float vectors generated by the Jina v3 embedding model. The configuration uses:

- **Type**: `knn_vector` with `dimension: 1024`
- **Method**: HNSW (Hierarchical Navigable Small World) for approximate nearest neighbor search
- **Similarity**: Cosine similarity for semantic comparison
- **Tuned parameters**: `ef_construction` and `m` values optimized for recall-versus-latency trade-offs

This field enables semantic similarity search that complements the lexical BM25 scores from text fields.

## Search Pipeline Configuration

The repository provides post-processing pipelines that normalize and fuse results from the parallel BM25 and vector searches.

### Reciprocal Rank Fusion (RRF)

The `HYBRID_RRF_PIPELINE` (lines 72-84) implements the default fusion strategy. RRF combines result rankings from lexical and semantic queries without requiring manual weight tuning, making it robust across different query types. The pipeline applies the fusion formula automatically after OpenSearch executes both query paths.

### Alternative Weighted Pipeline

An alternative `HYBRID_SEARCH_PIPELINE` is provided (commented in the source) for weighted average scoring. This allows explicit control over the alpha parameter when blending BM25 and vector scores, useful for domain-specific relevance tuning.

## Implementation Examples

### Creating the Hybrid Index

To instantiate the index with all mappings and attach the RRF pipeline:

```python
from src.services.opensearch.factory import make_opensearch_client
from src.services.opensearch.index_config_hybrid import (
    ARXIV_PAPERS_CHUNKS_INDEX,
    ARXIV_PAPERS_CHUNKS_MAPPING,
    HYBRID_RRF_PIPELINE,
)

client = make_opensearch_client()

if not client.indices.exists(index=ARXIV_PAPERS_CHUNKS_INDEX):
    client.indices.create(
        index=ARXIV_PAPERS_CHUNKS_INDEX,
        body=ARXIV_PAPERS_CHUNKS_MAPPING,
    )
    client.put_pipeline(id=HYBRID_RRF_PIPELINE["id"], body=HYBRID_RRF_PIPELINE)

```

This creates the `arxiv-papers-chunks` index with strict schema enforcement and enables the fusion pipeline defined in [`index_config_hybrid.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/index_config_hybrid.py) (lines 7-10, 72-84).

### Indexing Documents with Vector Embeddings

Documents must include a 1024-dimensional embedding that matches the vector field definition:

```python
from src.services.embeddings.jina_client import JinaEmbeddingClient
from src.services.opensearch.factory import make_opensearch_client
from src.services.opensearch.index_config_hybrid import ARXIV_PAPERS_CHUNKS_INDEX

embedder = JinaEmbeddingClient()
client = make_opensearch_client()

def index_chunk(chunk):
    embedding = embedder.embed(chunk["chunk_text"])
    
    doc = {
        "chunk_id": chunk["chunk_id"],
        "arxiv_id": chunk["arxiv_id"],
        "paper_id": chunk["paper_id"],
        "chunk_text": chunk["chunk_text"],
        "embedding": embedding,
        "title": chunk["title"],
        "abstract": chunk["abstract"],
        "embedding_model": "jina_v3",
    }
    
    client.index(index=ARXIV_PAPERS_CHUNKS_INDEX, body=doc, id=chunk["chunk_id"])

```

The `embedding` list must contain exactly 1024 float values to satisfy the `knn_vector` dimension constraint defined at lines 38-44.

### Executing Hybrid Queries

Use the `QueryBuilder` class from [`src/services.opensearch.query_builder.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services.opensearch.query_builder.py) to construct searches that leverage both BM25 and vector paths:

```python
from src.services.opensearch.query_builder import QueryBuilder
from src.services.opensearch.factory import make_opensearch_client
from src.services.opensearch.index_config_hybrid import (
    ARXIV_PAPERS_CHUNKS_INDEX,
    HYBRID_RRF_PIPELINE,
)

client = make_opensearch_client()

def hybrid_search(query_text, categories=None, size=10):
    qb = QueryBuilder(
        query=query_text,
        size=size,
        fields=["chunk_text^3", "title^2", "abstract"],
        categories=categories,
        search_chunks=True,
    )
    body = qb.build()
    body["pipeline"] = HYBRID_RRF_PIPELINE["id"]
    
    return client.search(index=ARXIV_PAPERS_CHUNKS_INDEX, body=body)

```

The `search_chunks=True` parameter targets the chunk-level fields defined in the mapping, while the `pipeline` reference triggers the RRF fusion of lexical and semantic scores.

## Summary

- The **strict mapping** in [`src/services/opensearch/index_config_hybrid.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/opensearch/index_config_hybrid.py) prevents production schema drift by rejecting undefined fields
- **1024-dimensional vectors** use HNSW with cosine similarity, optimized for the Jina v3 embedding model
- **Reciprocal Rank Fusion** automatically merges BM25 and vector search results without manual weight tuning
- **Dual analyzers** provide both aggressive stemming (Snowball) for content matching and standard tokenization for author names

## Frequently Asked Questions

### What vector dimension does the hybrid index use?

The `embedding` field stores **1024-dimensional** dense vectors. This dimensionality matches the output of the Jina v3 embedding model used throughout the repository, as specified in [`index_config_hybrid.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/index_config_hybrid.py) (lines 38-44).

### How does the strict mapping protect the hybrid search pipeline?

The `"dynamic": "strict"` setting (line 24) rejects any document containing fields not explicitly defined in the mapping. This prevents accidental schema pollution that could break hybrid queries expecting specific field types for BM25 scoring or vector search.

### What is the purpose of the Reciprocal Rank Fusion pipeline?

The `HYBRID_RRF_PIPELINE` fuses result rankings from parallel BM25 and k-NN vector searches using the RRF formula (1/k + rank). This approach eliminates the need to manually tune alpha weights for blending lexical and semantic scores, providing robust performance across diverse query types.

### Which analyzers process text fields in the hybrid index?

The mapping configures two analyzers: `text_analyzer` applies standard tokenization, lowercase conversion, stopword removal, and Snowball stemming to content fields (`chunk_text`, `title`, `abstract`), while `standard_analyzer` uses standard tokenization with stopwords only for the `authors` field to preserve name integrity.