OpenSearch Index Mapping for Hybrid Search: BM25 + Vector Implementation

The production-agentic-rag-course repository defines a strict OpenSearch index mapping named arxiv-papers-chunks that fuses BM25 lexical search with dense vector similarity using HNSW algorithm and Reciprocal Rank Fusion (RRF) for unified scoring.

This OpenSearch index mapping for hybrid search powers the arXiv papers retrieval system in the jamwithai/production-agentic-rag-course repository. The configuration in src/services/opensearch/index_config_hybrid.py implements a production-ready schema that combines traditional inverted-index search with approximate nearest neighbor (ANN) vector search, enabling semantic + lexical retrieval over scientific paper chunks.

Hybrid Index Architecture

The index uses a strict schema enforcement strategy to prevent accidental field injection, ensuring production stability. This architecture supports two parallel search paths that are later fused into a single relevance-ranked result set.

Index Settings and Configuration

According to src/services/opensearch/index_config_hybrid.py (lines 11-22), the index settings optimize for development while enabling k-NN capabilities:

• 1 shard, 0 replicas – Configured for single-node development environments to minimize overhead
• index.knn enabled – Activates OpenSearch's native vector search engine with cosine similarity as the default space type
• Custom analyzers – Two analyzers handle text processing:
  • standard_analyzer: Tokenizes with standard tokenizer plus stopword filtering
  • text_analyzer: Extends standard analysis with lowercase normalization and Snowball stemming for aggressive term reduction

These settings ensure that both keyword matching and vector similarity operate on optimally processed text.

Strict Mapping Enforcement

The mapping declares "dynamic": "strict" (line 24), which rejects any documents containing fields not explicitly defined in the schema. This safeguard prevents schema drift in production pipelines and ensures that the hybrid search logic always operates against known field types.

Field Definitions for Multi-Modal Search

The mapping defines three distinct field categories to support hybrid retrieval: identifier keywords, analyzed text, and dense vectors.

Keyword and Text Fields

The schema includes keyword fields for exact-match identifiers like chunk_id, arxiv_id, and paper_id. Text fields use the custom analyzers defined in settings:

• chunk_text, title, and abstract use text_analyzer (standard + lowercase + stop + snowball)
• authors uses standard_analyzer for less aggressive tokenization
• Each text field exposes a .keyword sub-field for exact-match filtering without re-analyzing

This dual-field strategy supports both full-text relevance scoring and precise filtering in hybrid queries.

Dense Vector Configuration

The vector field embedding (lines 38-44) stores 1024-dimensional float vectors generated by the Jina v3 embedding model. The configuration uses:

• Type: knn_vector with dimension: 1024
• Method: HNSW (Hierarchical Navigable Small World) for approximate nearest neighbor search
• Similarity: Cosine similarity for semantic comparison
• Tuned parameters: ef_construction and m values optimized for recall-versus-latency trade-offs

This field enables semantic similarity search that complements the lexical BM25 scores from text fields.

Search Pipeline Configuration

The repository provides post-processing pipelines that normalize and fuse results from the parallel BM25 and vector searches.

Reciprocal Rank Fusion (RRF)

The HYBRID_RRF_PIPELINE (lines 72-84) implements the default fusion strategy. RRF combines result rankings from lexical and semantic queries without requiring manual weight tuning, making it robust across different query types. The pipeline applies the fusion formula automatically after OpenSearch executes both query paths.

Alternative Weighted Pipeline

An alternative HYBRID_SEARCH_PIPELINE is provided (commented in the source) for weighted average scoring. This allows explicit control over the alpha parameter when blending BM25 and vector scores, useful for domain-specific relevance tuning.

Implementation Examples

Creating the Hybrid Index

To instantiate the index with all mappings and attach the RRF pipeline:

from src.services.opensearch.factory import make_opensearch_client
from src.services.opensearch.index_config_hybrid import (
    ARXIV_PAPERS_CHUNKS_INDEX,
    ARXIV_PAPERS_CHUNKS_MAPPING,
    HYBRID_RRF_PIPELINE,
)

client = make_opensearch_client()

if not client.indices.exists(index=ARXIV_PAPERS_CHUNKS_INDEX):
    client.indices.create(
        index=ARXIV_PAPERS_CHUNKS_INDEX,
        body=ARXIV_PAPERS_CHUNKS_MAPPING,
    )
    client.put_pipeline(id=HYBRID_RRF_PIPELINE["id"], body=HYBRID_RRF_PIPELINE)

This creates the arxiv-papers-chunks index with strict schema enforcement and enables the fusion pipeline defined in index_config_hybrid.py (lines 7-10, 72-84).

Indexing Documents with Vector Embeddings

Documents must include a 1024-dimensional embedding that matches the vector field definition:

from src.services.embeddings.jina_client import JinaEmbeddingClient
from src.services.opensearch.factory import make_opensearch_client
from src.services.opensearch.index_config_hybrid import ARXIV_PAPERS_CHUNKS_INDEX

embedder = JinaEmbeddingClient()
client = make_opensearch_client()

def index_chunk(chunk):
    embedding = embedder.embed(chunk["chunk_text"])
    
    doc = {
        "chunk_id": chunk["chunk_id"],
        "arxiv_id": chunk["arxiv_id"],
        "paper_id": chunk["paper_id"],
        "chunk_text": chunk["chunk_text"],
        "embedding": embedding,
        "title": chunk["title"],
        "abstract": chunk["abstract"],
        "embedding_model": "jina_v3",
    }
    
    client.index(index=ARXIV_PAPERS_CHUNKS_INDEX, body=doc, id=chunk["chunk_id"])

The embedding list must contain exactly 1024 float values to satisfy the knn_vector dimension constraint defined at lines 38-44.

Executing Hybrid Queries

Use the QueryBuilder class from src/services.opensearch.query_builder.py to construct searches that leverage both BM25 and vector paths:

from src.services.opensearch.query_builder import QueryBuilder
from src.services.opensearch.factory import make_opensearch_client
from src.services.opensearch.index_config_hybrid import (
    ARXIV_PAPERS_CHUNKS_INDEX,
    HYBRID_RRF_PIPELINE,
)

client = make_opensearch_client()

def hybrid_search(query_text, categories=None, size=10):
    qb = QueryBuilder(
        query=query_text,
        size=size,
        fields=["chunk_text^3", "title^2", "abstract"],
        categories=categories,
        search_chunks=True,
    )
    body = qb.build()
    body["pipeline"] = HYBRID_RRF_PIPELINE["id"]
    
    return client.search(index=ARXIV_PAPERS_CHUNKS_INDEX, body=body)

The search_chunks=True parameter targets the chunk-level fields defined in the mapping, while the pipeline reference triggers the RRF fusion of lexical and semantic scores.

Summary

• The strict mapping in src/services/opensearch/index_config_hybrid.py prevents production schema drift by rejecting undefined fields
• 1024-dimensional vectors use HNSW with cosine similarity, optimized for the Jina v3 embedding model
• Reciprocal Rank Fusion automatically merges BM25 and vector search results without manual weight tuning
• Dual analyzers provide both aggressive stemming (Snowball) for content matching and standard tokenization for author names

Frequently Asked Questions

What vector dimension does the hybrid index use?

The embedding field stores 1024-dimensional dense vectors. This dimensionality matches the output of the Jina v3 embedding model used throughout the repository, as specified in index_config_hybrid.py (lines 38-44).

How does the strict mapping protect the hybrid search pipeline?

The "dynamic": "strict" setting (line 24) rejects any document containing fields not explicitly defined in the mapping. This prevents accidental schema pollution that could break hybrid queries expecting specific field types for BM25 scoring or vector search.

What is the purpose of the Reciprocal Rank Fusion pipeline?

The HYBRID_RRF_PIPELINE fuses result rankings from parallel BM25 and k-NN vector searches using the RRF formula (1/k + rank). This approach eliminates the need to manually tune alpha weights for blending lexical and semantic scores, providing robust performance across diverse query types.

Which analyzers process text fields in the hybrid index?

The mapping configures two analyzers: text_analyzer applies standard tokenization, lowercase conversion, stopword removal, and Snowball stemming to content fields (chunk_text, title, abstract), while standard_analyzer uses standard tokenization with stopwords only for the authors field to preserve name integrity.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}