How to Configure Hybrid Search Combining Vector Embeddings with Tantivy Full-Text Search

To configure hybrid search combining vector embeddings with Tantivy full-text search, instantiate a HybridIndexFactory in your Pathway YAML configuration that orchestrates USearchKnnFactory for dense vector retrieval and TantivyBM25Factory for BM25 keyword matching, then assign this factory to your DocumentStore's retriever_factory parameter.

The pathwaycom/llm-app template provides a declarative way to configure hybrid search combining vector embeddings with Tantivy full-text search without writing custom Python code. By leveraging Pathway's indexing factories, you can simultaneously query dense embeddings for semantic similarity and BM25 indexes for precise keyword matching within the same retrieval pipeline.

Understanding the Hybrid Search Architecture

The hybrid retrieval system in Pathway operates through two independent index factories that work in parallel. When a query arrives, the system automatically routes the query text through both an embedding model for vector search and the Tantivy engine for BM25 scoring, then merges the results.

Vector Index Component: The USearchKnnFactory (built on the ultra-fast usearch library) stores dense embeddings and handles approximate nearest neighbor search using metrics like cosine similarity.

Full-Text Index Component: The TantivyBM25Factory wraps the Tantivy search engine to provide traditional inverted-index BM25 keyword matching.

Hybrid Orchestration: The HybridIndexFactory coordinates these sub-retrievers, sending queries to both indexes and merging result sets based on configurable scoring or weighting strategies before passing documents downstream to your RAG components.

Step-by-Step YAML Configuration

Pathway's LLM templates use YAML declarations to build the retrieval pipeline. To configure hybrid search combining vector embeddings with Tantivy full-text search, modify your templates/question_answering_rag/app.yaml file as follows.

Configure the Vector Index (USearchKnnFactory)

First, define the vector index that will store embeddings for semantic search:

$knn_index: !pw.indexing.USearchKnnFactory
  reserved_space: 1000
  embedder: $embedder
  metric: !pw.indexing.USearchKnnMetricKind.COS
  dimensions: 1536

The reserved_space parameter pre-allocates capacity for vectors, while dimensions must match your embedding model's output size (1536 for OpenAI's text-embedding-ada-002, for example).

Configure the Full-Text Index (TantivyBM25Factory)

Next, instantiate the BM25 index for keyword search:

$bm25_index: !pw.indexing.TantivyBM25Factory

This factory automatically handles tokenization and inverted index construction using the Tantivy Rust engine; no additional parameters are required for basic functionality.

Combine with HybridIndexFactory

Merge both retrievers using the HybridIndexFactory, which queries sub-indexes in parallel:

$hybrid_index_factory: !pw.indexing.HybridIndexFactory
  retriever_factories:
    - $knn_index
    - $bm25_index

Update the Document Store

Finally, assign the hybrid factory to your document store's retriever_factory:

$document_store: !pw.xpacks.llm.document_store.DocumentStore
  docs: $sources
  parser: $parser
  splitter: $splitter
  retriever_factory: $hybrid_index_factory

Running the Application

With the YAML configuration in place, the application requires no code changes. Execute the pipeline from the template directory:

cd templates/question_answering_rag
pip install -r requirements.txt
python app.py

The app.py file loads the configuration via pw.load_yaml and automatically provisions the underlying Rust engines (usearch and Tantivy). The REST API will now return results that blend semantic similarity with keyword relevance when you query endpoints like POST /v1/retrieve.

Advanced Configuration: Tuning Retrieval Weights

For finer control over result ranking, adjust the influence of each sub-retriever using weight_factors:

$hybrid_index_factory: !pw.indexing.HybridIndexFactory
  retriever_factories:
    - $knn_index
    - $bm25_index
  weight_factors: [0.7, 0.3]

This configuration assigns 70% weight to semantic (vector) matches and 30% to BM25 keyword matches. Refer to the Pathway API documentation for additional options such as k values for top-k retrieval from each index.

Key Source Files

• templates/question_answering_rag/app.yaml: Primary configuration file where you define the HybridIndexFactory and its sub-retrievers.
• templates/question_answering_rag/app.py: Entry point that loads the YAML configuration; no modifications required for hybrid search.
• templates/question_answering_rag/README.md: Contains the Hybrid Indexing section with conceptual explanations.
• templates/question_answering_rag/requirements.txt: Specifies dependencies including Pathway and its indexing extensions.

Summary

• Pathway enables hybrid search combining vector embeddings with Tantivy full-text search through declarative YAML configurations using HybridIndexFactory.
• The architecture combines USearchKnnFactory (dense vector search via usearch) and TantivyBM25Factory (BM25 keyword search via Tantivy).
• Configuration requires only updating templates/question_answering_rag/app.yaml to define both indexes and assign the hybrid factory to the DocumentStore.
• No Python code changes are necessary; the application automatically provisions Rust-based indexing engines when loading the configuration.
• Optional weight_factors parameter allows fine-tuning the balance between semantic and keyword relevance.

Frequently Asked Questions

Do I need to install separate dependencies for Tantivy and usearch?

No. The requirements.txt in templates/question_answering_rag already includes Pathway with its indexing extensions. When you run pip install -r requirements.txt, both the usearch and Tantivy Rust engines are installed automatically. Pathway provisions these engines transparently when you reference USearchKnnFactory or TantivyBM25Factory in your YAML configuration.

How does the hybrid index handle query routing?

The HybridIndexFactory automatically duplicates incoming queries: it sends the raw text to the TantivyBM25Factory for keyword matching and embeds the text (using your configured embedder) for the USearchKnnFactory. It then merges the two result sets according to your configuration before returning the final ranked documents to the RAG pipeline.

Can I use different embedding dimensions or metrics?

Yes. The USearchKnnFactory accepts any dimension size that matches your embedding model—common values include 384, 768, or 1536. You can also change the metric parameter to !pw.indexing.USearchKnnMetricKind.IP for inner product or L2 for Euclidean distance, depending on your specific embedding model's requirements.

What happens if Tantivy or usearch indexes are empty?

If either sub-index contains no documents, the HybridIndexFactory gracefully handles the empty result set by relying solely on the populated index. The hybrid retriever merges available results, so your application continues functioning even if one indexing backend has no matches for a particular query.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}