How Pathway External Vector Index Integrations Work: USearch and Qdrant Explained

Pathway external vector index integrations rely on a Rust trait abstraction layer that exposes factory methods for both in-memory USearch indexes and remote Qdrant collections, allowing the Python API to perform unified vector search operations across different backends.

Pathway provides a flexible plug-in architecture for integrating external vector search engines directly into streaming data pipelines. The Pathway external vector index system abstracts underlying implementation details through a factory pattern, enabling developers to switch between high-performance in-memory indexes and distributed remote services without modifying their pipeline logic.

The Architecture Behind Pathway External Vector Index Integrations

The Core Rust Trait and Factory Pattern

At the heart of the integration lies the ExternalIndexFactory trait implemented in Rust. This trait defines the interface for creating concrete index instances that handle add, remove, and search operations. The Python API accesses these factories through the PyExternalIndexFactory wrapper located in src/python_api/external_index_wrappers.rs, which exposes static methods for constructing specific index types.

Available Factory Implementations

Pathway currently provides two primary factory implementations:

• USearchKNNIndexFactory (src/external_integration/usearch_integration.rs): Creates in-memory HNSW indexes using the usearch crate for high-performance approximate nearest neighbor search.
• QdrantIndexFactory (src/external_integration/qdrant_integration.rs): Connects to remote Qdrant servers, manages collection creation, and handles network communication via a Tokio runtime.

Both factories return an Arc<dyn ExternalIndex> that the Pathway engine uses for vector operations, ensuring consistent behavior regardless of the backend.

Implementing USearch In-Memory Vector Search

Creating the USearch Factory

The USearch integration builds an in-memory HNSW index optimized for speed. To instantiate the factory, Python code calls ExternalIndexFactory.usearch_knn_factory() with parameters for dimensions, reserved space, and metric type:

import pathway as pw
from pathway.engine import ExternalIndexFactory, USearchMetricKind

# Configure the USearch factory for in-memory HNSW indexing

index_factory = ExternalIndexFactory.usearch_knn_factory(
    dimensions=128,
    reserved_space=1000,
    metric=USearchMetricKind.COS,
    connectivity=0,
    expansion_add=0,
    expansion_search=0,
)

Key Mapping and Index Operations

When make_instance is called, USearchKNNIndex::new initializes the HNSW structure and creates a KeyToU64IdMapper to translate Pathway's internal keys into the numeric IDs required by the usearch library. The implementation handles vector additions through index.add_one and removals via index.remove_one, storing raw &[f64] slices directly in the index.

The USearch implementation lives in src/external_integration/usearch_integration.rs, where the USearchKNNIndex struct manages the lifecycle of the underlying HNSW graph.

Connecting to Remote Qdrant Collections

Qdrant Factory Configuration

For distributed deployments, the Qdrant factory establishes a persistent connection to a Qdrant server. The QdrantIndex::new constructor starts a Tokio runtime, initializes the Qdrant client, and ensures the target collection exists before accepting operations:

from pathway.engine import ExternalIndexFactory

# Create a factory pointing to a remote Qdrant instance

qdrant_factory = ExternalIndexFactory.qdrant_factory(
    url="http://qdrant:6334",
    collection_name="demo_vectors",
    vector_size=128,
)

Batch Operations and Async Search

Unlike the synchronous USearch implementation, Qdrant operations utilize batching for efficiency. The add_batch method in src/external_integration/qdrant_integration.rs converts vectors to f32 format and uses the client's upsert_points method, while remove_batch calls delete_points. For search operations, search_one_async constructs query vectors, executes the remote query, and maps the returned numeric IDs back to Pathway keys with similarity scores.

Executing Vector Searches in Pathway Pipelines

The As-Of-Now Query Interface

Both integrations expose the same interface to the Pathway engine through the _external_index_as_of_now helper function. This generic operator—used by higher-level wrappers like USearchKnn.query_as_of_now in python/pathway/stdlib/indexing/nearest_neighbors.py—receives a table of queries, the vector column to index, and the configured factory.

It creates per-worker index instances via factory.make_instance(), streams add and delete operations using the engine's change-detection logic, and finally performs asynchronous KNN searches. The resulting matches populate a special column named _pw_index_reply containing the nearest neighbors and their distances, which can be post-processed using standard Pathway column operations.

# Example usage with pre-computed vectors

result = source_table._external_index_as_of_now(
    queries_table,
    index_column=source_table.vec,
    query_column=queries_table.vec,
    index_factory=qdrant_factory,  # or usearch_factory

    query_responses_limit_column=queries_table.limit,
).select(matches=pw.this._pw_index_reply)

Summary

• Pathway external vector index integrations use a Rust trait abstraction (ExternalIndexFactory) to unify USearch and Qdrant backends behind a consistent Python API.
• USearch provides in-memory HNSW indexing via USearchKNNIndexFactory with direct key-to-ID mapping in src/external_integration/usearch_integration.rs for low-latency operations.
• Qdrant enables remote distributed search through QdrantIndexFactory, utilizing Tokio for async communication and batching for throughput in src/external_integration/qdrant_integration.rs.
• Both implementations plug into the generic _external_index_as_of_now operator, allowing seamless switching between local and remote vector stores without pipeline code changes.

Frequently Asked Questions

How do I choose between USearch and Qdrant for my Pathway pipeline?

Choose USearch when you need low-latency, in-memory approximate nearest neighbor search within a single process, as it stores vectors directly in RAM using the HNSW algorithm with minimal overhead. Select Qdrant when you require persistence, distributed scalability, or multi-tenant vector storage across a network, since it connects to a remote service capable of handling billions of vectors via QdrantIndex::new.

What data types does Pathway expect for vector columns in external indexes?

Pathway's USearch integration accepts &[f64] (double-precision) slices for vector data, while the Qdrant integration automatically casts vectors to f32 (single-precision) before transmission to match Qdrant's float storage format. Both handle the conversion transparently when you pass Python lists or NumPy arrays through the pipeline operators.

Can I switch from USearch to Qdrant without rewriting my Pathway code?

Yes, the factory pattern allows you to swap backends by simply changing the factory object passed to _external_index_as_of_now or higher-level wrappers. The underlying table operations and query logic remain identical because both factories return an Arc<dyn ExternalIndex> from src/python_api/external_index_wrappers.rs that implements the same Rust trait interface, abstracting away the specific storage mechanism.

Where does the key mapping happen when using external vector indexes?

USearch maintains an internal KeyToU64IdMapper in src/external_integration/usearch_integration.rs to translate Pathway keys into the numeric IDs required by the usearch library. Qdrant similarly manages ID mapping within its client wrapper in src/external_integration/qdrant_integration.rs, ensuring that search results returned from the remote server correctly reference the original Pathway table rows through the search_one_async method.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}