# How Pathway External Vector Index Integrations Work: USearch and Qdrant Explained

> Discover how Pathway integrates USearch and Qdrant for unified vector search. Learn about the Rust trait abstraction powering seamless backend operations via the Python API.

- Repository: [Pathway/pathway](https://github.com/pathwaycom/pathway)
- Tags: internals
- Published: 2026-03-06

---

**Pathway external vector index integrations rely on a Rust trait abstraction layer that exposes factory methods for both in-memory USearch indexes and remote Qdrant collections, allowing the Python API to perform unified vector search operations across different backends.**

Pathway provides a flexible plug-in architecture for integrating external vector search engines directly into streaming data pipelines. The **Pathway external vector index** system abstracts underlying implementation details through a factory pattern, enabling developers to switch between high-performance in-memory indexes and distributed remote services without modifying their pipeline logic.

## The Architecture Behind Pathway External Vector Index Integrations

### The Core Rust Trait and Factory Pattern

At the heart of the integration lies the `ExternalIndexFactory` trait implemented in Rust. This trait defines the interface for creating concrete index instances that handle **add**, **remove**, and **search** operations. The Python API accesses these factories through the `PyExternalIndexFactory` wrapper located in [`src/python_api/external_index_wrappers.rs`](https://github.com/pathwaycom/pathway/blob/main/src/python_api/external_index_wrappers.rs), which exposes static methods for constructing specific index types.

### Available Factory Implementations

Pathway currently provides two primary factory implementations:

- **`USearchKNNIndexFactory`** ([`src/external_integration/usearch_integration.rs`](https://github.com/pathwaycom/pathway/blob/main/src/external_integration/usearch_integration.rs)): Creates in-memory HNSW indexes using the usearch crate for high-performance approximate nearest neighbor search.
- **`QdrantIndexFactory`** ([`src/external_integration/qdrant_integration.rs`](https://github.com/pathwaycom/pathway/blob/main/src/external_integration/qdrant_integration.rs)): Connects to remote Qdrant servers, manages collection creation, and handles network communication via a Tokio runtime.

Both factories return an `Arc<dyn ExternalIndex>` that the Pathway engine uses for vector operations, ensuring consistent behavior regardless of the backend.

## Implementing USearch In-Memory Vector Search

### Creating the USearch Factory

The USearch integration builds an in-memory HNSW index optimized for speed. To instantiate the factory, Python code calls `ExternalIndexFactory.usearch_knn_factory()` with parameters for dimensions, reserved space, and metric type:

```python
import pathway as pw
from pathway.engine import ExternalIndexFactory, USearchMetricKind

# Configure the USearch factory for in-memory HNSW indexing

index_factory = ExternalIndexFactory.usearch_knn_factory(
    dimensions=128,
    reserved_space=1000,
    metric=USearchMetricKind.COS,
    connectivity=0,
    expansion_add=0,
    expansion_search=0,
)

```

### Key Mapping and Index Operations

When `make_instance` is called, `USearchKNNIndex::new` initializes the HNSW structure and creates a `KeyToU64IdMapper` to translate Pathway's internal keys into the numeric IDs required by the usearch library. The implementation handles vector additions through `index.add_one` and removals via `index.remove_one`, storing raw `&[f64]` slices directly in the index.

The USearch implementation lives in [`src/external_integration/usearch_integration.rs`](https://github.com/pathwaycom/pathway/blob/main/src/external_integration/usearch_integration.rs), where the `USearchKNNIndex` struct manages the lifecycle of the underlying HNSW graph.

## Connecting to Remote Qdrant Collections

### Qdrant Factory Configuration

For distributed deployments, the Qdrant factory establishes a persistent connection to a Qdrant server. The `QdrantIndex::new` constructor starts a Tokio runtime, initializes the Qdrant client, and ensures the target collection exists before accepting operations:

```python
from pathway.engine import ExternalIndexFactory

# Create a factory pointing to a remote Qdrant instance

qdrant_factory = ExternalIndexFactory.qdrant_factory(
    url="http://qdrant:6334",
    collection_name="demo_vectors",
    vector_size=128,
)

```

### Batch Operations and Async Search

Unlike the synchronous USearch implementation, Qdrant operations utilize batching for efficiency. The `add_batch` method in [`src/external_integration/qdrant_integration.rs`](https://github.com/pathwaycom/pathway/blob/main/src/external_integration/qdrant_integration.rs) converts vectors to `f32` format and uses the client's `upsert_points` method, while `remove_batch` calls `delete_points`. For search operations, `search_one_async` constructs query vectors, executes the remote query, and maps the returned numeric IDs back to Pathway keys with similarity scores.

## Executing Vector Searches in Pathway Pipelines

### The As-Of-Now Query Interface

Both integrations expose the same interface to the Pathway engine through the `_external_index_as_of_now` helper function. This generic operator—used by higher-level wrappers like `USearchKnn.query_as_of_now` in [`python/pathway/stdlib/indexing/nearest_neighbors.py`](https://github.com/pathwaycom/pathway/blob/main/python/pathway/stdlib/indexing/nearest_neighbors.py)—receives a table of queries, the vector column to index, and the configured factory.

It creates per-worker index instances via `factory.make_instance()`, streams add and delete operations using the engine's change-detection logic, and finally performs asynchronous KNN searches. The resulting matches populate a special column named `_pw_index_reply` containing the nearest neighbors and their distances, which can be post-processed using standard Pathway column operations.

```python

# Example usage with pre-computed vectors

result = source_table._external_index_as_of_now(
    queries_table,
    index_column=source_table.vec,
    query_column=queries_table.vec,
    index_factory=qdrant_factory,  # or usearch_factory

    query_responses_limit_column=queries_table.limit,
).select(matches=pw.this._pw_index_reply)

```

## Summary

- **Pathway external vector index** integrations use a Rust trait abstraction (`ExternalIndexFactory`) to unify USearch and Qdrant backends behind a consistent Python API.
- **USearch** provides in-memory HNSW indexing via `USearchKNNIndexFactory` with direct key-to-ID mapping in [`src/external_integration/usearch_integration.rs`](https://github.com/pathwaycom/pathway/blob/main/src/external_integration/usearch_integration.rs) for low-latency operations.
- **Qdrant** enables remote distributed search through `QdrantIndexFactory`, utilizing Tokio for async communication and batching for throughput in [`src/external_integration/qdrant_integration.rs`](https://github.com/pathwaycom/pathway/blob/main/src/external_integration/qdrant_integration.rs).
- Both implementations plug into the generic `_external_index_as_of_now` operator, allowing seamless switching between local and remote vector stores without pipeline code changes.

## Frequently Asked Questions

### How do I choose between USearch and Qdrant for my Pathway pipeline?

Choose USearch when you need low-latency, in-memory approximate nearest neighbor search within a single process, as it stores vectors directly in RAM using the HNSW algorithm with minimal overhead. Select Qdrant when you require persistence, distributed scalability, or multi-tenant vector storage across a network, since it connects to a remote service capable of handling billions of vectors via `QdrantIndex::new`.

### What data types does Pathway expect for vector columns in external indexes?

Pathway's USearch integration accepts `&[f64]` (double-precision) slices for vector data, while the Qdrant integration automatically casts vectors to `f32` (single-precision) before transmission to match Qdrant's float storage format. Both handle the conversion transparently when you pass Python lists or NumPy arrays through the pipeline operators.

### Can I switch from USearch to Qdrant without rewriting my Pathway code?

Yes, the factory pattern allows you to swap backends by simply changing the factory object passed to `_external_index_as_of_now` or higher-level wrappers. The underlying table operations and query logic remain identical because both factories return an `Arc<dyn ExternalIndex>` from [`src/python_api/external_index_wrappers.rs`](https://github.com/pathwaycom/pathway/blob/main/src/python_api/external_index_wrappers.rs) that implements the same Rust trait interface, abstracting away the specific storage mechanism.

### Where does the key mapping happen when using external vector indexes?

USearch maintains an internal `KeyToU64IdMapper` in [`src/external_integration/usearch_integration.rs`](https://github.com/pathwaycom/pathway/blob/main/src/external_integration/usearch_integration.rs) to translate Pathway keys into the numeric IDs required by the usearch library. Qdrant similarly manages ID mapping within its client wrapper in [`src/external_integration/qdrant_integration.rs`](https://github.com/pathwaycom/pathway/blob/main/src/external_integration/qdrant_integration.rs), ensuring that search results returned from the remote server correctly reference the original Pathway table rows through the `search_one_async` method.