Vector Store Driver Abstraction in WeKnora: How Tencent Built Interchangeable Backends for Milvus and Qdrant

WeKnora implements a factory-registry pattern that abstracts vector storage behind a unified VectorStore model and RetrieveEngineService interface, allowing seamless swapping between Milvus, Qdrant, and other engines without modifying business logic.

The Tencent/WeKnora project solves the challenge of vendor lock-in in AI retrieval systems through a robust vector store driver abstraction. By decoupling the storage engine from application logic, the platform enables operators to migrate between vector databases or run multiple engines in parallel using identical codepaths.

Core Components of the Abstraction

The architecture rests on three tightly-coupled concepts: a generic persistence model, a typed enumeration of supported engines, and a factory function that bridges configuration to concrete clients.

The VectorStore Model (Engine-Agnostic Payload)

At the heart of the system lies the VectorStore struct defined in internal/types/vectorstore.go. This model persists all connection and indexing metadata in a single database row regardless of the underlying engine.

type VectorStore struct {
    ID               string                `json:"id"`           // UUID or virtual "__env_..." ID
    TenantID         uint64                `json:"tenant_id"`    // Multi-tenant scoping
    Name             string                `json:"name"`         // Human-readable identifier
    EngineType       RetrieverEngineType   `json:"engine_type"`  // e.g., MilvusRetrieverEngineType
    ConnectionConfig ConnectionConfig      `json:"connection_config"` // Driver-specific connection params
    IndexConfig      IndexConfig           `json:"index_config"` // Collection/index settings
}

The EngineType field acts as the dispatch key, while ConnectionConfig and IndexConfig store driver-specific parameters in a generic container. This design ensures that all backends share the same database schema, with the EngineType field alone determining which driver implementation gets instantiated.

RetrieverEngineType Enum (Engine Dispatch)

The supported backends are enumerated in internal/types/retriever.go as typed constants:

type RetrieverEngineType string

const (
    MilvusRetrieverEngineType          RetrieverEngineType = "milvus"
    QdrantRetrieverEngineType          RetrieverEngineType = "qdrant"
    ElasticsearchRetrieverEngineType   RetrieverEngineType = "elasticsearch"
    // Additional engines...
)

These values drive the factory switch in internal/container/engine_factory.go that creates the concrete driver at runtime.

Factory Pattern for Driver Instantiation

The EngineFactory function type converts a VectorStore metadata object into a working retrieval service. Defined in internal/types/interfaces/vectorstore.go, the factory signature is:

func(context.Context, types.VectorStore) (interfaces.RetrieveEngineService, error)

EngineFactory Implementation

The concrete factory in internal/container/engine_factory.go uses a type-switch to instantiate the correct driver. The createEngineServiceFromStore function (lines 66-78) implements this dispatch logic:

switch store.EngineType {
case types.MilvusRetrieverEngineType:
    return createMilvusEngine(ctx, store)
case types.QdrantRetrieverEngineType:
    return createQdrantEngine(store)
case types.ElasticsearchRetrieverEngineType:
    return createOpenSearchEngine(store)
default:
    return nil, fmt.Errorf("unsupported engine type: %s", store.EngineType)
}

Driver-Specific Builders

Each backend implements a dedicated builder function that translates the generic ConnectionConfig into SDK-specific client objects:

• Milvus: createMilvusEngine builds a milvusclient.Client using buildMilvusClientConfig (lines 51-59 in engine_factory.go)
• Qdrant: createQdrantEngine constructs a qdrant.Client with host, port, TLS, and API key parameters (lines 30-48)
• OpenSearch: createOpenSearchEngine initializes an OpenSearch client and injects the audit sink (lines 32-38)

Each builder returns a RetrieveEngineService implementation, ensuring that downstream code receives a uniform interface regardless of the underlying vector store technology.

Runtime Decoupling via StoreRegistry

To eliminate the need for consumers to know which driver backs a specific store, WeKnora implements a StoreRegistry interface in internal/types/interfaces/vectorstore.go:

type StoreRegistry interface {
    RegisterWithStoreID(storeID string, svc RetrieveEngineService)
    GetByStoreID(storeID string) (RetrieveEngineService, error)
    UnregisterByStoreID(storeID string)
}

When VectorStoreService creates a new store, it invokes the factory and immediately registers the resulting service:

svc, err := engineFactory(ctx, *store)
if err == nil {
    storeRegistry.RegisterWithStoreID(store.ID, svc)
}

Downstream components—such as knowledge-base search handlers—resolve engines purely by store ID:

// In a search handler
engine, err := storeRegistry.GetByStoreID(kb.StoreID)
if err != nil {
    return nil, err
}
results, err := engine.Retrieve(ctx, query, nil)

This registry pattern ensures that business logic remains completely isolated from driver-specific details, enabling true backend interchangeability.

Environment-Based Virtual Stores

For deployments using the RETRIEVE_DRIVER environment variable (e.g., milvus or qdrant), WeKnora supports virtual stores. The BuildEnvVectorStores function in internal/types/vectorstore.go (lines 98-112) creates synthetic VectorStore objects with IDs formatted as __env_milvus__ or __env_qdrant__.

These virtual stores feed through the identical factory and registry pipeline, guaranteeing that environment-configured stores and database-configured stores share identical driver codepaths without duplication.

Summary

• Unified Model: The VectorStore struct in internal/types/vectorstore.go persists all engine configurations in a single, driver-agnostic schema.
• Typed Dispatch: The RetrieverEngineType enum drives a factory switch that instantiates Milvus, Qdrant, or other clients based on the EngineType field.
• Factory Pattern: EngineFactory in internal/container/engine_factory.go encapsulates driver-specific client construction behind a common function signature.
• Runtime Registry: The StoreRegistry interface maps store IDs to active RetrieveEngineService instances, letting application code resolve backends without knowing the underlying engine.
• Zero-Code Swapping: Adding a new vector database requires only a new enum value, a builder function, and a case in the factory switch—no changes to search logic or handlers.

Frequently Asked Questions

How does WeKnora decide which vector store driver to use at runtime?

WeKnora consults the EngineType field of the VectorStore model persisted in the database. The EngineFactory performs a type-switch on this field to invoke the appropriate builder function (e.g., createMilvusEngine or createQdrantEngine), returning a concrete implementation of the RetrieveEngineService interface.

What interface must new vector store backends implement?

New backends must provide a builder function that returns a RetrieveEngineService interface, defined in internal/types/interfaces/vectorstore.go. This service must implement retrieval operations such as Retrieve, while the factory handles client initialization and configuration translation from the generic ConnectionConfig structure.

Can WeKnora support multiple vector store engines simultaneously?

Yes. The StoreRegistry maintains a runtime mapping of store IDs to engine instances, allowing different tenants or knowledge bases to use different backends concurrently. Each store row carries its own EngineType, and the factory creates the appropriate driver for each independently.

Where is the vector store configuration persisted?

Configuration resides in the vector_stores database table, modeled by the VectorStore struct in internal/types/vectorstore.go. This includes the EngineType, ConnectionConfig (host, port, credentials), and IndexConfig (collection names, dimensions), enabling complete restoration of the driver state across application restarts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}