How WeKnora Handles Multi-Instance Storage Backends: Architecture and Implementation

WeKnora handles multi-instance storage backends by storing each storage configuration as a distinct StorageBackend object with a UUID, binding knowledge bases to specific instances via the StorageBackendID field, and resolving the correct backend at runtime using the storage:// URI wrapper parsed by the FileServiceResolver.

Tencent's WeKnora is a knowledge-engineering platform that decouples storage configuration from knowledge base definitions. Unlike systems limited to a single storage engine per deployment, WeKnora's multi-instance storage backend architecture allows a single workspace to manage multiple independent object stores—such as MinIO, Amazon S3, and Tencent COS—simultaneously, with per-knowledge-base granularity. According to the Tencent/WeKnora source code, this flexibility is achieved through a combination of database-backed storage definitions, tenant-level defaults, and runtime URI resolution.

The StorageBackend Data Model

WeKnora's storage layer treats each backend as an independent database entity. The StorageBackend struct defined in [internal/types/storagebackend.go](https://github.com/Tencent/WeKnora/blob/main/internal/types/storagebackend.go) contains the fields necessary to isolate instances:

• ID: A UUID generated automatically on BeforeCreate that uniquely identifies the backend.
• TenantID: The workspace (tenant) that owns this backend.
• Provider: The underlying implementation (local, minio, s3, cos).
• Config: Provider-specific connection details including endpoints, credentials, and bucket names.
• Source: Indicates creation method ("user" for UI-created, "env" for process-wide defaults).
• Status: Soft-deletion control ("active" or "disabled").

Additionally, the Tenant struct in [internal/types/tenant.go](https://github.com/Tencent/WeKnora/blob/main/internal/types/tenant.go) declares DefaultStorageBackendID. This field establishes which backend to use when a knowledge base does not specify its own, creating a fallback chain from explicit binding to tenant default to environment variable.

Binding Knowledge Bases to Storage Instances

The binding mechanism centers on an optional field in the KnowledgeBase struct located in [internal/types/knowledgebase.go](https://github.com/Tencent/WeKnora/blob/main/internal/types/knowledgebase.go):

StorageBackendID *string `json:"storage_backend_id,omitempty"`

When StorageBackendID is set, the knowledge base is explicitly bound to that specific storage instance. If the field is nil, WeKnora falls back to the tenant's DefaultStorageBackendID, and ultimately to the process-wide default defined by the STORAGE_TYPE environment variable.

The helper method SharesStorageBackendWith in the same file compares two knowledge bases for backend compatibility:

func (kb *KnowledgeBase) SharesStorageBackendWith(other *KnowledgeBase, defaultBackendID, defaultProvider string) bool

This method first checks for explicit StorageBackendID matches on both sides. Only when neither knowledge base has an explicit ID does it compare the effective storage provider, ensuring accurate conflict detection during import and export operations.

Runtime Resolution with the storage:// URI

To address a specific backend instance at runtime, WeKnora uses a URI wrapper scheme. File references stored in the database use standard provider schemes (minio://, s3://), but to target a specific backend instance, the system wraps them as follows:

storage://<backend-id>/<provider>://<path>

Utility functions in [internal/types/knowledgebase.go](https://github.com/Tencent/WeKnora/blob/main/internal/types/knowledgebase.go) handle this conversion:

func BuildStorageBackendPath(backendID, providerPath string) string {
    return "storage://" + strings.TrimSpace(backendID) + "/" + providerPath
}

func ParseStorageBackendPath(path string) (backendID, providerPath string, ok bool)

When the server needs to retrieve a file, the FileServiceResolver in [internal/storageurl/resolver.go](https://github.com/Tencent/WeKnora/blob/main/internal/storageurl/resolver.go) performs the following steps:

1. Detect resource:// references and route to the tenant-wide default.
2. Parse the storage:// wrapper using ParseStorageBackendPath to extract the backendID.
3. Determine the provider by inspecting the inner URI scheme via ParseProviderScheme, falling back to StorageEngineConfig.DefaultProvider if absent.
4. Cache the resolved client per request using a map keyed by backendID:provider to avoid SDK client recreation.
5. Resolve the concrete backend by loading the StorageBackend row from the database and constructing a FileService via filesvc.NewFileServiceFromStorageConfig.
6. Fallback to a local file service if resolution fails or the provider is local.

Implementing Multi-Instance Storage in Practice

Creating a Storage Backend Programmatically

To create a new backend instance in code, use the StorageBackend type and repository pattern:

package main

import (
    "context"
    "github.com/Tencent/WeKnora/internal/types"
    "github.com/Tencent/WeKnora/internal/types/interfaces"
)

func createS3Backend(ctx context.Context, repo interfaces.StorageBackendRepository) error {
    backend := &types.StorageBackend{
        TenantID: 12345,
        Name:     "S3 Production",
        Provider: "s3",
        Config: types.StorageBackendConfig{
            Endpoint:        "https://s3.amazonaws.com",
            AccessKeyID:     "AKIAIOSFODNN7EXAMPLE",
            SecretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
            BucketName:      "prod-knowledge-base",
            UseSSL:          true,
        },
    }
    
    if err := backend.Validate(); err != nil {
        return err
    }
    
    return repo.Create(ctx, backend)
}

The Validate() method enforces provider-specific constraints, while BeforeCreate automatically generates the UUID that serves as the backendID for URI construction.

Generating Storage-Wrapped File References

When storing file metadata that must reference a specific backend, use the utility functions to build the wrapper:

import "github.com/Tencent/WeKnora/internal/types"

func makeFileReference(backendID, bucketPath string) string {
    // bucketPath format: "provider://bucket/key"
    return types.BuildStorageBackendPath(backendID, bucketPath)
}

// Example usage:
ref := makeFileReference("c3f5e1d2-9a4b-4d1a-a5f3-e7b9c2d5f6a7", "minio://documents/report.pdf")
// Result: storage://c3f5e1d2-9a4b-4d1a-a5f3-e7b9c2d5f6a7/minio://documents/report.pdf

Resolving File Services at Runtime

To stream or presign URLs for files stored across different backends, instantiate the resolver:

import (
    "github.com/Tencent/WeKnora/internal/storageurl"
    "github.com/Tencent/WeKnora/internal/types"
)

func getFileService(tenant *types.Tenant, storagePath string) (interfaces.FileService, error) {
    resolver := storageurl.NewFileServiceResolver(tenant, file.DefaultFileService)
    return resolver.ResolveFileService(storagePath)
}

// Usage:
svc, err := getFileService(tenant, "storage://abc123/cos://bucket/image.png")
if err != nil {
    // handle resolution failure
}
url, err := svc.Presign("cos://bucket/image.png") // Returns presigned URL

The resolver handles credential retrieval, client initialization, and caching automatically based on the backendID embedded in the path.

Security and Legacy Compatibility

Multi-instance storage requires careful credential management. In [internal/types/storagebackend.go](https://github.com/Tencent/WeKnora/blob/main/internal/types/storagebackend.go), the StorageBackendConfig encrypts sensitive fields like SecretAccessKey using AES encryption via utils.GetAESKey. When the API returns a backend configuration to clients, the NewStorageBackendResponse function masks secrets with a redacted placeholder to prevent credential leakage.

For existing deployments using the legacy single-storage configuration, WeKnora provides migration helpers. The StorageBackendFromLegacy function converts a StorageEngineConfig into a concrete StorageBackend row, while StorageBackendFromEnvironment creates backend definitions derived from environment variables, ensuring backward compatibility during upgrades.

Summary

• Database-backed instances: Each storage backend is a distinct row with a UUID, enabling unlimited instances per tenant.
• Explicit binding: Knowledge bases link to backends via StorageBackendID, with automatic fallback to tenant defaults.
• URI-based resolution: The storage://<id>/<provider>:// wrapper allows per-file backend selection parsed by FileServiceResolver.
• Runtime efficiency: Per-request caching of SDK clients prevents connection overhead when accessing multiple backends.
• Secure by default: Credentials are encrypted at rest and redacted in API responses.

Frequently Asked Questions

Can a single knowledge base use multiple storage backends simultaneously?

No, a knowledge base binds to exactly one storage backend via its StorageBackendID field. However, different knowledge bases within the same workspace can each bind to different backends, and file references within a knowledge base can technically point to other backends via explicit storage:// URIs, though this bypasses the knowledge base's default binding.

How does WeKnora handle storage backend credentials securely?

According to the source code in internal/types/storagebackend.go, secrets within StorageBackendConfig are encrypted using AES with a key retrieved from utils.GetAESKey. When the backend configuration is serialized for API responses, the NewStorageBackendResponse utility replaces sensitive values with a RedactedSecretPlaceholder, ensuring credentials never travel over the wire in plaintext.

What happens if a knowledge base doesn't specify a storage backend?

If StorageBackendID is nil, WeKnora checks the tenant's DefaultStorageBackendID field. If that is also unset, the system falls back to the process-wide default derived from the STORAGE_TYPE environment variable. This three-tier fallback ensures that legacy configurations continue to function while allowing gradual adoption of explicit multi-instance bindings.

Is it possible to migrate from a legacy single-storage setup to multi-instance?

Yes, WeKnora includes migration utilities in internal/types/storagebackend.go. The StorageBackendFromLegacy function converts a legacy StorageEngineConfig into a concrete StorageBackend database row, preserving existing connections. Additionally, StorageBackendFromEnvironment creates backend instances from environment variables for deployments that prefer infrastructure-as-code over UI management.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}