How to Set Up VikingFS with Custom Storage Backends in OpenViking

Configure VikingFS by defining a StorageConfig that specifies your AGFS client for file operations and a VectorDB adapter for semantic search, then initialize the singleton via init_viking_fs() with these custom components.

VikingFS is the unified file-system abstraction layer in the volcengine/OpenViking project that orchestrates file operations and semantic vector search. It delegates storage to pluggable backends: an AGFS client handles raw file I/O while a VectorDB backend manages embedding indices. This guide explains how to configure and extend these storage layers using the configuration system found in openviking_cli/utils/config/.

Understanding the VikingFS Storage Architecture

VikingFS (openviking/storage/viking_fs.py) acts as a singleton facade that forwards basic file operations to an underlying AGFS implementation and synchronizes semantic-search vectors with a VectorDB backend (openviking/storage/viking_vector_index_backend.py). When OpenViking starts (openviking/service/core.py), the bootstrap sequence loads a StorageConfig, creates the AGFS client via openviking/utils/agfs_utils.create_agfs_client(), instantiates the vector backend, and initializes the global VikingFS instance through init_viking_fs().

The StorageConfig Object

All storage customization flows through StorageConfig (openviking_cli/utils/config/storage_config.py), which bundles three sub-configurations:

• agfs (AGFSConfig): Defines the low-level file store backend (S3, local, HTTP, or custom client).
• vectordb (VectorDBBackendConfig): Defines the vector index backend for semantic search.
• workspace: Base directory that overrides deprecated path fields in sub-configs, ensuring a single source of truth for local storage.

The StorageConfig.resolve_paths() method automatically replaces legacy agfs.path or vectordb.path values with the workspace directory when specified.

Customizing the AGFS File Storage Backend

The AGFS (Abstract Global File System) layer is configured via AGFSConfig (openviking_cli/utils/config/agfs_config.py):

class AGFSConfig(BaseModel):
    mode: str = Field(default="http-client")   # "http-client" | "binding"

    backend: str = Field(default="s3")         # "s3", "local", or custom dotted path

    path: Optional[str] = None                 # deprecated, ignored if workspace set

Using Built-in AGFS Modes

HTTP client mode (mode: http-client) routes file operations through an HTTP interface, while binding mode (mode: binding) uses direct Python bindings for local filesystem access. To switch to the binding client with local storage, use this YAML configuration:

agfs:
  mode: binding
  backend: local

Implementing a Custom AGFS Client

To integrate a proprietary cloud store or custom protocol, implement a class that exposes the same public API as the standard AGFS client:

1. Implement required methods: read(path, offset=0, size=-1), write(path, data), ls(path), mkdir(path), rm(path), mv(src, dst), stat(path), and grep(path, pattern).
2. Make the class importable via a dotted Python path, such as my_pkg.my_agfs.CustomAGFS.
3. Reference it in your configuration:

agfs:
  mode: http-client
  backend: my_pkg.my_agfs.CustomAGFS

The AGFS manager (openviking/agfs_manager.py) dynamically imports your class using import_module when create_agfs_client() processes the configuration.

Customizing the VectorDB Backend for Semantic Search

Vector storage is configured via VectorDBBackendConfig (openviking_cli/utils/config/vectordb_config.py):

class VectorDBBackendConfig(BaseModel):
    backend: str = Field(default="local")   # "local", "http", "volcengine", "vikingdb", or custom class path

    name: Optional[str] = Field(default="context")
    url: Optional[str] = None               # required for "http"

    distance_metric: str = Field(default="cosine")
    dimension: int = Field(default=0)
    custom_params: Dict[str, Any] = Field(default_factory=dict)

Connecting to Remote Vector Services

HTTP vector service:

vectordb:
  backend: http
  url: http://localhost:5000
  distance_metric: l2
  dimension: 768

Volcengine VikingDB (managed cloud service):

vectordb:
  backend: volcengine
  volcengine:
    ak: <your-access-key>
    sk: <your-secret-key>
    region: cn-beijing

Implementing a Custom VectorDB Adapter

To use a proprietary vector database (e.g., Elasticsearch, Pinecone, or an internal service), implement the CollectionAdapter interface defined in openviking/storage/vectordb_adapters.py:

from openviking.storage.vectordb_adapters import CollectionAdapter
from typing import List, Dict, Any, Optional

class MyCustomAdapter(CollectionAdapter):
    def __init__(self, config: VectorDBBackendConfig):
        self.cfg = config
        # Initialize your client using config.custom_params

        
    @property
    def mode(self) -> str:
        return "http"  # or "local", "grpc", etc.

    
    def upsert(self, record: Dict[str, Any]) -> List[str]:
        """Index vectors and return list of IDs."""
        ...
    
    def query(self, query_vector: Optional[List[float]] = None, 
              filter: Optional[Dict] = None, limit: int = 10, **kwargs) -> List[Dict]:
        """Return matching records as list of dicts with 'id', 'score', 'metadata'."""
        ...
    
    def get(self, ids: List[str]) -> List[Dict]: ...
    def delete(self, ids: List[str]) -> None: ...
    def count(self) -> int: ...
    def clear(self) -> None: ...
    def close(self) -> None: ...

Register your adapter by setting the dotted class path as the backend value:

vectordb:
  backend: my_pkg.my_vector.MyCustomAdapter
  custom_params:
    host: my-vector-host
    api_key: secret123

The factory function create_collection_adapter() in openviking/storage/vectordb_adapters.py dynamically loads your class and passes the VectorDBBackendConfig to its constructor when VikingVectorIndexBackend initializes.

Initializing VikingFS with Custom Backends

Once your configuration defines custom backends, initialize the system programmatically:

import yaml
from openviking_cli.utils.config.storage_config import StorageConfig
from openviking.utils.agfs_utils import create_agfs_client
from openviking.storage.viking_fs import init_viking_fs
from openviking.storage.viking_vector_index_backend import VikingVectorIndexBackend
from openviking_cli.utils.rerank import RerankConfig

# 1. Load configuration

with open("ov.conf", "r") as f:
    raw_cfg = yaml.safe_load(f)

# 2. Build storage configuration (workspace resolves deprecated paths)

storage_cfg = StorageConfig(**raw_cfg["storage"])

# 3. Create custom AGFS client

agfs_client = create_agfs_client(storage_cfg.agfs)

# 4. Create vector store (optional, pass None if only file ops needed)

vector_store = VikingVectorIndexBackend(storage_cfg.vectordb)

# 5. Initialize singleton

vfs = init_viking_fs(
    agfs=agfs_client,
    query_embedder=None,          # Provide embedder for semantic search

    rerank_config=RerankConfig(),
    vector_store=vector_store,
    timeout=15,
    enable_recorder=False,
)

# Use the filesystem

await vfs.mkdir("viking://user/myspace/")
await vfs.write("viking://user/myspace/data.txt", b"content")
results = await vfs.find("viking://user/myspace", query="semantic query")

Summary

• VikingFS delegates storage to an AGFS client (file operations) and a VectorDB backend (semantic search), configured through StorageConfig.
• Custom AGFS clients must implement read, write, ls, mkdir, rm, mv, stat, and grep, then be referenced via dotted path in agfs.backend.
• Custom VectorDB adapters must implement the CollectionAdapter interface (upsert, query, get, delete, count, clear, close, mode), then be referenced via dotted path in vectordb.backend.
• The workspace field in StorageConfig overrides deprecated path fields, centralizing local storage configuration.
• Initialization follows a strict sequence: load config → create AGFS client → create vector backend → call init_viking_fs() to establish the singleton.

Frequently Asked Questions

What interface must a custom AGFS client implement?

Your custom AGFS client must implement the public methods found in the standard AGFS client: read(path, offset=0, size=-1), write(path, data), ls(path), mkdir(path), rm(path), mv(src, dst), stat(path), and grep(path, pattern). The constructor must accept a single AGFSConfig argument. Once implemented, reference the dotted class path in your YAML configuration under agfs.backend.

How do I switch from S3 to local file storage?

Set agfs.mode to binding and agfs.backend to local in your configuration file. This bypasses HTTP networking and uses direct Python bindings to the local filesystem. Ensure you specify a workspace directory in StorageConfig to define the base path for all file operations.

Can I use VikingFS for file operations without vector search?

Yes. When calling init_viking_fs(), pass vector_store=None and query_embedder=None. The resulting instance will support all file system operations (read, write, ls, etc.) but will raise errors if you attempt semantic search methods like find() or search() that require vector indexing.

Where does OpenViking load the storage configuration from?

By default, OpenViking loads configuration from ~/.openviking/ov.conf, but you can specify an alternative path via environment variables or command-line arguments. The StorageConfig class parses the [storage] section (or equivalent YAML key) to build the AGFS and VectorDB sub-configurations during service startup in openviking/service/core.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}