# How to Set Up VikingFS with Custom Storage Backends in OpenViking

> Discover how to set up VikingFS with custom storage backends in OpenViking. Configure StorageConfig with AGFS and VectorDB for enhanced file operations and semantic search.

- Repository: [Volcengine/OpenViking](https://github.com/volcengine/OpenViking)
- Tags: how-to-guide
- Published: 2026-03-08

---

**Configure VikingFS by defining a `StorageConfig` that specifies your AGFS client for file operations and a VectorDB adapter for semantic search, then initialize the singleton via `init_viking_fs()` with these custom components.**

VikingFS is the unified file-system abstraction layer in the `volcengine/OpenViking` project that orchestrates file operations and semantic vector search. It delegates storage to pluggable backends: an **AGFS** client handles raw file I/O while a **VectorDB** backend manages embedding indices. This guide explains how to configure and extend these storage layers using the configuration system found in `openviking_cli/utils/config/`.

## Understanding the VikingFS Storage Architecture

VikingFS ([`openviking/storage/viking_fs.py`](https://github.com/volcengine/OpenViking/blob/main/openviking/storage/viking_fs.py)) acts as a singleton facade that forwards basic file operations to an underlying AGFS implementation and synchronizes semantic-search vectors with a VectorDB backend ([`openviking/storage/viking_vector_index_backend.py`](https://github.com/volcengine/OpenViking/blob/main/openviking/storage/viking_vector_index_backend.py)). When OpenViking starts ([`openviking/service/core.py`](https://github.com/volcengine/OpenViking/blob/main/openviking/service/core.py)), the bootstrap sequence loads a `StorageConfig`, creates the AGFS client via `openviking/utils/agfs_utils.create_agfs_client()`, instantiates the vector backend, and initializes the global VikingFS instance through `init_viking_fs()`.

### The StorageConfig Object

All storage customization flows through **`StorageConfig`** ([`openviking_cli/utils/config/storage_config.py`](https://github.com/volcengine/OpenViking/blob/main/openviking_cli/utils/config/storage_config.py)), which bundles three sub-configurations:

- **`agfs`** (`AGFSConfig`): Defines the low-level file store backend (S3, local, HTTP, or custom client).
- **`vectordb`** (`VectorDBBackendConfig`): Defines the vector index backend for semantic search.
- **`workspace`**: Base directory that overrides deprecated `path` fields in sub-configs, ensuring a single source of truth for local storage.

The `StorageConfig.resolve_paths()` method automatically replaces legacy `agfs.path` or `vectordb.path` values with the `workspace` directory when specified.

## Customizing the AGFS File Storage Backend

The AGFS (Abstract Global File System) layer is configured via **`AGFSConfig`** ([`openviking_cli/utils/config/agfs_config.py`](https://github.com/volcengine/OpenViking/blob/main/openviking_cli/utils/config/agfs_config.py)):

```python
class AGFSConfig(BaseModel):
    mode: str = Field(default="http-client")   # "http-client" | "binding"

    backend: str = Field(default="s3")         # "s3", "local", or custom dotted path

    path: Optional[str] = None                 # deprecated, ignored if workspace set

```

### Using Built-in AGFS Modes

**HTTP client mode** (`mode: http-client`) routes file operations through an HTTP interface, while **binding mode** (`mode: binding`) uses direct Python bindings for local filesystem access. To switch to the binding client with local storage, use this YAML configuration:

```yaml
agfs:
  mode: binding
  backend: local

```

### Implementing a Custom AGFS Client

To integrate a proprietary cloud store or custom protocol, implement a class that exposes the same public API as the standard AGFS client:

1. **Implement required methods**: `read(path, offset=0, size=-1)`, `write(path, data)`, `ls(path)`, `mkdir(path)`, `rm(path)`, `mv(src, dst)`, `stat(path)`, and `grep(path, pattern)`.
2. **Make the class importable** via a dotted Python path, such as `my_pkg.my_agfs.CustomAGFS`.
3. **Reference it in your configuration**:

```yaml
agfs:
  mode: http-client
  backend: my_pkg.my_agfs.CustomAGFS

```

The AGFS manager ([`openviking/agfs_manager.py`](https://github.com/volcengine/OpenViking/blob/main/openviking/agfs_manager.py)) dynamically imports your class using `import_module` when `create_agfs_client()` processes the configuration.

## Customizing the VectorDB Backend for Semantic Search

Vector storage is configured via **`VectorDBBackendConfig`** ([`openviking_cli/utils/config/vectordb_config.py`](https://github.com/volcengine/OpenViking/blob/main/openviking_cli/utils/config/vectordb_config.py)):

```python
class VectorDBBackendConfig(BaseModel):
    backend: str = Field(default="local")   # "local", "http", "volcengine", "vikingdb", or custom class path

    name: Optional[str] = Field(default="context")
    url: Optional[str] = None               # required for "http"

    distance_metric: str = Field(default="cosine")
    dimension: int = Field(default=0)
    custom_params: Dict[str, Any] = Field(default_factory=dict)

```

### Connecting to Remote Vector Services

**HTTP vector service**:

```yaml
vectordb:
  backend: http
  url: http://localhost:5000
  distance_metric: l2
  dimension: 768

```

**Volcengine VikingDB** (managed cloud service):

```yaml
vectordb:
  backend: volcengine
  volcengine:
    ak: <your-access-key>
    sk: <your-secret-key>
    region: cn-beijing

```

### Implementing a Custom VectorDB Adapter

To use a proprietary vector database (e.g., Elasticsearch, Pinecone, or an internal service), implement the **`CollectionAdapter`** interface defined in [`openviking/storage/vectordb_adapters.py`](https://github.com/volcengine/OpenViking/blob/main/openviking/storage/vectordb_adapters.py):

```python
from openviking.storage.vectordb_adapters import CollectionAdapter
from typing import List, Dict, Any, Optional

class MyCustomAdapter(CollectionAdapter):
    def __init__(self, config: VectorDBBackendConfig):
        self.cfg = config
        # Initialize your client using config.custom_params

        
    @property
    def mode(self) -> str:
        return "http"  # or "local", "grpc", etc.

    
    def upsert(self, record: Dict[str, Any]) -> List[str]:
        """Index vectors and return list of IDs."""
        ...
    
    def query(self, query_vector: Optional[List[float]] = None, 
              filter: Optional[Dict] = None, limit: int = 10, **kwargs) -> List[Dict]:
        """Return matching records as list of dicts with 'id', 'score', 'metadata'."""
        ...
    
    def get(self, ids: List[str]) -> List[Dict]: ...
    def delete(self, ids: List[str]) -> None: ...
    def count(self) -> int: ...
    def clear(self) -> None: ...
    def close(self) -> None: ...

```

Register your adapter by setting the dotted class path as the backend value:

```yaml
vectordb:
  backend: my_pkg.my_vector.MyCustomAdapter
  custom_params:
    host: my-vector-host
    api_key: secret123

```

The factory function `create_collection_adapter()` in [`openviking/storage/vectordb_adapters.py`](https://github.com/volcengine/OpenViking/blob/main/openviking/storage/vectordb_adapters.py) dynamically loads your class and passes the `VectorDBBackendConfig` to its constructor when `VikingVectorIndexBackend` initializes.

## Initializing VikingFS with Custom Backends

Once your configuration defines custom backends, initialize the system programmatically:

```python
import yaml
from openviking_cli.utils.config.storage_config import StorageConfig
from openviking.utils.agfs_utils import create_agfs_client
from openviking.storage.viking_fs import init_viking_fs
from openviking.storage.viking_vector_index_backend import VikingVectorIndexBackend
from openviking_cli.utils.rerank import RerankConfig

# 1. Load configuration

with open("ov.conf", "r") as f:
    raw_cfg = yaml.safe_load(f)

# 2. Build storage configuration (workspace resolves deprecated paths)

storage_cfg = StorageConfig(**raw_cfg["storage"])

# 3. Create custom AGFS client

agfs_client = create_agfs_client(storage_cfg.agfs)

# 4. Create vector store (optional, pass None if only file ops needed)

vector_store = VikingVectorIndexBackend(storage_cfg.vectordb)

# 5. Initialize singleton

vfs = init_viking_fs(
    agfs=agfs_client,
    query_embedder=None,          # Provide embedder for semantic search

    rerank_config=RerankConfig(),
    vector_store=vector_store,
    timeout=15,
    enable_recorder=False,
)

# Use the filesystem

await vfs.mkdir("viking://user/myspace/")
await vfs.write("viking://user/myspace/data.txt", b"content")
results = await vfs.find("viking://user/myspace", query="semantic query")

```

## Summary

- **VikingFS** delegates storage to an **AGFS client** (file operations) and a **VectorDB backend** (semantic search), configured through `StorageConfig`.
- **Custom AGFS clients** must implement `read`, `write`, `ls`, `mkdir`, `rm`, `mv`, `stat`, and `grep`, then be referenced via dotted path in `agfs.backend`.
- **Custom VectorDB adapters** must implement the `CollectionAdapter` interface (`upsert`, `query`, `get`, `delete`, `count`, `clear`, `close`, `mode`), then be referenced via dotted path in `vectordb.backend`.
- The **workspace** field in `StorageConfig` overrides deprecated `path` fields, centralizing local storage configuration.
- Initialization follows a strict sequence: load config → create AGFS client → create vector backend → call `init_viking_fs()` to establish the singleton.

## Frequently Asked Questions

### What interface must a custom AGFS client implement?

Your custom AGFS client must implement the public methods found in the standard AGFS client: `read(path, offset=0, size=-1)`, `write(path, data)`, `ls(path)`, `mkdir(path)`, `rm(path)`, `mv(src, dst)`, `stat(path)`, and `grep(path, pattern)`. The constructor must accept a single `AGFSConfig` argument. Once implemented, reference the dotted class path in your YAML configuration under `agfs.backend`.

### How do I switch from S3 to local file storage?

Set `agfs.mode` to `binding` and `agfs.backend` to `local` in your configuration file. This bypasses HTTP networking and uses direct Python bindings to the local filesystem. Ensure you specify a `workspace` directory in `StorageConfig` to define the base path for all file operations.

### Can I use VikingFS for file operations without vector search?

Yes. When calling `init_viking_fs()`, pass `vector_store=None` and `query_embedder=None`. The resulting instance will support all file system operations (`read`, `write`, `ls`, etc.) but will raise errors if you attempt semantic search methods like `find()` or `search()` that require vector indexing.

### Where does OpenViking load the storage configuration from?

By default, OpenViking loads configuration from `~/.openviking/ov.conf`, but you can specify an alternative path via environment variables or command-line arguments. The `StorageConfig` class parses the `[storage]` section (or equivalent YAML key) to build the AGFS and VectorDB sub-configurations during service startup in [`openviking/service/core.py`](https://github.com/volcengine/OpenViking/blob/main/openviking/service/core.py).