VelesDB Architecture: From Client SDKs to the Persistent Storage Layer

VelesDB implements a modular, layered stack separating client SDKs, an Axum HTTP server, a core HNSW vector engine, and a memory-mapped storage backend with WAL durability.

The VelesDB architecture is engineered as a high-performance, modular vector database stack developed in the cyberlife-coder/velesdb repository. It isolates client interaction, HTTP service, core indexing, and persistent storage into distinct layers, enabling seamless deployment across Python, Rust, WebAssembly, and CLI environments while maintaining ACID durability through memory-mapped files and write-ahead logging.

Client SDK Layer: Multi-Language Support

VelesDB exposes a unified interface through native SDKs that ultimately communicate via HTTP or direct Rust core embedding. Each SDK abstracts the underlying REST calls into ergonomic language-specific APIs.

• Python SDK: Exposes a Collection object via PyO3 bindings. Entry point: velesdb package. Source: examples/python/fusion_strategies.py.
• Rust SDK: Direct embedding via velesdb_core::Database. Entry point: Database::create_collection. Source: crates/velesdb-core/src/lib.rs.
• WebAssembly SDK: Browser-compatible bindings using wasm-bindgen. Key functions: store_insert and store_get. Source: crates/velesdb-wasm/src/store_insert.rs.
• CLI/Tauri: Native binary in velesdb-cli crate. Entry point: main.rs. Source: crates/velesdb-cli/src/main.rs.

All SDKs serialize requests to JSON and transmit them to the Axum server, or call the core directly when running in embedded mode.

Server Layer: Axum REST API Facade

The server layer acts as a thin HTTP façade built on the Axum framework. It handles request validation, CORS, and Swagger UI documentation before delegating to the core engine.

The server initializes in crates/velesdb-server/src/main.rs, setting up the router and shared application state:

let app = Router::new()
    .merge(api_router)
    .merge(graph_router)
    .layer(CorsLayer::permissive())
    .with_state(app_state);

Route handlers in crates/velesdb-server/src/handlers/*.rs parse JSON payloads and forward them to the core Database. For example, the upsert endpoint:

async fn upsert_points(
    State(state): State<AppState>,
    Path(name): Path<String>,
    Json(payload): Json<UpsertPayload>,
) -> Result<Json<UpsertResult>, ApiError> {
    let collection = state.db.get_collection(&name)
        .ok_or_else(|| ApiError::NotFound)?;
    collection.upsert(payload.points)?;
    Ok(Json(UpsertResult::ok()))
}

The server maintains the core engine in an Arc<AppState> for thread-safe access across concurrent connections.

Core Engine: Database, Collections, and HNSW Index

The core engine manages the logical database structure, dividing responsibilities between the Database façade, individual Collection instances, and the HNSW vector index.

Database Façade

The Database struct in crates/velesdb-core/src/database.rs serves as the primary entry point, maintaining a RwLock<HashMap<String, Collection>> for thread-safe collection registry:

pub fn create_collection(&self, name: &str, dim: usize, metric: DistanceMetric) -> Result<()> {
    self.create_collection_with_options(name, dim, metric, StorageMode::default())
}

Collection and Vector Index

Each Collection (source: crates/velesdb-core/src/collection.rs) encapsulates:

• An HNSW index (HnswIndex) for approximate nearest neighbor search
• A payload store (ColumnStore) for metadata

The HNSW implementation in crates/velesdb-core/src/index/hnsw/index/mod.rs provides SIMD-accelerated ANN search:

pub struct HnswIndex {
    graph: ShardedGraph,
    // … configurable via HnswParams (max connections, efConstruction)
}

The index stores only vector IDs; raw embeddings remain in the storage layer to minimize memory pressure.

Storage Layer: Persistent Vector and Metadata Store

The storage layer persists data durably while optimizing for zero-copy reads and write-ahead logging (WAL) safety.

MmapStorage for Vectors

MmapStorage in crates/velesdb-core/src/storage/mmap.rs manages the vectors.dat file using memmap2::MmapMut:

pub fn retrieve_ref(&self, id: u64) -> io::Result<Option<VectorSliceGuard<'_>>> {
    let Some(offset) = self.index.get(id) else { return Ok(None) };
    let mmap = self.mmap.read();
    let ptr = unsafe { mmap.as_ptr().add(offset).cast::<f32>() };
    Ok(Some(VectorSliceGuard { ptr, len: self.dimension, ... }))
}

Key characteristics:

• Zero-copy reads: Returns VectorSliceGuard pointing directly into mmap memory
• Aggressive pre-allocation: reserve_capacity minimizes resizing
• Latency metrics: Tracks resizing overhead for P0 audits

LogPayloadStorage for Metadata

Metadata and JSON payloads append to payload.log via LogPayloadStorage (source: crates/velesdb-core/src/storage/log_payload.rs). This log-structured format supports range reads and bulk iteration without B-tree overhead.

Durability and Maintenance

• ShardedIndex: Maps ID → offset across multiple shards to reduce lock contention
• WAL: A BufWriter<File> in mmap.rs records every write before committing to the mmap, enabling crash recovery
• Compaction: crates/velesdb-core/src/storage/compaction.rs rewrites live vectors to new files, atomically renaming them (copy-on-write) and truncating the WAL
• Metrics: crates/velesdb-core/src/storage/metrics.rs exposes Prometheus-compatible histograms for fragmentation and latency

Graph Service Preview

An in-memory graph service (not yet persisted to disk) enables knowledge-graph RAG patterns. Defined in the server’s graph_router (source: crates/velesdb-server/src/main.rs), it provides:

let graph_router = Router::new()
    .route("/collections/{name}/graph/edges", get(get_edges).post(add_edge))
    .route("/collections/{name}/graph/traverse", post(traverse_graph))
    .with_state(graph_service);

Endpoints support edge addition, BFS/DFS traversal, and degree queries for combining vector similarity with graph expansion.

End-to-End Data Flow Example

The following Python demonstration traces a complete request through the VelesDB architecture:

from velesdb import Database, FusionStrategy

# 1. Client SDK connects via HTTP

db = Database("http://localhost:8080")
col = db.get_collection("docs")

# 2. Upsert flows: Python → HTTP POST → Axum handler → Collection::upsert → MmapStorage + WAL

col.upsert([
    {"id": 1,
     "vector": [0.1]*768,
     "payload": {"title": "Intro to Veles"}}
])

# 3. Search flows: HTTP POST → HnswIndex::search → MmapStorage::retrieve_ref

results = col.search(vector=[0.2]*768, top_k=5)

# 4. Graph expansion: In-memory GraphService processes BFS

graph = db.get_graph_store("knowledge")
neighbors = graph.traverse_bfs(source=results[0].id, max_depth=2, limit=10)

Flow breakdown:

1. Step 1: HTTP GET /collections/docs → Database::get_collection
2. Step 2: HTTP POST /collections/docs/points → Collection::upsert → MmapStorage::ensure_capacity + WAL write
3. Step 3: HTTP POST /collections/docs/search → HnswIndex::search → retrieve_ref zero-copy read
4. Step 4: Graph service traverses in-memory edges without storage layer I/O

Summary

• VelesDB architecture separates concerns into four distinct layers: Client SDKs, Axum Server, Core Engine, and Storage Backend.
• Client SDKs in Python, Rust, WASM, and CLI communicate via HTTP REST or direct core embedding.
• Axum server (crates/velesdb-server/src/main.rs) routes requests to the core Database facade.
• Core Engine manages collections with HNSW indices (crates/velesdb-core/src/index/hnsw/index/mod.rs) for ANN search and column stores for metadata.
• Storage Layer persists vectors in memory-mapped files (MmapStorage) with zero-copy reads, metadata in append-only logs (LogPayloadStorage), and guarantees durability via WAL and ShardedIndex.
• Compaction (crates/velesdb-core/src/storage/compaction.rs) reclaims space through copy-on-write file rewriting.

Frequently Asked Questions

How does VelesDB ensure data durability during crashes?

VelesDB implements a Write-Ahead Log (WAL) in the storage layer. Before any vector bytes commit to the memory-mapped file, the system appends the operation to a BufWriter<File> WAL on disk. On restart, the engine replays the WAL to restore the in-memory ShardedIndex and ensure consistency between the ID-to-offset mappings and the underlying vectors.dat file.

What is the difference between MmapStorage and LogPayloadStorage?

MmapStorage (crates/velesdb-core/src/storage/mmap.rs) handles raw f32 vector embeddings using memmap2::MmapMut for zero-copy access and aggressive pre-allocation. LogPayloadStorage (crates/velesdb-core/src/storage/log_payload.rs) manages JSON metadata in an append-only log format optimized for write throughput and range scans. This separation allows the system to optimize vector reads via memory mapping while treating metadata as opaque, variable-length records.

Can VelesDB run entirely without the HTTP server?

Yes. The Rust SDK and WebAssembly bindings can embed the core engine directly by instantiating velesdb_core::Database (source: crates/velesdb-core/src/lib.rs). This mode bypasses the Axum server layer entirely, calling Database::create_collection and Collection::upsert directly from native code or the browser via WASM, eliminating network latency for edge deployments.

How does the HNSW index interact with the storage layer?

The HNSW index (crates/velesdb-core/src/index/hnsw/index/mod.rs) maintains an in-memory graph structure containing only vector IDs, not the actual embedding bytes. During a search operation, the index identifies candidate IDs and calls MmapStorage::retrieve_ref to fetch the corresponding vectors via zero-copy memory mapping. This design keeps the index memory footprint small while leveraging the storage layer for actual vector data retrieval.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}