# VelesDB Architecture: From Client SDKs to the Persistent Storage Layer

> Explore the VelesDB architecture from client SDKs to its persistent storage layer. Understand its modular, layered stack featuring an Axum HTTP server, HNSW vector engine, and WAL durable storage.

- Repository: [Wiscale/velesdb](https://github.com/cyberlife-coder/velesdb)
- Tags: architecture
- Published: 2026-02-28

---

**VelesDB implements a modular, layered stack separating client SDKs, an Axum HTTP server, a core HNSW vector engine, and a memory-mapped storage backend with WAL durability.**

The VelesDB architecture is engineered as a high-performance, modular vector database stack developed in the `cyberlife-coder/velesdb` repository. It isolates client interaction, HTTP service, core indexing, and persistent storage into distinct layers, enabling seamless deployment across Python, Rust, WebAssembly, and CLI environments while maintaining ACID durability through memory-mapped files and write-ahead logging.

## Client SDK Layer: Multi-Language Support

VelesDB exposes a unified interface through native SDKs that ultimately communicate via HTTP or direct Rust core embedding. Each SDK abstracts the underlying REST calls into ergonomic language-specific APIs.

- **Python SDK**: Exposes a `Collection` object via PyO3 bindings. Entry point: `velesdb` package. Source: [`examples/python/fusion_strategies.py`](https://github.com/cyberlife-coder/velesdb/blob/main/examples/python/fusion_strategies.py).
- **Rust SDK**: Direct embedding via `velesdb_core::Database`. Entry point: `Database::create_collection`. Source: [`crates/velesdb-core/src/lib.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/lib.rs).
- **WebAssembly SDK**: Browser-compatible bindings using `wasm-bindgen`. Key functions: `store_insert` and `store_get`. Source: [`crates/velesdb-wasm/src/store_insert.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-wasm/src/store_insert.rs).
- **CLI/Tauri**: Native binary in `velesdb-cli` crate. Entry point: [`main.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/main.rs). Source: [`crates/velesdb-cli/src/main.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-cli/src/main.rs).

All SDKs serialize requests to JSON and transmit them to the Axum server, or call the core directly when running in embedded mode.

## Server Layer: Axum REST API Facade

The server layer acts as a thin HTTP façade built on the Axum framework. It handles request validation, CORS, and Swagger UI documentation before delegating to the core engine.

The server initializes in [`crates/velesdb-server/src/main.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-server/src/main.rs), setting up the router and shared application state:

```rust
let app = Router::new()
    .merge(api_router)
    .merge(graph_router)
    .layer(CorsLayer::permissive())
    .with_state(app_state);

```

Route handlers in `crates/velesdb-server/src/handlers/*.rs` parse JSON payloads and forward them to the core `Database`. For example, the upsert endpoint:

```rust
async fn upsert_points(
    State(state): State<AppState>,
    Path(name): Path<String>,
    Json(payload): Json<UpsertPayload>,
) -> Result<Json<UpsertResult>, ApiError> {
    let collection = state.db.get_collection(&name)
        .ok_or_else(|| ApiError::NotFound)?;
    collection.upsert(payload.points)?;
    Ok(Json(UpsertResult::ok()))
}

```

The server maintains the core engine in an `Arc<AppState>` for thread-safe access across concurrent connections.

## Core Engine: Database, Collections, and HNSW Index

The core engine manages the logical database structure, dividing responsibilities between the `Database` façade, individual `Collection` instances, and the HNSW vector index.

### Database Façade

The `Database` struct in [`crates/velesdb-core/src/database.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/database.rs) serves as the primary entry point, maintaining a `RwLock<HashMap<String, Collection>>` for thread-safe collection registry:

```rust
pub fn create_collection(&self, name: &str, dim: usize, metric: DistanceMetric) -> Result<()> {
    self.create_collection_with_options(name, dim, metric, StorageMode::default())
}

```

### Collection and Vector Index

Each `Collection` (source: [`crates/velesdb-core/src/collection.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/collection.rs)) encapsulates:
- An **HNSW index** (`HnswIndex`) for approximate nearest neighbor search
- A **payload store** (`ColumnStore`) for metadata

The HNSW implementation in [`crates/velesdb-core/src/index/hnsw/index/mod.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/index/hnsw/index/mod.rs) provides SIMD-accelerated ANN search:

```rust
pub struct HnswIndex {
    graph: ShardedGraph,
    // … configurable via HnswParams (max connections, efConstruction)
}

```

The index stores only vector IDs; raw embeddings remain in the storage layer to minimize memory pressure.

## Storage Layer: Persistent Vector and Metadata Store

The storage layer persists data durably while optimizing for zero-copy reads and write-ahead logging (WAL) safety.

### MmapStorage for Vectors

`MmapStorage` in [`crates/velesdb-core/src/storage/mmap.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/storage/mmap.rs) manages the `vectors.dat` file using `memmap2::MmapMut`:

```rust
pub fn retrieve_ref(&self, id: u64) -> io::Result<Option<VectorSliceGuard<'_>>> {
    let Some(offset) = self.index.get(id) else { return Ok(None) };
    let mmap = self.mmap.read();
    let ptr = unsafe { mmap.as_ptr().add(offset).cast::<f32>() };
    Ok(Some(VectorSliceGuard { ptr, len: self.dimension, ... }))
}

```

Key characteristics:
- **Zero-copy reads**: Returns `VectorSliceGuard` pointing directly into mmap memory
- **Aggressive pre-allocation**: `reserve_capacity` minimizes resizing
- **Latency metrics**: Tracks resizing overhead for P0 audits

### LogPayloadStorage for Metadata

Metadata and JSON payloads append to `payload.log` via `LogPayloadStorage` (source: [`crates/velesdb-core/src/storage/log_payload.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/storage/log_payload.rs)). This log-structured format supports range reads and bulk iteration without B-tree overhead.

### Durability and Maintenance

- **ShardedIndex**: Maps `ID → offset` across multiple shards to reduce lock contention
- **WAL**: A `BufWriter<File>` in [`mmap.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/mmap.rs) records every write before committing to the mmap, enabling crash recovery
- **Compaction**: [`crates/velesdb-core/src/storage/compaction.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/storage/compaction.rs) rewrites live vectors to new files, atomically renaming them (copy-on-write) and truncating the WAL
- **Metrics**: [`crates/velesdb-core/src/storage/metrics.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/storage/metrics.rs) exposes Prometheus-compatible histograms for fragmentation and latency

## Graph Service Preview

An in-memory graph service (not yet persisted to disk) enables knowledge-graph RAG patterns. Defined in the server’s `graph_router` (source: [`crates/velesdb-server/src/main.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-server/src/main.rs)), it provides:

```rust
let graph_router = Router::new()
    .route("/collections/{name}/graph/edges", get(get_edges).post(add_edge))
    .route("/collections/{name}/graph/traverse", post(traverse_graph))
    .with_state(graph_service);

```

Endpoints support edge addition, BFS/DFS traversal, and degree queries for combining vector similarity with graph expansion.

## End-to-End Data Flow Example

The following Python demonstration traces a complete request through the VelesDB architecture:

```python
from velesdb import Database, FusionStrategy

# 1. Client SDK connects via HTTP

db = Database("http://localhost:8080")
col = db.get_collection("docs")

# 2. Upsert flows: Python → HTTP POST → Axum handler → Collection::upsert → MmapStorage + WAL

col.upsert([
    {"id": 1,
     "vector": [0.1]*768,
     "payload": {"title": "Intro to Veles"}}
])

# 3. Search flows: HTTP POST → HnswIndex::search → MmapStorage::retrieve_ref

results = col.search(vector=[0.2]*768, top_k=5)

# 4. Graph expansion: In-memory GraphService processes BFS

graph = db.get_graph_store("knowledge")
neighbors = graph.traverse_bfs(source=results[0].id, max_depth=2, limit=10)

```

**Flow breakdown**:
1. **Step 1**: HTTP `GET /collections/docs` → `Database::get_collection`
2. **Step 2**: HTTP `POST /collections/docs/points` → `Collection::upsert` → `MmapStorage::ensure_capacity` + WAL write
3. **Step 3**: HTTP `POST /collections/docs/search` → `HnswIndex::search` → `retrieve_ref` zero-copy read
4. **Step 4**: Graph service traverses in-memory edges without storage layer I/O

## Summary

- **VelesDB architecture** separates concerns into four distinct layers: Client SDKs, Axum Server, Core Engine, and Storage Backend.
- **Client SDKs** in Python, Rust, WASM, and CLI communicate via HTTP REST or direct core embedding.
- **Axum server** ([`crates/velesdb-server/src/main.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-server/src/main.rs)) routes requests to the core `Database` facade.
- **Core Engine** manages collections with **HNSW indices** ([`crates/velesdb-core/src/index/hnsw/index/mod.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/index/hnsw/index/mod.rs)) for ANN search and column stores for metadata.
- **Storage Layer** persists vectors in **memory-mapped files** (`MmapStorage`) with zero-copy reads, metadata in **append-only logs** (`LogPayloadStorage`), and guarantees durability via **WAL** and **ShardedIndex**.
- **Compaction** ([`crates/velesdb-core/src/storage/compaction.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/storage/compaction.rs)) reclaims space through copy-on-write file rewriting.

## Frequently Asked Questions

### How does VelesDB ensure data durability during crashes?

VelesDB implements a **Write-Ahead Log (WAL)** in the storage layer. Before any vector bytes commit to the memory-mapped file, the system appends the operation to a `BufWriter<File>` WAL on disk. On restart, the engine replays the WAL to restore the in-memory `ShardedIndex` and ensure consistency between the ID-to-offset mappings and the underlying `vectors.dat` file.

### What is the difference between MmapStorage and LogPayloadStorage?

**MmapStorage** ([`crates/velesdb-core/src/storage/mmap.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/storage/mmap.rs)) handles raw `f32` vector embeddings using `memmap2::MmapMut` for zero-copy access and aggressive pre-allocation. **LogPayloadStorage** ([`crates/velesdb-core/src/storage/log_payload.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/storage/log_payload.rs)) manages JSON metadata in an append-only log format optimized for write throughput and range scans. This separation allows the system to optimize vector reads via memory mapping while treating metadata as opaque, variable-length records.

### Can VelesDB run entirely without the HTTP server?

Yes. The Rust SDK and WebAssembly bindings can embed the **core engine** directly by instantiating `velesdb_core::Database` (source: [`crates/velesdb-core/src/lib.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/lib.rs)). This mode bypasses the Axum server layer entirely, calling `Database::create_collection` and `Collection::upsert` directly from native code or the browser via WASM, eliminating network latency for edge deployments.

### How does the HNSW index interact with the storage layer?

The **HNSW index** ([`crates/velesdb-core/src/index/hnsw/index/mod.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/index/hnsw/index/mod.rs)) maintains an in-memory graph structure containing only vector IDs, not the actual embedding bytes. During a search operation, the index identifies candidate IDs and calls `MmapStorage::retrieve_ref` to fetch the corresponding vectors via zero-copy memory mapping. This design keeps the index memory footprint small while leveraging the storage layer for actual vector data retrieval.