# How VelesDB's Storage Layer Combines Memory-Mapped Files and WAL for High-Performance Vector Storage

> Discover how VelesDB's storage layer uses memory-mapped files and WAL for efficient zero-copy vector access and durable payload storage, boosting performance.

- Repository: [Wiscale/velesdb](https://github.com/cyberlife-coder/velesdb)
- Tags: internals
- Published: 2026-02-28

---

**VelesDB's storage layer implements a dual-layer architecture using memory-mapped files for zero-copy vector access and an append-only Write-Ahead Log with snapshots for durable payload storage.**

VelesDB is an open-source vector database developed by cyberlife-coder that stores high-dimensional embeddings and associated metadata. The storage engine employs a sophisticated dual-layer design that balances raw performance with crash safety. This article examines how VelesDB's storage layer leverages memory-mapped files and WAL mechanisms to achieve low-latency reads while guaranteeing durability.

## Memory-Mapped Vector Storage

The vector storage layer in [`crates/velesdb-core/src/storage/mmap.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/storage/mmap.rs) uses **memory-mapped files** to store raw `f32` vector data, enabling zero-copy reads directly from the operating system's page cache.

### File Layout and Initialization

Vector data resides in a contiguous file (`vectors.dat`) where each vector occupies `dimension * 4` bytes aligned to `f32`. The `MmapStorage` struct initializes this file with aggressive pre-allocation:

```rust
let data_path = path.join("vectors.dat");
let data_file = OpenOptions::new()
    .read(true).write(true).create(true).open(&data_path)?;
if data_file.metadata()?.len() == 0 {
    data_file.set_len(Self::INITIAL_SIZE)?;
}
let mmap = unsafe { MmapMut::map_mut(&data_file)? };

```

A **sharded in-memory index** (`ShardedIndex`) maintained in [`crates/velesdb-core/src/storage/sharded_index.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/storage/sharded_index.rs) maps vector IDs to byte offsets within this file.

### WAL-Protected Writes

Before writing vector bytes to the mmap, VelesDB appends a lightweight entry to `vectors.wal` to ensure durability:

```rust
self.wal.write().write_all(&[1u8])?;          // marker
self.wal.write().write_all(&id.to_le_bytes())?;
self.wal.write().write_all(&vector_bytes)?;
self.wal.write().flush()?;                    // fsync for durability

```

This pattern ensures that even if the process crashes before the mmap pages are flushed to disk, the vector can be recovered from the WAL on restart.

### Zero-Copy Reads with VectorSliceGuard

The `retrieve_ref` method returns a `VectorSliceGuard` (defined in [`crates/velesdb-core/src/storage/guard.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/storage/guard.rs)) that provides direct access to the mapped memory without copying data:

```rust
let mmap = self.mmap.read();
let ptr = unsafe { mmap.as_ptr().add(offset).cast::<f32>() };
Ok(Some(VectorSliceGuard { _guard: mmap, ptr, len: self.dimension, … }))

```

Holding a read lock on the mmap while returning a `&[f32]` slice prevents use-after-free scenarios while eliminating memcpy overhead.

### Dynamic Resizing and Epoch Safety

When capacity is exhausted, `ensure_capacity` (lines 167-208) grows the backing file using exponential growth with a minimum 64 MiB headroom. After resizing:

1. The file descriptor is extended with `ftruncate` or `set_len`
2. A new `MmapMut` is created replacing the old mapping
3. An **epoch counter** (`remap_epoch`) increments atomically

Existing `VectorSliceGuard` instances check this epoch on drop, ensuring that readers holding stale mappings cannot access invalid memory after a remap operation.

### Compaction

Deleted vectors create garbage in `vectors.dat`. The `compact()` method (lines 315-337) rewrites active vectors to a new temporary file, atomically renames it over the original, and reopens the file handle. This operation reclaims disk space without requiring downtime, as the sharded index is updated to reflect new offsets atomically.

## Log-Structured Payload Storage

JSON payloads use a different strategy implemented in [`crates/velesdb-core/src/storage/log_payload.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/storage/log_payload.rs). Variable-length data benefits from **log-structured storage** rather than fixed-size mmap slots.

### Append-Only WAL Format

Payload operations write to `payloads.log` using a binary format:

- **1 byte**: Marker (`1` = store, `2` = delete)
- **8 bytes**: Little-endian ID
- **4 bytes**: Payload length (for stores only)
- **N bytes**: JSON payload bytes

```rust
wal.write_all(&[1u8])?;
wal.write_all(&id.to_le_bytes())?;
wal.write_all(&len_u32.to_le_bytes())?;
wal.write_all(&payload_bytes)?;
wal.flush()?;
self.index.write().insert(id, pos + 9);

```

Each `store` or `delete` operation triggers an explicit `flush()` to guarantee durability before acknowledging the write.

### Snapshot-Based Recovery

To avoid replaying the entire WAL on startup, VelesDB periodically creates binary snapshots (`payloads.snapshot`) containing:

- Magic bytes `VSNP`
- Version byte
- WAL position at snapshot time
- Entry count
- Serialized `(id, offset)` pairs
- CRC32 checksum

During cold start, `LogPayloadStorage` loads the index from the snapshot and replays only WAL entries written after the snapshot's recorded position, achieving O(1) recovery time regardless of log size.

### Durability Guarantees

The implementation ensures consistency through careful ordering:

1. The WAL is flushed before creating a snapshot
2. The snapshot writer flushes the WAL first (`self.wal.write().flush()`) before writing snapshot data
3. The `Drop` implementation for storage structs flushes both WAL and mmap buffers to disk on shutdown

## Practical Implementation Examples

### Initializing Storage Components

```rust
use velesdb_core::storage::{MmapStorage, LogPayloadStorage};
use std::path::Path;

let path = Path::new("/tmp/velesdb");
let mut vec_store = MmapStorage::new(path, 768)?;  // 768 dimensions
let mut payload_store = LogPayloadStorage::new(path)?;

```

### Storing Vectors and Payloads

```rust
let id = 42u64;
let vector: Vec<f32> = vec![0.0; 768];
vec_store.store(id, &vector)?;

let payload = serde_json::json!({ "title": "Example", "tags": ["rust","db"] });
payload_store.store(id, &payload)?;

```

### Zero-Copy Vector Retrieval

```rust
if let Some(guard) = vec_store.retrieve_ref(id)? {
    let slice: &[f32] = &*guard;  // No memcpy occurs here
    println!("First component = {}", slice[0]);
}  // Read lock released automatically

```

### Snapshot Management

```rust
if payload_store.should_create_snapshot() {
    payload_store.create_snapshot()?;
}

// Recovery happens automatically in LogPayloadStorage::new()
let recovered_store = LogPayloadStorage::new(path)?;

```

### Storage Compaction

```rust
let reclaimed = vec_store.compact()?;
println!("Reclaimed {} bytes", reclaimed);

```

## Key Source Files

Understanding VelesDB's storage architecture requires familiarity with these specific implementation files:

- **[`crates/velesdb-core/src/storage/mmap.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/storage/mmap.rs)**: Core `MmapStorage` implementation including resizing, epoch tracking, and compaction
- **[`crates/velesdb-core/src/storage/log_payload.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/storage/log_payload.rs)**: Append-only WAL and snapshot logic for JSON payloads
- **[`crates/velesdb-core/src/storage/guard.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/storage/guard.rs)**: `VectorSliceGuard` providing safe zero-copy views with epoch validation
- **[`crates/velesdb-core/src/storage/vector_io.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/storage/vector_io.rs)**: Low-level serialization of `f32` vectors into mmap regions
- **[`crates/velesdb-core/src/storage/compaction.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/storage/compaction.rs)**: Algorithm for rewriting active vectors to reclaim space
- **[`crates/velesdb-core/src/storage/sharded_index.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/storage/sharded_index.rs)**: Concurrency-friendly ID-to-offset mapping using sharded locks
- **[`crates/velesdb-core/src/storage/metrics.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/crates/velesdb-core/src/storage/metrics.rs)**: Performance monitoring for resize operations and WAL latency

## Summary

VelesDB's storage layer achieves high performance through architectural specialization:

- **Memory-mapped vectors** provide zero-copy read access with nanosecond-level latency for similarity search
- **Dual WAL system** protects both vector and payload writes against crashes without synchronous mmap flushing
- **Epoch-based safety** prevents use-after-remap bugs during file growth operations
- **Snapshot recovery** enables O(1) cold starts for payload storage regardless of write history
- **Background compaction** reclaims space from deleted vectors while maintaining data consistency

This design separates concerns between fixed-size numerical data (vectors) and variable-size metadata (payloads), applying the optimal persistence strategy for each workload type.

## Frequently Asked Questions

### How does VelesDB ensure crash safety without synchronous mmap flushing?

VelesDB uses a **Write-Ahead Log (WAL)** for both storage layers. Vector writes append a small record to `vectors.wal` and call `flush()` before updating the mmap. Payload writes append to `payloads.log` with explicit flushing. If a crash occurs, the system replays these logs on restart to restore consistency, treating the mmap as a cache rather than the source of truth.

### What prevents reading invalid memory when the vector file resizes?

The `VectorSliceGuard` in [`guard.rs`](https://github.com/cyberlife-coder/velesdb/blob/main/guard.rs) validates a **remap epoch** counter before dereferencing. When `ensure_capacity` grows the file and creates a new `MmapMut`, it increments this atomic counter. Any existing guards holding the old epoch detect the mismatch on access, preventing undefined behavior from stale pointers.

### Why does VelesDB use different storage strategies for vectors versus payloads?

Vectors are fixed-size (`dimension * 4` bytes), making random access via memory-mapping efficient and enabling zero-copy reads. Payloads are variable-length JSON that would fragment a fixed-size mmap. The **log-structured approach** for payloads optimizes for append-only writes and fast recovery via snapshots, while the **mmap approach** for vectors optimizes for read-heavy similarity search workloads.

### How does snapshot recovery work in the payload storage layer?

`create_snapshot` writes the current in-memory index to `payloads.snapshot` prefixed with the current WAL position. On startup, `LogPayloadStorage::new` loads this index and seeks to the saved position in `payloads.log`, replaying only newer entries. This reduces recovery time from O(n) to O(delta) where delta represents writes since the last snapshot.