How VelesDB's Storage Layer Combines Memory-Mapped Files and WAL for High-Performance Vector Storage

VelesDB's storage layer implements a dual-layer architecture using memory-mapped files for zero-copy vector access and an append-only Write-Ahead Log with snapshots for durable payload storage.

VelesDB is an open-source vector database developed by cyberlife-coder that stores high-dimensional embeddings and associated metadata. The storage engine employs a sophisticated dual-layer design that balances raw performance with crash safety. This article examines how VelesDB's storage layer leverages memory-mapped files and WAL mechanisms to achieve low-latency reads while guaranteeing durability.

Memory-Mapped Vector Storage

The vector storage layer in crates/velesdb-core/src/storage/mmap.rs uses memory-mapped files to store raw f32 vector data, enabling zero-copy reads directly from the operating system's page cache.

File Layout and Initialization

Vector data resides in a contiguous file (vectors.dat) where each vector occupies dimension * 4 bytes aligned to f32. The MmapStorage struct initializes this file with aggressive pre-allocation:

let data_path = path.join("vectors.dat");
let data_file = OpenOptions::new()
    .read(true).write(true).create(true).open(&data_path)?;
if data_file.metadata()?.len() == 0 {
    data_file.set_len(Self::INITIAL_SIZE)?;
}
let mmap = unsafe { MmapMut::map_mut(&data_file)? };

A sharded in-memory index (ShardedIndex) maintained in crates/velesdb-core/src/storage/sharded_index.rs maps vector IDs to byte offsets within this file.

WAL-Protected Writes

Before writing vector bytes to the mmap, VelesDB appends a lightweight entry to vectors.wal to ensure durability:

self.wal.write().write_all(&[1u8])?;          // marker
self.wal.write().write_all(&id.to_le_bytes())?;
self.wal.write().write_all(&vector_bytes)?;
self.wal.write().flush()?;                    // fsync for durability

This pattern ensures that even if the process crashes before the mmap pages are flushed to disk, the vector can be recovered from the WAL on restart.

Zero-Copy Reads with VectorSliceGuard

The retrieve_ref method returns a VectorSliceGuard (defined in crates/velesdb-core/src/storage/guard.rs) that provides direct access to the mapped memory without copying data:

let mmap = self.mmap.read();
let ptr = unsafe { mmap.as_ptr().add(offset).cast::<f32>() };
Ok(Some(VectorSliceGuard { _guard: mmap, ptr, len: self.dimension, … }))

Holding a read lock on the mmap while returning a &[f32] slice prevents use-after-free scenarios while eliminating memcpy overhead.

Dynamic Resizing and Epoch Safety

When capacity is exhausted, ensure_capacity (lines 167-208) grows the backing file using exponential growth with a minimum 64 MiB headroom. After resizing:

1. The file descriptor is extended with ftruncate or set_len
2. A new MmapMut is created replacing the old mapping
3. An epoch counter (remap_epoch) increments atomically

Existing VectorSliceGuard instances check this epoch on drop, ensuring that readers holding stale mappings cannot access invalid memory after a remap operation.

Compaction

Deleted vectors create garbage in vectors.dat. The compact() method (lines 315-337) rewrites active vectors to a new temporary file, atomically renames it over the original, and reopens the file handle. This operation reclaims disk space without requiring downtime, as the sharded index is updated to reflect new offsets atomically.

Log-Structured Payload Storage

JSON payloads use a different strategy implemented in crates/velesdb-core/src/storage/log_payload.rs. Variable-length data benefits from log-structured storage rather than fixed-size mmap slots.

Append-Only WAL Format

Payload operations write to payloads.log using a binary format:

• 1 byte: Marker (1 = store, 2 = delete)
• 8 bytes: Little-endian ID
• 4 bytes: Payload length (for stores only)
• N bytes: JSON payload bytes

wal.write_all(&[1u8])?;
wal.write_all(&id.to_le_bytes())?;
wal.write_all(&len_u32.to_le_bytes())?;
wal.write_all(&payload_bytes)?;
wal.flush()?;
self.index.write().insert(id, pos + 9);

Each store or delete operation triggers an explicit flush() to guarantee durability before acknowledging the write.

Snapshot-Based Recovery

To avoid replaying the entire WAL on startup, VelesDB periodically creates binary snapshots (payloads.snapshot) containing:

• Magic bytes VSNP
• Version byte
• WAL position at snapshot time
• Entry count
• Serialized (id, offset) pairs
• CRC32 checksum

During cold start, LogPayloadStorage loads the index from the snapshot and replays only WAL entries written after the snapshot's recorded position, achieving O(1) recovery time regardless of log size.

Durability Guarantees

The implementation ensures consistency through careful ordering:

1. The WAL is flushed before creating a snapshot
2. The snapshot writer flushes the WAL first (self.wal.write().flush()) before writing snapshot data
3. The Drop implementation for storage structs flushes both WAL and mmap buffers to disk on shutdown

Practical Implementation Examples

Initializing Storage Components

use velesdb_core::storage::{MmapStorage, LogPayloadStorage};
use std::path::Path;

let path = Path::new("/tmp/velesdb");
let mut vec_store = MmapStorage::new(path, 768)?;  // 768 dimensions
let mut payload_store = LogPayloadStorage::new(path)?;

Storing Vectors and Payloads

let id = 42u64;
let vector: Vec<f32> = vec![0.0; 768];
vec_store.store(id, &vector)?;

let payload = serde_json::json!({ "title": "Example", "tags": ["rust","db"] });
payload_store.store(id, &payload)?;

Zero-Copy Vector Retrieval

if let Some(guard) = vec_store.retrieve_ref(id)? {
    let slice: &[f32] = &*guard;  // No memcpy occurs here
    println!("First component = {}", slice[0]);
}  // Read lock released automatically

Snapshot Management

if payload_store.should_create_snapshot() {
    payload_store.create_snapshot()?;
}

// Recovery happens automatically in LogPayloadStorage::new()
let recovered_store = LogPayloadStorage::new(path)?;

Storage Compaction

let reclaimed = vec_store.compact()?;
println!("Reclaimed {} bytes", reclaimed);

Key Source Files

Understanding VelesDB's storage architecture requires familiarity with these specific implementation files:

• crates/velesdb-core/src/storage/mmap.rs: Core MmapStorage implementation including resizing, epoch tracking, and compaction
• crates/velesdb-core/src/storage/log_payload.rs: Append-only WAL and snapshot logic for JSON payloads
• crates/velesdb-core/src/storage/guard.rs: VectorSliceGuard providing safe zero-copy views with epoch validation
• crates/velesdb-core/src/storage/vector_io.rs: Low-level serialization of f32 vectors into mmap regions
• crates/velesdb-core/src/storage/compaction.rs: Algorithm for rewriting active vectors to reclaim space
• crates/velesdb-core/src/storage/sharded_index.rs: Concurrency-friendly ID-to-offset mapping using sharded locks
• crates/velesdb-core/src/storage/metrics.rs: Performance monitoring for resize operations and WAL latency

Summary

VelesDB's storage layer achieves high performance through architectural specialization:

• Memory-mapped vectors provide zero-copy read access with nanosecond-level latency for similarity search
• Dual WAL system protects both vector and payload writes against crashes without synchronous mmap flushing
• Epoch-based safety prevents use-after-remap bugs during file growth operations
• Snapshot recovery enables O(1) cold starts for payload storage regardless of write history
• Background compaction reclaims space from deleted vectors while maintaining data consistency

This design separates concerns between fixed-size numerical data (vectors) and variable-size metadata (payloads), applying the optimal persistence strategy for each workload type.

Frequently Asked Questions

How does VelesDB ensure crash safety without synchronous mmap flushing?

VelesDB uses a Write-Ahead Log (WAL) for both storage layers. Vector writes append a small record to vectors.wal and call flush() before updating the mmap. Payload writes append to payloads.log with explicit flushing. If a crash occurs, the system replays these logs on restart to restore consistency, treating the mmap as a cache rather than the source of truth.

What prevents reading invalid memory when the vector file resizes?

The VectorSliceGuard in guard.rs validates a remap epoch counter before dereferencing. When ensure_capacity grows the file and creates a new MmapMut, it increments this atomic counter. Any existing guards holding the old epoch detect the mismatch on access, preventing undefined behavior from stale pointers.

Why does VelesDB use different storage strategies for vectors versus payloads?

Vectors are fixed-size (dimension * 4 bytes), making random access via memory-mapping efficient and enabling zero-copy reads. Payloads are variable-length JSON that would fragment a fixed-size mmap. The log-structured approach for payloads optimizes for append-only writes and fast recovery via snapshots, while the mmap approach for vectors optimizes for read-heavy similarity search workloads.

How does snapshot recovery work in the payload storage layer?

create_snapshot writes the current in-memory index to payloads.snapshot prefixed with the current WAL position. On startup, LogPayloadStorage::new loads this index and seeks to the saved position in payloads.log, replaying only newer entries. This reduces recovery time from O(n) to O(delta) where delta represents writes since the last snapshot.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}