Understanding Alibaba zvec Storage Backends: Memory, mmap, and BufferPool

Alibaba zvec provides three production-ready storage backends—memory, mmap, and BufferPool—that share a unified BaseForwardStore interface, allowing zero-copy reads via MmapForwardStore or explicit memory management via BufferPoolForwardStore depending on workload characteristics.

The zvec vector database engine abstracts disk I/O behind a pluggable storage backend interface, enabling developers to optimize for latency, throughput, or memory constraints. Understanding these alibaba zvec storage backends is essential for tuning vector search performance, as the choice between memory-mapped files and managed buffer pools directly impacts CPU overhead and cache efficiency.

How zvec Abstracts Storage Backends

The storage layer is defined by the BaseForwardStore interface, which declares methods like fetch, scan, and physic_schema. Concrete implementations are selected via the StorageType enum defined in src/include/zvec/core/interface/index_param.h:

enum class StorageType { kNone, kMMAP, kMemory, kBufferPool };

When constructing an index, IndexFactory::CreateStorage (located in tools/core/helper.h) instantiates concrete classes:

• IndexFactory::CreateStorage("MMapFileStorage") returns a MmapForwardStore
• IndexFactory::CreateStorage("BufferPoolStorage") returns a BufferPoolForwardStore

Memory-Mapped File Backend (mmap)

The mmap backend leverages the operating system's virtual memory manager to provide zero-copy access to vector files. This is ideal for large, read-only datasets that fit within the virtual address space.

Low-Level Memory Mapping with MMapFile

The ailego::MMapFile class in src/include/zvec/ailego/io/mmap_file.h wraps platform-specific mmap syscalls with move-only semantics to prevent double-unmap errors. Key capabilities include:

• Zero-copy reads: The read(const void **data, size_t len) method returns a pointer directly into the mapped region, eliminating memcpy overhead.
• Warm-up and locking: The warmup() method prefetches pages into RAM, while lock() pins them to prevent swapping.
• Flexible modes: Supports read-only and shared mappings via the open method.

High-Level Vector Access with MmapForwardStore

MmapForwardStore (defined in src/db/index/storage/mmap_forward_store.h and implemented in .cc) builds on MMapFile to provide Arrow-compatible vector access. When Open() is called, it memory-maps the file and constructs Arrow schema metadata.

The fetch and scan methods translate column lists and row indices into Arrow Table or RecordBatchReader objects. Because the underlying file is memory-mapped, Arrow operates directly on the mapped buffer without copying data. The class also provides FindRowGroupForRow and GetRowGroupOffset helpers to navigate Parquet row groups efficiently.

BufferPool Backend

For workloads requiring explicit memory management or mixed read/write patterns, the BufferPool backend provides a managed in-process cache.

Managed Memory with BufferManager

The ailego::BufferManager class in src/include/zvec/ailego/buffer/buffer_manager.h implements a singleton pattern to provide a global per-process memory pool. It allocates fixed-size pages that can be reused across multiple storage operations, reducing fragmentation for many small Arrow reads.

Callers obtain a BufferPool* via GetPool() and interact with it through Allocate, Release, and Pin operations. This design integrates with Arrow's RandomAccessFile interface while keeping memory usage under explicit application control.

Explicit Caching with BufferPoolForwardStore

BufferPoolForwardStore (located in src/db/index/storage/bufferpool_forward_store.h and .cc) implements the same BaseForwardStore interface as its mmap counterpart but copies data into buffers provided by the BufferManager.

During Open(), it creates a parquet::arrow::FileReader that reads through Arrow's io::ReadableFile. When fetch or scan is called, column chunks are copied into pooled buffers and wrapped in Arrow ArrayData structures. This approach benefits high-throughput recall workloads where repeated access to overlapping column sets can avoid reallocations through buffer reuse.

Configuring Storage Backends in Practice

Backend selection occurs at index construction time through the StorageType enum. The configuration flows through core_interface::StorageOptions::type, which defaults to kNone.

Factory methods in tools/core/helper.h instantiate concrete implementations based on string identifiers:

// For memory-mapped storage
auto store = IndexFactory::CreateStorage("MMapFileStorage");

// For buffer pool storage  
auto store = IndexFactory::CreateStorage("BufferPoolStorage");

Command-line tools often expose flags such as --storage_type=MMAP or --storage_type=BUFFERPOOL, which set options.storage_options.type accordingly before constructing the index.

Practical Code Examples

Creating an Index with mmap

#include "zvec/core/interface/index_param.h"
#include "zvec/core/interface/index_factory.h"

zvec::core::interface::IndexParam param;
param.storage_options.type = zvec::core::interface::StorageType::kMMAP;

auto index = zvec::core::interface::IndexFactory::CreateIndex(param);
// Returns an index backed by MmapForwardStore with zero-copy reads

Creating an Index with BufferPool

zvec::core::interface::IndexParam param;
param.storage_options.type = zvec::core::interface::StorageType::kBufferPool;

auto index = zvec::core::interface::IndexFactory::CreateIndex(param);
// Uses BufferPoolForwardStore with explicit buffer management

Warming Up a Memory-Mapped File

// Assuming store is a MmapForwardStore already opened
store->mmap_file_.warmup();   // Prefetch pages into RAM
store->mmap_file_.lock();     // Pin pages (prevents swapping)

These calls are exposed via the configuration flag proxima.mmap_file.storage.memory_warmup in benchmark scripts.

Direct Low-Level MMapFile Usage

#include "zvec/ailego/io/mmap_file.h"

ailego::MMapFile mmap;
if (mmap.open("/path/to/vectors.vecs", true, true)) {
    const void* data;
    size_t sz = mmap.read(&data, /*len=*/4096);   // Zero-copy read
    // data points directly into the mapped region
}

This approach is useful for custom tooling that does not require the full Arrow stack.

Key Source Files Reference

Component	File Path	Description
Storage Type Enum	src/include/zvec/core/interface/index_param.h	Defines StorageType enum with kMMAP, kBufferPool, and kMemory values
Memory-Mapped File	src/include/zvec/ailego/io/mmap_file.h	Implements ailego::MMapFile with zero-copy reads, warmup, and locking
Mmap Forward Store	src/db/index/storage/mmap_forward_store.h	High-level Arrow integration for memory-mapped vector files
Buffer Manager	src/include/zvec/ailego/buffer/buffer_manager.h	Singleton memory pool for reusable buffer pages
BufferPool Forward Store	src/db/index/storage/bufferpool_forward_store.h	Managed buffer implementation with explicit caching
Storage Factory	tools/core/helper.h	Contains IndexFactory::CreateStorage for runtime backend selection

Summary

• Alibaba zvec storage backends are interchangeable implementations of the BaseForwardStore interface, selected via the StorageType enum (kMMAP, kBufferPool, or kMemory).
• The mmap backend (MmapForwardStore) leverages ailego::MMapFile to provide zero-copy reads directly from the operating system's page cache, ideal for large read-only vector files.
• The BufferPool backend (BufferPoolForwardStore) uses the singleton ailego::BufferManager to allocate reusable memory pages, reducing allocation churn for high-throughput workloads with many small reads.
• Both backends expose identical APIs (fetch, scan, physic_schema) and integrate with Apache Arrow, allowing seamless switching through configuration parameters without application code changes.
• The design isolates low-level file handling (MMapFile / BufferManager) from higher-level Arrow/Parquet logic, keeping the codebase modular and testable.

Frequently Asked Questions

What is the difference between mmap and BufferPool in zvec?

The mmap backend maps files directly into the process's virtual address space using ailego::MMapFile, enabling zero-copy reads where the OS handles paging automatically. The BufferPool backend copies data from disk into explicitly managed memory pages via ailego::BufferManager, giving the application control over caching and eviction policies. Choose mmap for large sequential reads and BufferPool for workloads requiring fine-grained memory management or mixed read/write patterns.

How do I configure zvec to use a specific storage backend?

Set the storage_options.type field in IndexParam to StorageType::kMMAP or StorageType::kBufferPool before calling IndexFactory::CreateIndex(). Alternatively, command-line tools in the repository accept flags like --storage_type=MMAP or --storage_type=BUFFERPOOL, which the factory methods in tools/core/helper.h parse to instantiate the correct MmapForwardStore or BufferPoolForwardStore implementation.

When should I use the memory warmup feature with mmap?

Enable memory warmup by calling MMapFile::warmup() or setting proxima.mmap_file.storage.memory_warmup to true when serving latency-critical vector search queries from large files that exceed physical RAM. This triggers sequential prefetching of pages into memory and optionally lock() to pin them, eliminating page faults during query execution. Skip warmup if the working set fits entirely in memory or if you prefer to let the OS demand-paging system manage caching naturally.

Can I use BufferPool and mmap interchangeably in the same application?

Yes, both backends implement the identical BaseForwardStore interface with fetch, scan, and physic_schema methods, allowing runtime polymorphism through the IndexFactory. You can instantiate different indexes with different storage types within the same process—one using kMMAP for a large static dataset and another using kBufferPool for a dynamic, high-throughput workload—without modifying application logic that consumes the forward store API.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}