# Understanding Alibaba zvec Storage Backends: Memory, mmap, and BufferPool

> Explore Alibaba zvec storage backends: memory, mmap, and BufferPool. Learn about zero-copy reads and memory management for optimal performance with the unified BaseForwardStore interface.

- Repository: [Alibaba/zvec](https://github.com/alibaba/zvec)
- Tags: internals
- Published: 2026-02-16

---

**Alibaba zvec provides three production-ready storage backends—memory, mmap, and BufferPool—that share a unified `BaseForwardStore` interface, allowing zero-copy reads via `MmapForwardStore` or explicit memory management via `BufferPoolForwardStore` depending on workload characteristics.**

The `zvec` vector database engine abstracts disk I/O behind a pluggable storage backend interface, enabling developers to optimize for latency, throughput, or memory constraints. Understanding these alibaba zvec storage backends is essential for tuning vector search performance, as the choice between memory-mapped files and managed buffer pools directly impacts CPU overhead and cache efficiency.

## How zvec Abstracts Storage Backends

The storage layer is defined by the `BaseForwardStore` interface, which declares methods like `fetch`, `scan`, and `physic_schema`. Concrete implementations are selected via the `StorageType` enum defined in [`src/include/zvec/core/interface/index_param.h`](https://github.com/alibaba/zvec/blob/main/src/include/zvec/core/interface/index_param.h):

```cpp
enum class StorageType { kNone, kMMAP, kMemory, kBufferPool };

```

When constructing an index, `IndexFactory::CreateStorage` (located in [`tools/core/helper.h`](https://github.com/alibaba/zvec/blob/main/tools/core/helper.h)) instantiates concrete classes:

- `IndexFactory::CreateStorage("MMapFileStorage")` returns a `MmapForwardStore`
- `IndexFactory::CreateStorage("BufferPoolStorage")` returns a `BufferPoolForwardStore`

## Memory-Mapped File Backend (mmap)

The **mmap** backend leverages the operating system's virtual memory manager to provide zero-copy access to vector files. This is ideal for large, read-only datasets that fit within the virtual address space.

### Low-Level Memory Mapping with MMapFile

The `ailego::MMapFile` class in [`src/include/zvec/ailego/io/mmap_file.h`](https://github.com/alibaba/zvec/blob/main/src/include/zvec/ailego/io/mmap_file.h) wraps platform-specific `mmap` syscalls with move-only semantics to prevent double-unmap errors. Key capabilities include:

- **Zero-copy reads**: The `read(const void **data, size_t len)` method returns a pointer directly into the mapped region, eliminating `memcpy` overhead.
- **Warm-up and locking**: The `warmup()` method prefetches pages into RAM, while `lock()` pins them to prevent swapping.
- **Flexible modes**: Supports read-only and shared mappings via the `open` method.

### High-Level Vector Access with MmapForwardStore

`MmapForwardStore` (defined in [`src/db/index/storage/mmap_forward_store.h`](https://github.com/alibaba/zvec/blob/main/src/db/index/storage/mmap_forward_store.h) and implemented in `.cc`) builds on `MMapFile` to provide Arrow-compatible vector access. When `Open()` is called, it memory-maps the file and constructs Arrow schema metadata.

The `fetch` and `scan` methods translate column lists and row indices into Arrow `Table` or `RecordBatchReader` objects. Because the underlying file is memory-mapped, Arrow operates directly on the mapped buffer without copying data. The class also provides `FindRowGroupForRow` and `GetRowGroupOffset` helpers to navigate Parquet row groups efficiently.

## BufferPool Backend

For workloads requiring explicit memory management or mixed read/write patterns, the **BufferPool** backend provides a managed in-process cache.

### Managed Memory with BufferManager

The `ailego::BufferManager` class in [`src/include/zvec/ailego/buffer/buffer_manager.h`](https://github.com/alibaba/zvec/blob/main/src/include/zvec/ailego/buffer/buffer_manager.h) implements a singleton pattern to provide a global per-process memory pool. It allocates fixed-size pages that can be reused across multiple storage operations, reducing fragmentation for many small Arrow reads.

Callers obtain a `BufferPool*` via `GetPool()` and interact with it through `Allocate`, `Release`, and `Pin` operations. This design integrates with Arrow's `RandomAccessFile` interface while keeping memory usage under explicit application control.

### Explicit Caching with BufferPoolForwardStore

`BufferPoolForwardStore` (located in [`src/db/index/storage/bufferpool_forward_store.h`](https://github.com/alibaba/zvec/blob/main/src/db/index/storage/bufferpool_forward_store.h) and `.cc`) implements the same `BaseForwardStore` interface as its mmap counterpart but copies data into buffers provided by the `BufferManager`.

During `Open()`, it creates a `parquet::arrow::FileReader` that reads through Arrow's `io::ReadableFile`. When `fetch` or `scan` is called, column chunks are copied into pooled buffers and wrapped in Arrow `ArrayData` structures. This approach benefits high-throughput recall workloads where repeated access to overlapping column sets can avoid reallocations through buffer reuse.

## Configuring Storage Backends in Practice

Backend selection occurs at index construction time through the `StorageType` enum. The configuration flows through `core_interface::StorageOptions::type`, which defaults to `kNone`.

Factory methods in [`tools/core/helper.h`](https://github.com/alibaba/zvec/blob/main/tools/core/helper.h) instantiate concrete implementations based on string identifiers:

```cpp
// For memory-mapped storage
auto store = IndexFactory::CreateStorage("MMapFileStorage");

// For buffer pool storage  
auto store = IndexFactory::CreateStorage("BufferPoolStorage");

```

Command-line tools often expose flags such as `--storage_type=MMAP` or `--storage_type=BUFFERPOOL`, which set `options.storage_options.type` accordingly before constructing the index.

## Practical Code Examples

### Creating an Index with mmap

```cpp
#include "zvec/core/interface/index_param.h"
#include "zvec/core/interface/index_factory.h"

zvec::core::interface::IndexParam param;
param.storage_options.type = zvec::core::interface::StorageType::kMMAP;

auto index = zvec::core::interface::IndexFactory::CreateIndex(param);
// Returns an index backed by MmapForwardStore with zero-copy reads

```

### Creating an Index with BufferPool

```cpp
zvec::core::interface::IndexParam param;
param.storage_options.type = zvec::core::interface::StorageType::kBufferPool;

auto index = zvec::core::interface::IndexFactory::CreateIndex(param);
// Uses BufferPoolForwardStore with explicit buffer management

```

### Warming Up a Memory-Mapped File

```cpp
// Assuming store is a MmapForwardStore already opened
store->mmap_file_.warmup();   // Prefetch pages into RAM
store->mmap_file_.lock();     // Pin pages (prevents swapping)

```

These calls are exposed via the configuration flag `proxima.mmap_file.storage.memory_warmup` in benchmark scripts.

### Direct Low-Level MMapFile Usage

```cpp
#include "zvec/ailego/io/mmap_file.h"

ailego::MMapFile mmap;
if (mmap.open("/path/to/vectors.vecs", true, true)) {
    const void* data;
    size_t sz = mmap.read(&data, /*len=*/4096);   // Zero-copy read
    // data points directly into the mapped region
}

```

This approach is useful for custom tooling that does not require the full Arrow stack.

## Key Source Files Reference

| Component | File Path | Description |
|-----------|-----------|-------------|
| **Storage Type Enum** | [`src/include/zvec/core/interface/index_param.h`](https://github.com/alibaba/zvec/blob/main/src/include/zvec/core/interface/index_param.h) | Defines `StorageType` enum with `kMMAP`, `kBufferPool`, and `kMemory` values |
| **Memory-Mapped File** | [`src/include/zvec/ailego/io/mmap_file.h`](https://github.com/alibaba/zvec/blob/main/src/include/zvec/ailego/io/mmap_file.h) | Implements `ailego::MMapFile` with zero-copy reads, warmup, and locking |
| **Mmap Forward Store** | [`src/db/index/storage/mmap_forward_store.h`](https://github.com/alibaba/zvec/blob/main/src/db/index/storage/mmap_forward_store.h) | High-level Arrow integration for memory-mapped vector files |
| **Buffer Manager** | [`src/include/zvec/ailego/buffer/buffer_manager.h`](https://github.com/alibaba/zvec/blob/main/src/include/zvec/ailego/buffer/buffer_manager.h) | Singleton memory pool for reusable buffer pages |
| **BufferPool Forward Store** | [`src/db/index/storage/bufferpool_forward_store.h`](https://github.com/alibaba/zvec/blob/main/src/db/index/storage/bufferpool_forward_store.h) | Managed buffer implementation with explicit caching |
| **Storage Factory** | [`tools/core/helper.h`](https://github.com/alibaba/zvec/blob/main/tools/core/helper.h) | Contains `IndexFactory::CreateStorage` for runtime backend selection |

## Summary

- **Alibaba zvec storage backends** are interchangeable implementations of the `BaseForwardStore` interface, selected via the `StorageType` enum (`kMMAP`, `kBufferPool`, or `kMemory`).
- The **mmap backend** (`MmapForwardStore`) leverages `ailego::MMapFile` to provide zero-copy reads directly from the operating system's page cache, ideal for large read-only vector files.
- The **BufferPool backend** (`BufferPoolForwardStore`) uses the singleton `ailego::BufferManager` to allocate reusable memory pages, reducing allocation churn for high-throughput workloads with many small reads.
- Both backends expose identical APIs (`fetch`, `scan`, `physic_schema`) and integrate with Apache Arrow, allowing seamless switching through configuration parameters without application code changes.
- The design isolates low-level file handling (`MMapFile` / `BufferManager`) from higher-level Arrow/Parquet logic, keeping the codebase modular and testable.

## Frequently Asked Questions

### What is the difference between mmap and BufferPool in zvec?

The **mmap** backend maps files directly into the process's virtual address space using `ailego::MMapFile`, enabling zero-copy reads where the OS handles paging automatically. The **BufferPool** backend copies data from disk into explicitly managed memory pages via `ailego::BufferManager`, giving the application control over caching and eviction policies. Choose mmap for large sequential reads and BufferPool for workloads requiring fine-grained memory management or mixed read/write patterns.

### How do I configure zvec to use a specific storage backend?

Set the `storage_options.type` field in `IndexParam` to `StorageType::kMMAP` or `StorageType::kBufferPool` before calling `IndexFactory::CreateIndex()`. Alternatively, command-line tools in the repository accept flags like `--storage_type=MMAP` or `--storage_type=BUFFERPOOL`, which the factory methods in [`tools/core/helper.h`](https://github.com/alibaba/zvec/blob/main/tools/core/helper.h) parse to instantiate the correct `MmapForwardStore` or `BufferPoolForwardStore` implementation.

### When should I use the memory warmup feature with mmap?

Enable memory warmup by calling `MMapFile::warmup()` or setting `proxima.mmap_file.storage.memory_warmup` to true when serving latency-critical vector search queries from large files that exceed physical RAM. This triggers sequential prefetching of pages into memory and optionally `lock()` to pin them, eliminating page faults during query execution. Skip warmup if the working set fits entirely in memory or if you prefer to let the OS demand-paging system manage caching naturally.

### Can I use BufferPool and mmap interchangeably in the same application?

Yes, both backends implement the identical `BaseForwardStore` interface with `fetch`, `scan`, and `physic_schema` methods, allowing runtime polymorphism through the `IndexFactory`. You can instantiate different indexes with different storage types within the same process—one using `kMMAP` for a large static dataset and another using `kBufferPool` for a dynamic, high-throughput workload—without modifying application logic that consumes the forward store API.