Rust Implementation of the Buffer Manager and Storage Layer in pgrust

The pgrust project reimplements PostgreSQL's shared-buffer manager and storage subsystem in pure Rust using a seam-based architecture where the BufferManager struct in crates/backend/storage/buffer/bufmgr/src/mgr.rs centralizes descriptor arrays, page storage, and content locks, exposing operations via testable seam functions registered at runtime.

The pgrust repository provides a Rust-native implementation of PostgreSQL's buffer management and storage layer, translating the original C architecture from storage/buffer/bufmgr.c into a modular, testable design. This implementation preserves PostgreSQL's behavioral semantics while leveraging Rust's type safety and memory guarantees. The core system coordinates buffer descriptors, page memory, and per-backend pinning through a seam-based API that decouples interface from implementation.

Core Architecture of the Buffer Manager

The BufferManager Singleton

In crates/backend/storage/buffer/bufmgr/src/mgr.rs, the BufferManager struct owns the global state backing the shared buffer pool. It maintains the descriptor array (BufferDesc) tracking each buffer's metadata, the page-byte array holding actual BLCKSZ-sized pages, per-buffer content locks (lwlock::LWLock), and the per-backend pin-count map (refcount). All public operations delegate to a singleton instance accessed via BufferManager::global_expect(), ensuring every call sees a consistent view of the pool.

Seam Functions and Public API

The buffer manager exposes operations via seam functions defined in crates/backend/storage/buffer/bufmgr/src/lib.rs. These seams are registered at runtime through init_seams(), allowing higher-level crates to call buffer operations without depending on concrete implementation details. Key seams include:

• read_buffer: Synchronous ReadBuffer(rel, blkno) that pins a buffer and returns its Buffer id (lines 63-70)
• release_buffer: Drops one pin via ReleaseBuffer(buf) (lines 69-74)
• lock_buffer and lock_buffer_exclusive: Acquire content locks in shared or exclusive mode (lines 107-110)
• mark_buffer_dirty: Sets the dirty flag on a pinned buffer required for write-back (lines 55-62)
• extend_buffered_rel: Allocates a new block on a relation and returns a write-locked, pinned buffer (lines 66-75)
• flush_one_buffer: Writes a single dirty buffer to disk (lines 80-87)
• init_buffer_manager_access: Initializes the per-backend pin map and registers cleanup hooks (lines 85-90)

Storage Layer Components

Primitive Types and Abstractions

The storage layer relies on type definitions in crates/_support/types/storage/src/lib.rs, which provides primitive types like RelFileLocator, BlockNumber, ForkNumber, and ReadBufferMode. These types map PostgreSQL relations onto physical on-disk file identifiers and are used throughout the storage code to maintain type safety across the boundary between logical tables and physical storage.

Page Management and Storage Manager

Page-level utilities reside in crates/backend/storage/page/src/lib.rs, implementing functions such as page_init, page_set_lsn, and page_is_new that operate on raw &mut [u8] slices. The storage manager (smgr) in crates/backend/storage/smgr/src/lib.rs handles whole-page I/O to the underlying file system, exposing smgrextend, smgrread, and smgrwrite. The buffer manager invokes these functions on cache misses to read pages from disk or write them back during flushing.

Shared Memory Infrastructure

Shared-memory allocation for the buffer pool is managed in crates/backend/storage/ipc/src/ipci_core.rs. This module calculates buffer_manager_shmem_size and performs buffer_manager_shmem_init to establish the shared memory segment that backs the global buffer pool, enabling multi-process access to the same physical page cache.

Operational Workflow

The buffer manager and storage layer interact through a four-stage pipeline during backend operation.

1. Backend Startup: The system calls init_buffer_manager_access from crates/backend/utils/init/postinit/src/lib.rs to allocate the per-process pin map and register the UnlockBuffers cleanup hook.

2. Relation Access: When executors or extensions require a block, they invoke the read_buffer seam. If the block resides in the pool, the buffer is pinned immediately; otherwise, the manager calls smgrread to fetch the page from disk, allocates a descriptor, and returns the buffer id.

3. Modification: After changing page content, callers invoke mark_buffer_dirty to set the buffer's dirty flag. During transaction commit or checkpoint, flush_one_buffer (or the background writer) writes modified pages back to stable storage via smgrwrite.

4. Extension: The extend_buffered_rel seam creates new blocks on a relation's fork, locks the new page exclusively, and returns a buffer ready for data insertion, mirroring PostgreSQL's ExtendBufferedRel behavior.

All operations remain stateless from the caller's perspective, with the global BufferManager holding mutable state while seams provide a clean, testable API surface.

Working with the Buffer Manager in Practice

The seam-based design allows code to interact with the buffer pool through simple function calls that resolve to the global BufferManager instance. The with_buffer_page function provides safe access to underlying page bytes, mirroring the C macro BufferGetPage.

use pgrust::bufmgr_seams as bufmgr;
use pgrust::types_storage::storage::{ReadBufferMode, RelFileLocator};
use pgrust::types_storage::buf::BufferAccessStrategy;
use pgrust::types_core::primitive::{BlockNumber, ForkNumber};

/// Read a page from a relation (pinning it)
fn read_page_example(rel: &rel::Relation, blkno: BlockNumber) -> PgResult<Buffer> {
    // Calls the `read_buffer` seam → `ReadBuffer(rel, blkno)`
    bufmgr::read_buffer::call(rel, blkno)
}

/// Extend a relation by one block (useful for INSERT)
fn extend_one_block(rel: &rel::Relation, fork: ForkNumber) -> PgResult<Buffer> {
    // Mirrors `ExtendBufferedRel(rel, fork, …)`
    bufmgr::extend_buffered_rel::call(rel, fork)
}

/// Mark the buffer dirty and write it back explicitly
fn dirty_and_flush(buf: Buffer) -> PgResult<()> {
    bufmgr::mark_buffer_dirty::call(buf);
    // Flush the single dirty buffer to disk
    bufmgr::flush_one_buffer::call(buf)
}

/// Work with the underlying page bytes safely
fn modify_page(buf: Buffer) -> PgResult<()> {
    // `with_buffer_page` gives a mutable slice of the page content.
    bufmgr::with_buffer_page::call(buf, &mut |page| {
        // Example: zero-out the whole page
        for byte in page.iter_mut() {
            *byte = 0;
        }
        Ok(())
    })
}

These calls work uniformly whether the backing storage is an in-memory simulation or a live PostgreSQL data directory, as the seams delegate to BufferManager::global_expect().

Summary

• The pgrust buffer manager reimplements PostgreSQL's bufmgr.c in Rust, maintaining architectural compatibility while improving testability through seams.
• The BufferManager struct in crates/backend/storage/buffer/bufmgr/src/mgr.rs centralizes buffer descriptors, page storage, content locks, and per-backend pinning maps.
• Seam functions in crates/backend/storage/buffer/bufmgr/src/lib.rs provide a runtime-wired API that decouples callers from implementation details.
• The storage layer comprises primitive types (_support/types/storage), page utilities (storage/page), the storage manager (storage/smgr), and shared-memory infrastructure (storage/ipc).
• Operations follow the PostgreSQL workflow: initialize via init_buffer_manager_access, access via read_buffer, modify via mark_buffer_dirty, and extend via extend_buffered_rel.

Frequently Asked Questions

What is a seam in pgrust's buffer manager?

A seam is a small, testable function exported from crates/backend/storage/buffer/bufmgr/src/lib.rs that wraps buffer manager operations. Seams are registered at runtime via init_seams(), allowing higher-level crates to call functions like read_buffer or mark_buffer_dirty without importing implementation details from mgr.rs. This pattern enables dependency injection for unit testing while maintaining PostgreSQL-compatible semantics.

How does pgrust handle buffer pinning and unpinning?

Buffer pinning is managed through the per-backend pin-count map (refcount) stored in the global BufferManager. When code calls the read_buffer seam, the manager increments the pin count for that buffer descriptor. Calling release_buffer decrements the count. The UnlockBuffers cleanup hook registered during init_buffer_manager_access ensures pins are released automatically if a backend exits unexpectedly, preventing resource leaks.

What is the relationship between the buffer manager and smgr?

The storage manager (smgr) in crates/backend/storage/smgr/src/lib.rs operates below the buffer manager, handling direct file I/O. When the buffer manager encounters a cache miss during read_buffer, it invokes smgrread to fetch the page from disk. Similarly, flush_one_buffer delegates to smgrwrite to persist dirty pages. This separation allows the buffer manager to focus on caching and concurrency while smgr manages physical storage layout.

How does pgrust ensure thread safety in the buffer pool?

Thread safety is achieved through content locks (lwlock::LWLock) associated with each buffer descriptor and the global singleton pattern of BufferManager::global_expect(). The seam-based API ensures that all mutable state access passes through the centralized manager, while Rust's ownership model prevents data races on the page slices handed out via with_buffer_page. Shared memory initialization in ipci_core.rs establishes the proper memory layout for multi-process access as required by PostgreSQL's architecture.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}