How pgrust Implements Write-Ahead Logging (WAL): A Rust Port of PostgreSQL's Architecture

pgrust implements Write-Ahead Logging by porting PostgreSQL's C-based WAL machinery to safe Rust, organizing the core logic into the wal crate for record definitions and the xloginsert module for the five-step insertion pipeline that assembles, compresses, and writes records to disk.

The malisper/pgrust project provides a Rust-native reimplementation of PostgreSQL's storage engine, including a faithful port of its Write-Ahead Logging (WAL) system. This implementation preserves PostgreSQL's exact semantics and recovery guarantees while leveraging Rust's memory safety, with the core WAL vocabulary, record structures, and insertion logic distributed across the wal and xloginsert crates.

WAL Record Architecture and Core Types

The foundation of pgrust's WAL system resides in the wal crate, which defines the constants, headers, and decoded structures necessary for both record creation and replay.

The XLogRecord Header Structure

In crates/_support/types/wal/src/wal.rs, pgrust defines the XLogRecord header that precedes every WAL entry on disk. This struct mirrors PostgreSQL's binary layout exactly:

#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub struct XLogRecord {
    xl_tot_len: uint32,
    xl_xid:    TransactionId,
    xl_prev:   XLogRecPtr,
    xl_info:   uint8,
    xl_rmid:   RmgrId,
    xl_crc:    pg_crc32c,
}

The file also exports the resource-manager ID constants (e.g., RM_XLOG_ID, RM_XACT_ID, RM_SMGR_ID) and flag bits (XLR_INFO_MASK, XLR_SPECIAL_REL_UPDATE, XLR_CHECK_CONSISTENCY) that identify which subsystem owns the record and how it should be processed.

Decoded Record Structures for Recovery

For replay operations, crates/_support/types/wal/src/lib.rs provides DecodedXLogRecord, which unpacks the header and provides convenient accessors for the main data payload and block references. Each block reference is represented as a DecodedBkpBlock containing the relation tag, fork number, block number, and optional full-page image or data slice. The RedoRecord view type exposes a simplified interface specifically for resource-manager redo functions, offering methods like info(), xid(), record_origin(), and has_block_ref() that correspond to PostgreSQL's XLogRecGet* helpers.

The WAL Insertion Pipeline

The xloginsert crate (crates/backend/access/transam/xloginsert/src/lib.rs) implements the procedural API for constructing and emitting WAL records, following PostgreSQL's five-step pattern exactly.

Step-by-Step Record Construction

Pgrust constructs a WAL record through a strict sequence of API calls that manage thread-local insertion state:

1. Start a new record – XLogBeginInsert() verifies that WAL insertion is allowed and marks the thread-local XLogInsertState as active.

2. Register buffers – XLogRegisterBuffer(block_id, buffer, flags) copies the page image from a Buffer and stores the relation metadata. The copy is performed in alloc_block() and stored inside RegBuf.page.

3. Add arbitrary data – XLogRegisterData(data) appends bytes to the main chunk, while XLogRegisterBufData(block_id, data) attaches data to a specific block reference.

4. Set record flags – XLogSetRecordFlags(flags) (e.g., XLOG_INCLUDE_ORIGIN) modifies the xl_info field.

5. Insert the record – XLogInsert(rmid, info) finalizes the operation by calling XLogRecordAssemble and handing the assembled spans to the low-level xlog_insert_record seam.

Record Assembly and Compression

The XLogRecordAssemble function in xloginsert/src/lib.rs performs the heavy lifting of converting registered buffers and data into the on-disk format:

• Full-page decision – XLogCheckBufferNeedsBackup compares the page's LSN against the current redo pointer to determine if a full-page image is required.
• Hole removal – For standard pages, the code calculates the hole between lower and upper in the page header and excludes it from the image.
• Compression – XLogCompressBackupBlock compresses the image using PGLZ (PostgreSQL's LZ compression) when beneficial.
• CRC calculation – The implementation computes a CRC32C accumulator over the header (excluding the CRC field itself) and every data span, matching PostgreSQL's COMP_CRC32C logic.
• Size guarding – If the assembled record exceeds XLOG_RECORD_MAX_SIZE (approximately 1 GB), the function returns an error before writing.

Writing to WAL Segment Files

Once assembled, XLogInsert obtains the current full-page-write flag and redo pointer via transam_xlog_seams::get_full_page_write_info. It then calls xlog_insert_record (implemented in the transam_xlog crate) which receives the slice of byte spans, the full-page-write LSN, flags, and the number of full-page images. This seam writes the data atomically while holding the WAL insertion lock, returning the end LSN upon completion. The insertion loop handles retry logic for full-page-write conflicts before clearing state with XLogResetInsertion.

Recovery and Redo Processing

During crash recovery, pgrust uses the DecodedXLogRecord type to iterate over WAL entries. The decoder provides methods to inspect block images (block_image_apply()), access per-block data, and read the main payload. Resource managers receive a RedoRecord view generated from the decoded record, allowing them to reconstruct page state by applying the saved images or replaying the logical operations encoded in the record data.

Summary

• pgrust's WAL implementation mirrors PostgreSQL's C architecture but is written in safe Rust, located primarily in the wal and xloginsert crates.
• Record headers are defined by XLogRecord in wal/src/wal.rs, while decoded structures like DecodedXLogRecord and DecodedBkpBlock live in wal/src/lib.rs.
• Insertion follows a five-step pattern: begin, register buffers, register data, set flags, and insert via XLogInsert in xloginsert/src/lib.rs.
• Assembly handles full-page image decisions, hole removal, PGLZ compression via XLogCompressBackupBlock, and CRC32C calculation.
• Actual disk writes occur through the xlog_insert_record seam in the transam_xlog crate, which manages segment file I/O and atomicity.

Frequently Asked Questions

How does pgrust's WAL implementation differ from PostgreSQL's original C code?

While pgrust preserves the exact binary format, semantics, and recovery guarantees of PostgreSQL's WAL, it reimplements the logic in safe Rust. The core types like XLogRecord and DecodedXLogRecord use Rust's struct layout and memory safety guarantees instead of C pointers and manual memory management, but the file organization in crates/_support/types/wal/ and crates/backend/access/transam/xloginsert/ directly corresponds to PostgreSQL's access/xlog* source files.

What is the maximum size of a single WAL record in pgrust?

Pgrust enforces the same limit as PostgreSQL: XLOG_RECORD_MAX_SIZE, which is approximately 1 GB. The XLogRecordAssemble function checks the assembled record size against this constant and returns an error if the limit is exceeded, preventing unbounded memory usage during record construction.

How does pgrust calculate CRC values for WAL records?

The implementation uses CRC32C (Castagnoli) calculated over the entire record header (excluding the xl_crc field itself) and all attached data spans. This occurs in XLogRecordAssemble within xloginsert/src/lib.rs, using the same accumulator logic as PostgreSQL's COMP_CRC32C macro to ensure compatibility with existing PostgreSQL WAL files.

Which compression algorithm does pgrust use for full-page images?

Pgrust currently uses PGLZ (PostgreSQL's historical LZ compression) for compressing full-page images. The XLogCompressBackupBlock function in xloginsert/src/lib.rs handles this compression when the "hole" removed from a standard page still leaves a large enough image to benefit from compression, though the specific compression routine may be extended to support newer algorithms in future versions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}