Write-Ahead Logging (WAL) Architecture in pgrust: A Rust Reimplementation of PostgreSQL's Transaction Log

pgrust implements PostgreSQL's WAL system in safe Rust through the wal and xloginsert crates, providing record assembly, compression, and crash recovery while maintaining binary compatibility with PostgreSQL's transaction log format.

The pgrust project reimplements PostgreSQL's core engine in safe Rust, including its Write-Ahead Logging (WAL) architecture that ensures ACID compliance and crash recovery. This Rust-based WAL system mirrors the original C implementation's semantics while leveraging Rust's memory safety guarantees. The architecture spans multiple crates, with the core logic residing in wal for data types and xloginsert for record construction.

Core WAL Components and Crate Structure

The WAL implementation is distributed across two primary crates that handle distinct responsibilities:

• wal crate (crates/_support/types/wal/): Defines the WAL record vocabulary, constants, and decoded structures like XLogRecord and DecodedXLogRecord in wal/src/wal.rs and wal/src/lib.rs.
• xloginsert crate (crates/backend/access/transam/xloginsert/): Provides the WAL insertion API including XLogBeginInsert, XLogRegisterBuffer, and XLogInsert in xloginsert/src/lib.rs.
• Seam layer: The transam_xlog crate exposes the low-level xlog_insert_record seam that writes assembled records to WAL segment files.

WAL Record Vocabulary and Header Structures

The foundation of pgrust's WAL system lies in its strict adherence to PostgreSQL's binary format, defined in crates/_support/types/wal/src/wal.rs.

Resource Manager IDs and Flags

The crate defines standard resource-manager IDs and flag constants that match PostgreSQL's access/xlog* headers:

/// Resource‑manager IDs
pub const RM_XLOG_ID:  RmgrId = 0;
pub const RM_XACT_ID:  RmgrId = 1;
pub const RM_SMGR_ID:  RmgrId = 2;

/// Flags used in `xl_info`
pub const XLR_INFO_MASK: uint8 = 0x0F;
pub const XLR_SPECIAL_REL_UPDATE: uint8 = 0x01;
pub const XLR_CHECK_CONSISTENCY:   uint8 = 0x02;

The XLogRecord Header

Every WAL record begins with the XLogRecord header struct, defined in crates/_support/types/wal/src/lib.rs, which stores metadata including length, transaction ID, and CRC:

#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub struct XLogRecord {
    xl_tot_len: uint32,
    xl_xid:    TransactionId,
    xl_prev:   XLogRecPtr,
    xl_info:   uint8,
    xl_rmid:   RmgrId,
    xl_crc:    pg_crc32c,
}

During recovery, the system uses DecodedXLogRecord, which contains the header plus the main data slice and an array of DecodedBkpBlock structs describing block references, images, and per-block data.

The WAL Insertion Pipeline

The xloginsert crate implements a five-step insertion workflow that matches PostgreSQL's C implementation exactly. This process is orchestrated through thread-local state managed in XLogInsertState.

Step 1-3: Initialization and Buffer Registration

Record construction begins with XLogBeginInsert(), which verifies WAL insertion is allowed and activates the thread-local state. Buffer registration follows through XLogRegisterBuffer(block_id, buffer, flags), which copies page images:

// 1️⃣ Start a new WAL record
XLogBeginInsert()?;

// 2️⃣ Register the modified page (force a full‑page image)
XLogRegisterBuffer(0, buffer, REGBUF_FORCE_IMAGE)?;

// 3️⃣ Attach any extra payload
XLogRegisterData(data)?;

The alloc_block() function performs the copy, storing the page inside RegBuf.page along with the relation tag, fork number, and block number. Developers can also attach per-block data using XLogRegisterBufData(block_id, data).

Step 4-5: Assembly and Insertion

Before insertion, XLogSetRecordFlags(flags) sets metadata flags like XLOG_INCLUDE_ORIGIN. The XLogInsert(rmid, info) function then triggers XLogRecordAssemble, which performs several critical operations:

• Full-page decision: XLogCheckBufferNeedsBackup determines if a full-page image is required based on the current redo pointer and the page's LSN.
• Hole removal: For standard pages, the code computes holes using the lower/upper values from the page header.
• Compression: Optionally compresses images using PGLZ via XLogCompressBackupBlock.
• CRC calculation: Computes CRC32C over the header (minus the CRC field) and all data spans.
• Size validation: Enforces the XLOG_RECORD_MAX_SIZE limit of approximately 1 GB.

The assembled record passes to xlog_insert_record in the transam_xlog_seams module, which atomically writes the spans to WAL segment files while holding the WAL insertion lock.

Recovery and Redo Processing

During crash recovery, the redo functions consume DecodedXLogRecord structures. The decoder provides PostgreSQL-compatible helper methods like info(), xid(), has_block_ref(), and block_image_apply().

Resource managers receive a RedoRecord view containing the info byte, main data slice, and block presence flags. This abstraction, defined in crates/_support/types/wal/src/lib.rs, allows redo functions to apply changes without managing low-level memory unsafe operations.

Practical WAL Insertion Example

The following example demonstrates inserting a custom WAL record for resource manager RM_GENERIC_ID:

use pgrust::backend::access::transam::xloginsert::*;
use pgrust::wal::{RM_GENERIC_ID, XLR_SPECIAL_REL_UPDATE};

fn log_my_event(buffer: Buffer, data: &[u8]) -> PgResult<XLogRecPtr> {
    // 1️⃣ Start a new WAL record
    XLogBeginInsert()?;

    // 2️⃣ Register the modified page (force a full‑page image)
    XLogRegisterBuffer(0, buffer, REGBUF_FORCE_IMAGE)?;

    // 3️⃣ Attach any extra payload
    XLogRegisterData(data)?;

    // 4️⃣ Set a flag (e.g., include replication origin)
    XLogSetRecordFlags(XLOG_INCLUDE_ORIGIN);

    // 5️⃣ Insert the record – RM = generic, info = special‑rel‑update
    XLogInsert(RM_GENERIC_ID, XLR_SPECIAL_REL_UPDATE)
}

For recovery, iterate over decoded blocks:

use pgrust::wal::{DecodedXLogRecord, DecodedBkpBlock};

fn process_record(rec: &DecodedXLogRecord) {
    // Access header fields
    let xid = rec.xid();
    let info = rec.info();

    // Iterate over block references
    for (i, blk) in rec.blocks().iter().enumerate() {
        if blk.has_image() && blk.apply_image() {
            // Restore the page image...
        }
        if let Some(data) = blk.data() {
            // Apply per‑block data...
        }
    }

    // Main data payload
    let payload = rec.data();
    // …handle payload…
}

Summary

• Safe Rust implementation: pgrust replicates PostgreSQL's WAL semantics in safe Rust, eliminating memory safety vulnerabilities while preserving binary compatibility.
• Modular crate structure: The wal crate handles data types and constants, while xloginsert manages record construction and assembly.
• Five-step insertion: The XLogBeginInsert → XLogRegisterBuffer → XLogRegisterData → XLogSetRecordFlags → XLogInsert pipeline mirrors PostgreSQL's exact workflow.
• Compression and validation: Records undergo PGLZ compression (via XLogCompressBackupBlock) and CRC32C validation during assembly.
• Recovery support: The DecodedXLogRecord and RedoRecord abstractions provide type-safe access to WAL data during crash recovery.

Frequently Asked Questions

How does pgrust maintain binary compatibility with PostgreSQL's WAL format?

pgrust defines identical constants, struct layouts, and bit flags to PostgreSQL's C headers in crates/_support/types/wal/src/wal.rs. The XLogRecord struct uses the same field ordering and sizes as the C implementation, ensuring that WAL files produced by pgrust are readable by PostgreSQL and vice versa.

What is the maximum size of a single WAL record in pgrust?

The XLogRecordAssemble function enforces a hard limit of XLOG_RECORD_MAX_SIZE (approximately 1 GB) during record construction. If the assembled record exceeds this size, the insertion returns an error before reaching the xlog_insert_record seam.

How does pgrust handle full-page writes during WAL insertion?

The XLogCheckBufferNeedsBackup function determines whether a full-page image is required based on the current redo pointer and the page's LSN. When REGBUF_FORCE_IMAGE is passed to XLogRegisterBuffer, or when the page LSN indicates it was modified before the last checkpoint, the system copies the entire page image and optionally compresses it using PGLZ.

Which compression algorithm does pgrust use for WAL full-page images?

According to the xloginsert implementation in crates/backend/access/transam/xloginsert/src/lib.rs, pgrust uses PGLZ (PostgreSQL's LZ-based compression) via the XLogCompressBackupBlock function. This matches PostgreSQL's default compression method for full-page images.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}