How pgrust Achieves Disk Compatibility with PostgreSQL: Byte-Level Replication Techniques

pgrust achieves disk compatibility with PostgreSQL by implementing C-compatible struct layouts, explicit byte-for-byte serializers, and native-endian handling that replicate every on-disk structure from heap tuples to WAL records, allowing seamless interchangeability between Rust and C PostgreSQL instances.

pgrust is a Rust re-implementation of PostgreSQL's core engine designed to maintain full binary interchangeability with standard PostgreSQL data directories. To achieve disk compatibility with PostgreSQL, the project meticulously reproduces identical byte layouts for heap tuples, TOAST objects, control files, and WAL records. This ensures that a PostgreSQL data directory created by official binaries can be opened by a pgrust server without any conversion or migration step.

C-Compatible Struct Definitions

The foundation of disk compatibility lies in using #[repr(C)] attributes to ensure Rust structs match the memory layout, padding, and alignment of PostgreSQL's C structures exactly.

In crates/backend/access/common/heaptuple/src/lib.rs, the HeapTupleHeaderChoice struct is defined with C-compatible layout at line 25. This guarantees that the header fields—such as transaction IDs and flags—occupy the exact same byte offsets as they do in the C implementation.

Explicit Serialization Functions

While struct layout ensures memory alignment, explicit serialization functions control the exact byte order written to disk. The heap_tuple_to_disk_image and heap_copytuple_from_disk_image functions in crates/backend/access/common/heaptuple/src/lib.rs (lines 2503–2510 and 2549–2562) handle the precise serialization logic.

These functions write fields in the exact order and size that PostgreSQL's heap_fill_tuple and heap_copytuple expect, including PostgreSQL-specific varlena tagging, null-bitmap emission, and MAXALIGN padding calculations.

use pgrust::heaptuple::{heap_tuple_to_disk_image, heap_copytuple_from_disk_image};
use pgrust::mcx::Mcx;
use pgrust::pgvec::PgVec;

/// Serialize a Rust `FormedTuple` to the exact PostgreSQL heap‑tuple image.
fn serialize_tuple(mcx: Mcx<'_>, tuple: &FormedTuple<'_>) -> PgResult<PgVec<'_, u8>> {
    // Returns a byte vector that can be written directly to a data file.
    heap_tuple_to_disk_image(mcx, tuple)
}

/// Deserialize a raw heap‑tuple image back into a `FormedTuple`.
fn deserialize_tuple(
    mcx: Mcx<'_>,
    raw: &[u8],
    t_self: ItemPointerData,
    t_tableoid: Oid,
) -> PgResult<FormedTuple<'_>> {
    // `t_len` is the total size of the raw image.
    let t_len = raw.len() as u32;
    heap_copytuple_from_disk_image(mcx, t_len, t_self, t_tableoid, raw)
}

Native-Endian Handling

PostgreSQL stores many values in native-endian format. When serializing transaction IDs and other header fields, pgrust uses to_ne_bytes() to ensure the on-disk representation matches the platform's native byte order.

This is implemented in the HeapTupleHeaderChoice::THeap branch within crates/backend/access/common/heaptuple/src/lib.rs, where fields like t_xmin and t_xmax are converted using native-endian byte serialization to maintain compatibility with C PostgreSQL binaries.

TOAST and Varlena Support

The Oversized Attribute Storage Technique (TOAST) requires special handling for external on-disk pointers. In crates/contrib/test_decoding/src/ondisk.rs (lines 30–36), the varatt_is_external_ondisk helper function reproduces the VARTAG_ONDISK flag behavior.

This ensures that toasted values maintain the same external identifiers and pointer structures as the C implementation, allowing pgrust to read and write TOAST data that remains compatible with existing PostgreSQL databases.

Control Files and WAL Structures

Beyond heap tuples, pgrust replicates the fixed-size binary formats for control files and write-ahead log records.

The ControlFileData structure in crates/common/controldata_utils/src/lib.rs is serialized to exactly sizeof(ControlFileData) bytes at the start of the file, matching PostgreSQL's control file layout. Similarly, crates/backend/utils/cache/relmapper/src/lib.rs handles the RelMapFile format, while crates/_support/types/wal/src/xlogutils.rs manages WAL record headers and payloads.

use pgrust::common::controldata_utils::ControlFileData;

/// Read the on‑disk `pg_control` file and parse it.
fn read_control_file(path: &Path) -> std::io::Result<ControlFileData> {
    let bytes = std::fs::read(path)?;
    // `ControlFileData::from_bytes` expects the exact byte layout.
    Ok(ControlFileData::from_bytes(&bytes))
}

Binary Verification Against C Reference Images

To prevent accidental layout drift, pgrust includes unit tests that compare Rust-generated binary output against dumps produced by C PostgreSQL. The crates/backend/utils/sort/tuplesort/src/lib.rs file contains tests at lines 2145–2150 that validate the on-disk IndexTuple image against reference data.

These tests read binary dumps produced by PostgreSQL and verify that the Rust serializer produces identical byte slices, guarding against regressions in the disk compatibility layer.

Summary

• C-compatible layouts: Using #[repr(C)] attributes ensures struct memory alignment matches PostgreSQL's C structures exactly.
• Explicit serialization: Functions like heap_tuple_to_disk_image control byte order, padding, and field placement to match heap_fill_tuple.
• Native-endian handling: The to_ne_bytes() method ensures transaction IDs and headers match platform-specific endian expectations.
• TOAST compatibility: The varatt_is_external_ondisk helper preserves external pointer formats for oversized attributes.
• Control and WAL replication: Fixed-size structures like ControlFileData and RelMapFile maintain exact binary layouts.
• Regression testing: Unit tests compare Rust output against C PostgreSQL binary dumps to verify byte-level compatibility.

Frequently Asked Questions

What is disk compatibility in the context of pgrust?

Disk compatibility means that pgrust can read and write PostgreSQL data files in the exact same binary format as the original C implementation. This allows a PostgreSQL data directory to be used interchangeably between official PostgreSQL binaries and the pgrust engine without any export, import, or conversion process.

How does pgrust handle PostgreSQL's MAXALIGN padding requirements?

The heap_tuple_to_disk_image function in crates/backend/access/common/heaptuple/src/lib.rs explicitly calculates and applies MAXALIGN padding during serialization. This ensures that tuple headers and data fields align to the same byte boundaries as the C implementation, maintaining the exact on-disk size and layout.

Can pgrust read existing PostgreSQL WAL files?

Yes, pgrust implements compatible WAL record headers in crates/_support/types/wal/src/xlogutils.rs that match PostgreSQL's format. The Rust implementation can read existing WAL files from a standard PostgreSQL installation and also write new records that the C PostgreSQL server can replay, ensuring complete interoperability in the write-ahead log.

What prevents structural drift between pgrust and PostgreSQL updates?

The project maintains unit tests in files like crates/backend/utils/sort/tuplesort/src/lib.rs that compare Rust-generated binary images against reference dumps produced by C PostgreSQL. These tests fail if any byte differs, ensuring that updates to either codebase that would affect on-disk layout are caught immediately during development.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}