# How pgrust Achieves Disk Compatibility with PostgreSQL: Byte-Level Replication Techniques

> Discover how pgrust achieves disk compatibility with PostgreSQL using byte-level replication of C structs, serializers, and native endian handling for seamless data interchange between Rust and C instances.

- Repository: [Michael Malis/pgrust](https://github.com/malisper/pgrust)
- Tags: internals
- Published: 2026-07-13

---

**pgrust achieves disk compatibility with PostgreSQL by implementing C-compatible struct layouts, explicit byte-for-byte serializers, and native-endian handling that replicate every on-disk structure from heap tuples to WAL records, allowing seamless interchangeability between Rust and C PostgreSQL instances.**

pgrust is a Rust re-implementation of PostgreSQL's core engine designed to maintain full binary interchangeability with standard PostgreSQL data directories. To achieve disk compatibility with PostgreSQL, the project meticulously reproduces identical byte layouts for heap tuples, TOAST objects, control files, and WAL records. This ensures that a PostgreSQL data directory created by official binaries can be opened by a pgrust server without any conversion or migration step.

## C-Compatible Struct Definitions

The foundation of disk compatibility lies in using `#[repr(C)]` attributes to ensure Rust structs match the memory layout, padding, and alignment of PostgreSQL's C structures exactly.

In [`crates/backend/access/common/heaptuple/src/lib.rs`](https://github.com/malisper/pgrust/blob/main/crates/backend/access/common/heaptuple/src/lib.rs), the `HeapTupleHeaderChoice` struct is defined with C-compatible layout at line 25. This guarantees that the header fields—such as transaction IDs and flags—occupy the exact same byte offsets as they do in the C implementation.

## Explicit Serialization Functions

While struct layout ensures memory alignment, explicit serialization functions control the exact byte order written to disk. The `heap_tuple_to_disk_image` and `heap_copytuple_from_disk_image` functions in [`crates/backend/access/common/heaptuple/src/lib.rs`](https://github.com/malisper/pgrust/blob/main/crates/backend/access/common/heaptuple/src/lib.rs) (lines 2503–2510 and 2549–2562) handle the precise serialization logic.

These functions write fields in the exact order and size that PostgreSQL's `heap_fill_tuple` and `heap_copytuple` expect, including PostgreSQL-specific varlena tagging, null-bitmap emission, and MAXALIGN padding calculations.

```rust
use pgrust::heaptuple::{heap_tuple_to_disk_image, heap_copytuple_from_disk_image};
use pgrust::mcx::Mcx;
use pgrust::pgvec::PgVec;

/// Serialize a Rust `FormedTuple` to the exact PostgreSQL heap‑tuple image.
fn serialize_tuple(mcx: Mcx<'_>, tuple: &FormedTuple<'_>) -> PgResult<PgVec<'_, u8>> {
    // Returns a byte vector that can be written directly to a data file.
    heap_tuple_to_disk_image(mcx, tuple)
}

/// Deserialize a raw heap‑tuple image back into a `FormedTuple`.
fn deserialize_tuple(
    mcx: Mcx<'_>,
    raw: &[u8],
    t_self: ItemPointerData,
    t_tableoid: Oid,
) -> PgResult<FormedTuple<'_>> {
    // `t_len` is the total size of the raw image.
    let t_len = raw.len() as u32;
    heap_copytuple_from_disk_image(mcx, t_len, t_self, t_tableoid, raw)
}

```

## Native-Endian Handling

PostgreSQL stores many values in native-endian format. When serializing transaction IDs and other header fields, pgrust uses `to_ne_bytes()` to ensure the on-disk representation matches the platform's native byte order.

This is implemented in the `HeapTupleHeaderChoice::THeap` branch within [`crates/backend/access/common/heaptuple/src/lib.rs`](https://github.com/malisper/pgrust/blob/main/crates/backend/access/common/heaptuple/src/lib.rs), where fields like `t_xmin` and `t_xmax` are converted using native-endian byte serialization to maintain compatibility with C PostgreSQL binaries.

## TOAST and Varlena Support

The Oversized Attribute Storage Technique (TOAST) requires special handling for external on-disk pointers. In [`crates/contrib/test_decoding/src/ondisk.rs`](https://github.com/malisper/pgrust/blob/main/crates/contrib/test_decoding/src/ondisk.rs) (lines 30–36), the `varatt_is_external_ondisk` helper function reproduces the `VARTAG_ONDISK` flag behavior.

This ensures that toasted values maintain the same external identifiers and pointer structures as the C implementation, allowing pgrust to read and write TOAST data that remains compatible with existing PostgreSQL databases.

## Control Files and WAL Structures

Beyond heap tuples, pgrust replicates the fixed-size binary formats for control files and write-ahead log records.

The `ControlFileData` structure in [`crates/common/controldata_utils/src/lib.rs`](https://github.com/malisper/pgrust/blob/main/crates/common/controldata_utils/src/lib.rs) is serialized to exactly `sizeof(ControlFileData)` bytes at the start of the file, matching PostgreSQL's control file layout. Similarly, [`crates/backend/utils/cache/relmapper/src/lib.rs`](https://github.com/malisper/pgrust/blob/main/crates/backend/utils/cache/relmapper/src/lib.rs) handles the `RelMapFile` format, while [`crates/_support/types/wal/src/xlogutils.rs`](https://github.com/malisper/pgrust/blob/main/crates/_support/types/wal/src/xlogutils.rs) manages WAL record headers and payloads.

```rust
use pgrust::common::controldata_utils::ControlFileData;

/// Read the on‑disk `pg_control` file and parse it.
fn read_control_file(path: &Path) -> std::io::Result<ControlFileData> {
    let bytes = std::fs::read(path)?;
    // `ControlFileData::from_bytes` expects the exact byte layout.
    Ok(ControlFileData::from_bytes(&bytes))
}

```

## Binary Verification Against C Reference Images

To prevent accidental layout drift, pgrust includes unit tests that compare Rust-generated binary output against dumps produced by C PostgreSQL. The [`crates/backend/utils/sort/tuplesort/src/lib.rs`](https://github.com/malisper/pgrust/blob/main/crates/backend/utils/sort/tuplesort/src/lib.rs) file contains tests at lines 2145–2150 that validate the on-disk `IndexTuple` image against reference data.

These tests read binary dumps produced by PostgreSQL and verify that the Rust serializer produces identical byte slices, guarding against regressions in the disk compatibility layer.

## Summary

- **C-compatible layouts**: Using `#[repr(C)]` attributes ensures struct memory alignment matches PostgreSQL's C structures exactly.
- **Explicit serialization**: Functions like `heap_tuple_to_disk_image` control byte order, padding, and field placement to match `heap_fill_tuple`.
- **Native-endian handling**: The `to_ne_bytes()` method ensures transaction IDs and headers match platform-specific endian expectations.
- **TOAST compatibility**: The `varatt_is_external_ondisk` helper preserves external pointer formats for oversized attributes.
- **Control and WAL replication**: Fixed-size structures like `ControlFileData` and `RelMapFile` maintain exact binary layouts.
- **Regression testing**: Unit tests compare Rust output against C PostgreSQL binary dumps to verify byte-level compatibility.

## Frequently Asked Questions

### What is disk compatibility in the context of pgrust?

Disk compatibility means that pgrust can read and write PostgreSQL data files in the exact same binary format as the original C implementation. This allows a PostgreSQL data directory to be used interchangeably between official PostgreSQL binaries and the pgrust engine without any export, import, or conversion process.

### How does pgrust handle PostgreSQL's MAXALIGN padding requirements?

The `heap_tuple_to_disk_image` function in [`crates/backend/access/common/heaptuple/src/lib.rs`](https://github.com/malisper/pgrust/blob/main/crates/backend/access/common/heaptuple/src/lib.rs) explicitly calculates and applies MAXALIGN padding during serialization. This ensures that tuple headers and data fields align to the same byte boundaries as the C implementation, maintaining the exact on-disk size and layout.

### Can pgrust read existing PostgreSQL WAL files?

Yes, pgrust implements compatible WAL record headers in [`crates/_support/types/wal/src/xlogutils.rs`](https://github.com/malisper/pgrust/blob/main/crates/_support/types/wal/src/xlogutils.rs) that match PostgreSQL's format. The Rust implementation can read existing WAL files from a standard PostgreSQL installation and also write new records that the C PostgreSQL server can replay, ensuring complete interoperability in the write-ahead log.

### What prevents structural drift between pgrust and PostgreSQL updates?

The project maintains unit tests in files like [`crates/backend/utils/sort/tuplesort/src/lib.rs`](https://github.com/malisper/pgrust/blob/main/crates/backend/utils/sort/tuplesort/src/lib.rs) that compare Rust-generated binary images against reference dumps produced by C PostgreSQL. These tests fail if any byte differs, ensuring that updates to either codebase that would affect on-disk layout are caught immediately during development.