What Is the LTX Replication Format? SQLite-to-S3 Streaming in celld Explained

The LTX (Litestream Transaction) format is a binary transaction log encoding used by celld to stream SQLite WAL changes to S3-compatible object stores, consisting of a 100-byte header, LZ4-compressed page blocks, a varint page index, and a 16-byte CRC-64-ISO trailer.

celld, developed by Deno, implements continuous SQLite replication by capturing committed transactions from the Write-Ahead Log (WAL) and encoding them into LTX files. These files provide byte-compatible interoperability with the reference Litestream v0.5 implementation, enabling reliable disaster recovery and read-replica scaling across S3, Cloudflare R2, and MinIO.

LTX File Structure and Binary Layout

The on-disk LTX format defined in crates/ltx/reference/ltx-format.md organizes transaction data into four logical sections. All multi-byte integers are big-endian.

Header Section (100 Bytes)

The fixed-size header begins with magic bytes LTX1 and contains metadata required for validation and restoration:

• Magic bytes: LTX1 (4 bytes)
• Flags: Format compatibility indicators
• Page size: SQLite page size (e.g., 4096)
• Commit size: Number of pages in the transaction
• Transaction range: Minimum and maximum TXID (16 bytes total)
• Timestamp: Unix timestamp in nanoseconds
• WAL offsets: Salt and offset information for WAL validation
• Node ID: 20-byte unique identifier for the generating node
• Reserved: Padding to reach exactly 100 bytes

This header is implemented in the encoding logic within crates/ltx/src/ltx.rs.

Page Block (LZ4 Compressed)

Following the header, the page block contains one or more database pages:

• Each page is prefixed by a 6-byte PageHeader comprising the page number (pgno, 4 bytes) and flags (2 bytes).
• Page data is compressed as an independent LZ4 frame.
• The block terminates with an empty PageHeader where pgno == 0.

This compression strategy minimizes storage costs while maintaining single-page decompression capability during random access restores.

Page Index (Varint Encoded)

The page index provides random access mappings without requiring full file scans:

• Encoded as varint tuples: (pgno, offset, size) for every page in ascending order.
• Terminated by a zero pgno value.
• Followed by a mandatory 64-bit size field indicating the total length of the index region.

This structure allows Replica::restore in crates/ltx/src/replica.rs to efficiently locate specific pages when rebuilding a database from incremental LTX files.

Trailer (16 Bytes)

The file concludes with cryptographic checksums using CRC-64-ISO with the high-bit flag set:

• PostApplyChecksum: Database state hash after applying this transaction
• FileChecksum: Entire file integrity verification

This checksum strategy protects against corruption during upload to or download from object storage.

Transaction Capture and Replication Flow

celld replicates SQLite through three distinct phases implemented across the celld-ltx crate.

WAL Monitoring and Capture

The Db wrapper in crates/ltx/src/db.rs manages the SQLite connection and WAL observation:

• Db::capture_loop continuously monitors the WAL for committed transactions.
• For each commit, it generates an L0 LTX file in a local capture directory.
• Filename follows the convention <minTXID:016x>-<maxTXID:016x>.ltx (e.g., 0000000000000001-0000000000000001.ltx).

When MinTXID == 1, the file represents a snapshot containing a full database copy with PreApplyChecksum set to zero. Incremental files contain only changed pages.

Upload to S3-Compatible Storage

The Replica<C> struct in crates/ltx/src/replica.rs handles synchronization:

• Replica::sync iterates over new L0 files in the capture directory.
• It uploads each file via the configured ReplicaClient implementation.
• For S3 destinations, ObjectStoreClient in crates/ltx/src/client/object_store.rs provides the concrete implementation.

The upload process guarantees byte-identical output compatible with the Go reference implementation, verified by conformance tests such as differential_xtool.rs.

Database Restoration

During startup or disaster recovery:

• Replica::restore downloads required LTX files from the bucket.
• It concatenates files in transaction order.
• Applies them to a fresh SQLite database file, ensuring logical state consistency with the source.

S3 Integration and Client Configuration

URL Parsing and Authentication

The replica_url.rs module parses S3 URLs with query-string configuration:

• Supports s3:// scheme with parameters: region, endpoint, force-path-style, skip-verify.
• Constructs an object_store::Client implementing the five ReplicaClient methods: list, download, upload, delete, and head.

This mirrors the Go Litestream NewReplicaClientFromURL behavior for configuration compatibility.

Object Store Abstraction

ObjectStoreClient abstracts S3, R2, and MinIO behind the ReplicaClient trait:

• Lists existing LTX files to determine replication lag.
• Streams uploads directly from the capture directory.
• Validates downloaded files using the trailer checksums before application.

Practical Implementation Example

The following Rust example demonstrates configuring SQLite-to-S3 replication using celld's public API:

use celld_ltx::{Db, Replica};
use celld_ltx::client::object_store::ObjectStoreClient;
use std::path::Path;

// 1. Open the source database and initialize WAL capture
let db = Db::open_path("local.db")?;
let capture_dir = Path::new("/tmp/celld-capture");

// 2. Configure S3 replica client with endpoint options
let replica = ObjectStoreClient::new_from_url(
    "s3://my-bucket/celld-replica?region=us-east-1&force-path-style=true"
)?;

// 3. Initialize replication engine and sync to S3
let mut repl = Replica::new(db, replica, capture_dir.into());
repl.sync().await?;  // Uploads pending LTX files

// 4. Restore database on a new node or after failure
let restored_path = Path::new("/tmp/restored.db");
let restored_db = Replica::restore(&replica, restored_path, 0).await?;

Uploaded files appear in the S3 bucket with hexadecimal transaction range filenames, ready for incremental restoration or Litestream-compatible recovery.

Summary

• LTX format encodes SQLite WAL transactions into four sections: a 100-byte header, LZ4-compressed pages, a varint page index, and a CRC-64-ISO trailer.
• Capture process uses Db::capture_loop in crates/ltx/src/db.rs to generate snapshot or incremental files locally.
• S3 replication leverages Replica::sync and ObjectStoreClient to upload files while maintaining byte-compatibility with Litestream v0.5.
• Restoration via Replica::restore downloads and applies LTX files in transaction order to reconstruct database state.
• Checksum strategy employs CRC-64-ISO with high-bit flag for end-to-end integrity verification.

Frequently Asked Questions

What does LTX stand for in celld replication?

LTX stands for Litestream Transaction. It is the binary file format originally defined by the Litestream project (v0.5) that celld implements in Rust to provide continuous SQLite replication to object storage.

How does celld ensure data integrity during S3 uploads?

celld uses CRC-64-ISO checksums with the high-bit flag set in the LTX trailer section, matching the Go reference implementation. The FileChecksum validates the entire file after download, while PostApplyChecksum verifies the logical database state, protecting against both transport corruption and bit-rot in object storage.

Can celld restore databases from standard Litestream backups?

Yes. According to the conformance tests in the celld source code (such as differential_xtool.rs), the LTX encoder in crates/ltx/src/ltx.rs produces byte-identical output to the Go Litestream v0.5 implementation. This ensures that LTX files written by celld can be read by Litestream, and vice versa, enabling hybrid Rust/Go infrastructure.

What is the difference between snapshot and incremental LTX files?

A snapshot LTX file (where MinTXID == 1) contains a complete copy of the database with PreApplyChecksum set to zero. Incremental files contain only the pages modified since the previous transaction. During restoration, celld applies snapshots first, then subsequent incremental files in transaction ID order to reach the current state.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}