What Is the LTX (Lite Transaction) System and How Does It Ensure SQLite Durability?

The LTX (Lite Transaction) system is an append-only transaction log that stores SQLite database snapshots in an immutable, checksummed binary format, providing durability through CRC64-ISO verification, rolling snapshot validation, and atomic remote replication.

The LTX (Lite Transaction) subsystem in the denoland/celld repository replaces traditional SQLite WAL files with a lightweight, deterministic format designed for cloud-native deployments. By encoding SQLite pages into immutable files with cryptographic integrity checks, this system ensures database changes remain durable across process crashes and network boundaries.

Immutable File Format and Page Storage

The foundation of LTX durability lies in its strictly append-only file structure. Each LTX file consists of a complete header, a stream of LZ4-compressed pages, and a terminal trailer. Because files in crates/ltx/src/ltx.rs are never mutated after creation—a design enforced by the storage layer—partial writes or crashes cannot corrupt previously committed data.

Header and Trailer Structure

The Header struct defined in crates/ltx/src/ltx.rs contains critical metadata including page_size, commit transaction ID, and min_txid/max_txid ranges. For snapshot files where Header.min_txid == 1, the header stores a pre_apply_checksum that creates a cryptographic chain linking every subsequent page to its predecessor.

Checksum-Based Integrity Verification

LTX implements multiple layers of checksums to detect corruption before data reaches the SQLite engine.

CRC64-ISO Page Verification

Every page written to an LTX file passes through the checksum_page function, which generates a CRC64-ISO hash. When reading via Ltx::open, the iter_pages method automatically recomputes and verifies these checksums. Any mismatch between the stored and computed checksum triggers an immediate error, preventing corrupted pages from being applied to the database.

File-Wide CRC64 Trailer

After all pages are written, the finalize() method appends an 8-byte CRC64-ISO trailer calculated over the entire file content. This protects against truncation or corruption of the file itself, not just individual pages.

Rolling Snapshot Checksums

For database snapshots, the Header::pre_apply_checksum field implements a rolling validation mechanism. As each page is appended to the snapshot, the checksum updates to incorporate the previous page's data. This creates an unbroken chain of cryptographic evidence within crates/ltx/src/ltx.rs, ensuring that any tampering or corruption in the middle of a snapshot file is immediately detectable when the header validation runs via header.validate().

Remote Replication and Durability

The Replica object in crates/ltx/src/replica.rs extends durability beyond local disk by implementing atomic replication to remote object stores like S3 or MinIO.

Atomic Upload with Validation

The Replica::sync method only uploads LTX files after they have been fully validated locally. Using the Decoder.DecodeDatabaseTo stream decoder during restoration, replicas verify the entire LTX stream checksum before considering the sync successful. Because the replica uploads immutable, finalized files rather than partial streams, remote stores always contain durable, verifiable copies of the SQLite database.

Working with LTX Files

The following examples demonstrate how to create, validate, and replicate LTX files using the Rust API.

Creating and Finalizing an LTX File

use celld::ltx::{Ltx, Header, Pos, TXID, CHECKSUM_FLAG};

let mut writer = Ltx::create("mydb.ltx")?;
let header = Header {
    version: Ltx::VERSION,
    flags: 0,
    page_size: 4096,
    commit: 1,
    min_txid: TXID(1),
    max_txid: TXID(1),
    timestamp: chrono::Utc::now().timestamp(),
    pre_apply_checksum: 0,
    wal_offset: 0,
    wal_size: 0,
    wal_salt1: 0,
    wal_salt2: 0,
    node_id: 42,
};
writer.write_header(&header)?;

// Append a single SQLite page (page 2) with a checksum.
let page_no = 2u32;
let page_data = vec![0u8; 4096]; // normally read from SQLite file.
let checksum = celld::ltx::checksum_page(page_no, &page_data);
writer.write_page(page_no, &page_data, checksum)?;

// Finalize – writes the trailer with a file‑wide CRC64.
writer.finalize()?;

Reading and Validating LTX Files

use celld::ltx::Ltx;

let mut reader = Ltx::open("mydb.ltx")?;
let header = reader.read_header()?;               // validates magic & version
assert!(header.validate().is_ok());               // checks invariants

// Iterate pages, each page's checksum is verified automatically.
for (pgno, data, checksum) in reader.iter_pages()? {
    // `checksum` includes the CRC64 flag; any mismatch will raise an error.
    process_page(pgno, data);
}

Replicating to Remote Storage

use celld::ltx::replica::{Replica, ReplicaClient};
use celld::ltx::client::File; // simple file‑based client for demo

let client = File::new("./remote_store");
let mut replica = Replica::new_client_only(client);
replica.seed_pos(Pos::ZERO); // start from a known durable position

// Perform a one‑shot sync – uploads every new LTX file.
tokio::runtime::Runtime::new()?.block_on(async {
    replica.sync().await.expect("sync failed");
});

Core Source Files

The LTX (Lite Transaction) implementation spans several key modules in the denoland/celld codebase:

• crates/ltx/src/ltx.rs – Core file format implementing Header, Trailer, CRC64-ISO hasher, and the checksum_page function.
• crates/ltx/src/wal.rs – WAL-style page handling and LZ4 compression framing.
• crates/ltx/src/store.rs – Low-level storage abstraction for atomic file operations.
• crates/ltx/src/replica.rs – Replication engine with Replica::sync and restore orchestration via Decoder.DecodeDatabaseTo.
• crates/ltx/src/client/mod.rs – Trait definitions for remote storage clients (S3, file, etc.).
• crates/ltx/src/db.rs – Integration with SQLite for snapshot creation and database restoration.

Summary

• Immutable files: LTX files are write-once in crates/ltx/src/ltx.rs, preventing corruption of historical data during crashes.
• Cryptographic integrity: CRC64-ISO checksums at the page level (checksum_page) and file level (finalize) detect any bit rot or partial writes.
• Chain verification: The pre_apply_checksum header field creates a cryptographic chain for snapshots, ensuring every page validates against its predecessors.
• Remote durability: The Replica object in crates/ltx/src/replica.rs uploads only validated LTX files, extending durability to object stores like S3.
• LZ4 compression: Pages are compressed before checksumming, optimizing storage without sacrificing integrity verification.

Frequently Asked Questions

What does LTX stand for in the celld project?

LTX stands for Lite Transaction. It represents a lightweight alternative to SQLite's native WAL (Write-Ahead Log) format, specifically designed for serializing database snapshots and incremental changes in a streamable, cloud-native binary format.

How does LTX detect data corruption during a system crash?

LTX employs multiple detection layers: individual pages are protected by CRC64-ISO checksums computed via checksum_page, while the entire file is protected by a terminal trailer checksum written during finalize(). Because files are immutable, any partial write from a crash is detectable as a missing or mismatched trailer, while iter_pages() validates each page's checksum on read in crates/ltx/src/ltx.rs.

Can LTX files be replicated to S3 or MinIO?

Yes. The Replica struct in crates/ltx/src/replica.rs implements sync() to upload validated LTX files to any object store implementing the client trait defined in crates/ltx/src/client/mod.rs. The format's immutability makes it ideal for object storage, as files are only uploaded once and never modified.

What compression does the LTX format use?

LTX uses LZ4 compression for page data, implemented in crates/ltx/src/wal.rs. Pages are compressed before checksum calculation, ensuring that integrity verification covers the compressed payload while minimizing storage overhead for remote replication.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}