# What Is the LTX Replication Format? SQLite-to-S3 Streaming in celld Explained

> Explore the LTX replication format celld uses for efficient SQLite to S3 streaming. Understand its binary transaction log, header, compressed blocks, and index for robust data replication.

- Repository: [Deno/celld](https://github.com/denoland/celld)
- Tags: deep-dive
- Published: 2026-08-10

---

**The LTX (Litestream Transaction) format is a binary transaction log encoding used by celld to stream SQLite WAL changes to S3-compatible object stores, consisting of a 100-byte header, LZ4-compressed page blocks, a varint page index, and a 16-byte CRC-64-ISO trailer.**

celld, developed by Deno, implements continuous SQLite replication by capturing committed transactions from the Write-Ahead Log (WAL) and encoding them into LTX files. These files provide byte-compatible interoperability with the reference Litestream v0.5 implementation, enabling reliable disaster recovery and read-replica scaling across S3, Cloudflare R2, and MinIO.

## LTX File Structure and Binary Layout

The on-disk LTX format defined in [`crates/ltx/reference/ltx-format.md`](https://github.com/denoland/celld/blob/main/crates/ltx/reference/ltx-format.md) organizes transaction data into four logical sections. All multi-byte integers are **big-endian**.

### Header Section (100 Bytes)

The fixed-size header begins with magic bytes `LTX1` and contains metadata required for validation and restoration:

- **Magic bytes**: `LTX1` (4 bytes)
- **Flags**: Format compatibility indicators
- **Page size**: SQLite page size (e.g., 4096)
- **Commit size**: Number of pages in the transaction
- **Transaction range**: Minimum and maximum TXID (16 bytes total)
- **Timestamp**: Unix timestamp in nanoseconds
- **WAL offsets**: Salt and offset information for WAL validation
- **Node ID**: 20-byte unique identifier for the generating node
- **Reserved**: Padding to reach exactly 100 bytes

This header is implemented in the encoding logic within [`crates/ltx/src/ltx.rs`](https://github.com/denoland/celld/blob/main/crates/ltx/src/ltx.rs).

### Page Block (LZ4 Compressed)

Following the header, the **page block** contains one or more database pages:

- Each page is prefixed by a 6-byte `PageHeader` comprising the page number (`pgno`, 4 bytes) and flags (2 bytes).
- Page data is compressed as an independent **LZ4** frame.
- The block terminates with an empty `PageHeader` where `pgno == 0`.

This compression strategy minimizes storage costs while maintaining single-page decompression capability during random access restores.

### Page Index (Varint Encoded)

The page index provides random access mappings without requiring full file scans:

- Encoded as **varint** tuples: `(pgno, offset, size)` for every page in ascending order.
- Terminated by a zero `pgno` value.
- Followed by a mandatory **64-bit size field** indicating the total length of the index region.

This structure allows `Replica::restore` in [`crates/ltx/src/replica.rs`](https://github.com/denoland/celld/blob/main/crates/ltx/src/replica.rs) to efficiently locate specific pages when rebuilding a database from incremental LTX files.

### Trailer (16 Bytes)

The file concludes with cryptographic checksums using **CRC-64-ISO** with the high-bit flag set:

- `PostApplyChecksum`: Database state hash after applying this transaction
- `FileChecksum`: Entire file integrity verification

This checksum strategy protects against corruption during upload to or download from object storage.

## Transaction Capture and Replication Flow

celld replicates SQLite through three distinct phases implemented across the `celld-ltx` crate.

### WAL Monitoring and Capture

The `Db` wrapper in [`crates/ltx/src/db.rs`](https://github.com/denoland/celld/blob/main/crates/ltx/src/db.rs) manages the SQLite connection and WAL observation:

- `Db::capture_loop` continuously monitors the WAL for committed transactions.
- For each commit, it generates an **L0 LTX file** in a local capture directory.
- Filename follows the convention `<minTXID:016x>-<maxTXID:016x>.ltx` (e.g., `0000000000000001-0000000000000001.ltx`).

When `MinTXID == 1`, the file represents a **snapshot** containing a full database copy with `PreApplyChecksum` set to zero. Incremental files contain only changed pages.

### Upload to S3-Compatible Storage

The `Replica<C>` struct in [`crates/ltx/src/replica.rs`](https://github.com/denoland/celld/blob/main/crates/ltx/src/replica.rs) handles synchronization:

- `Replica::sync` iterates over new L0 files in the capture directory.
- It uploads each file via the configured `ReplicaClient` implementation.
- For S3 destinations, `ObjectStoreClient` in [`crates/ltx/src/client/object_store.rs`](https://github.com/denoland/celld/blob/main/crates/ltx/src/client/object_store.rs) provides the concrete implementation.

The upload process guarantees **byte-identical** output compatible with the Go reference implementation, verified by conformance tests such as [`differential_xtool.rs`](https://github.com/denoland/celld/blob/main/differential_xtool.rs).

### Database Restoration

During startup or disaster recovery:

- `Replica::restore` downloads required LTX files from the bucket.
- It concatenates files in transaction order.
- Applies them to a fresh SQLite database file, ensuring logical state consistency with the source.

## S3 Integration and Client Configuration

### URL Parsing and Authentication

The [`replica_url.rs`](https://github.com/denoland/celld/blob/main/replica_url.rs) module parses S3 URLs with query-string configuration:

- Supports `s3://` scheme with parameters: `region`, `endpoint`, `force-path-style`, `skip-verify`.
- Constructs an `object_store::Client` implementing the five `ReplicaClient` methods: `list`, `download`, `upload`, `delete`, and `head`.

This mirrors the Go Litestream `NewReplicaClientFromURL` behavior for configuration compatibility.

### Object Store Abstraction

`ObjectStoreClient` abstracts S3, R2, and MinIO behind the `ReplicaClient` trait:

- Lists existing LTX files to determine replication lag.
- Streams uploads directly from the capture directory.
- Validates downloaded files using the trailer checksums before application.

## Practical Implementation Example

The following Rust example demonstrates configuring SQLite-to-S3 replication using celld's public API:

```rust
use celld_ltx::{Db, Replica};
use celld_ltx::client::object_store::ObjectStoreClient;
use std::path::Path;

// 1. Open the source database and initialize WAL capture
let db = Db::open_path("local.db")?;
let capture_dir = Path::new("/tmp/celld-capture");

// 2. Configure S3 replica client with endpoint options
let replica = ObjectStoreClient::new_from_url(
    "s3://my-bucket/celld-replica?region=us-east-1&force-path-style=true"
)?;

// 3. Initialize replication engine and sync to S3
let mut repl = Replica::new(db, replica, capture_dir.into());
repl.sync().await?;  // Uploads pending LTX files

// 4. Restore database on a new node or after failure
let restored_path = Path::new("/tmp/restored.db");
let restored_db = Replica::restore(&replica, restored_path, 0).await?;

```

Uploaded files appear in the S3 bucket with hexadecimal transaction range filenames, ready for incremental restoration or Litestream-compatible recovery.

## Summary

- **LTX format** encodes SQLite WAL transactions into four sections: a 100-byte header, LZ4-compressed pages, a varint page index, and a CRC-64-ISO trailer.
- **Capture process** uses `Db::capture_loop` in [`crates/ltx/src/db.rs`](https://github.com/denoland/celld/blob/main/crates/ltx/src/db.rs) to generate snapshot or incremental files locally.
- **S3 replication** leverages `Replica::sync` and `ObjectStoreClient` to upload files while maintaining byte-compatibility with Litestream v0.5.
- **Restoration** via `Replica::restore` downloads and applies LTX files in transaction order to reconstruct database state.
- **Checksum strategy** employs CRC-64-ISO with high-bit flag for end-to-end integrity verification.

## Frequently Asked Questions

### What does LTX stand for in celld replication?

LTX stands for **Litestream Transaction**. It is the binary file format originally defined by the Litestream project (v0.5) that celld implements in Rust to provide continuous SQLite replication to object storage.

### How does celld ensure data integrity during S3 uploads?

celld uses **CRC-64-ISO checksums** with the high-bit flag set in the LTX trailer section, matching the Go reference implementation. The `FileChecksum` validates the entire file after download, while `PostApplyChecksum` verifies the logical database state, protecting against both transport corruption and bit-rot in object storage.

### Can celld restore databases from standard Litestream backups?

Yes. According to the conformance tests in the celld source code (such as [`differential_xtool.rs`](https://github.com/denoland/celld/blob/main/differential_xtool.rs)), the LTX encoder in [`crates/ltx/src/ltx.rs`](https://github.com/denoland/celld/blob/main/crates/ltx/src/ltx.rs) produces **byte-identical** output to the Go Litestream v0.5 implementation. This ensures that LTX files written by celld can be read by Litestream, and vice versa, enabling hybrid Rust/Go infrastructure.

### What is the difference between snapshot and incremental LTX files?

A **snapshot** LTX file (where `MinTXID == 1`) contains a complete copy of the database with `PreApplyChecksum` set to zero. **Incremental** files contain only the pages modified since the previous transaction. During restoration, celld applies snapshots first, then subsequent incremental files in transaction ID order to reach the current state.