How celld Continuously Replicates SQLite Databases to S3-Compatible Storage

celld captures every SQLite write as LTX (Logical Transaction) files and uploads them to S3-compatible storage using a continuous Tokio-backed sync loop, enabling fault-tolerant, incremental replication.

The denoland/celld project implements a write-once-read-many replication model that streams SQLite transactions to object storage without interrupting the live database. By combining SQLite’s backup API with an LTX file format compatible with Litestream, celld ensures that any S3-compatible bucket—including AWS S3, MinIO, or Cloudflare R2—can serve as a durable backend for continuous database replication.

The LTX Replication Architecture

celld’s replication pipeline centers on three coordinated components: a managed database handle that captures transactions, a replica client that interfaces with remote storage, and a background sync loop that bridges the two.

Capturing Writes as LTX Files

Every write operation is persisted locally before reaching the cloud. In crates/celld/replication.rs, the sqlite_snapshot function creates a transient read-only copy of the live database using SQLite’s backup API. This snapshot does not interfere with the WAL (Write-Ahead Log), allowing concurrent transactions to proceed uninterrupted.

The capture logic writes each committed transaction as an L0-level LTX file into a local directory structured as <watch>/<cell>/ltx/e<epoch>/. These files represent logical transactions (LTX) that are immutable once written. The crate::db::Db type manages this process, ensuring that the local LTX log grows sequentially by TXID (Transaction ID).

The Sync Loop

The Replica<C> type in crates/ltx/src/replica.rs owns both the Db instance and a ReplicaClient implementation. Its sync() method drives the continuous replication:

1. List new L0 LTX files in the local watch directory.
2. Upload via client.write_ltx_file() for each new TXID.
3. Advance the replica position atomically after each successful upload.

If the sync encounters errors indicating the replica has diverged from the remote state, the implementation resets the cached position. This forces the next sync iteration to recompute the correct TXID, ensuring idempotent recovery without duplicate data. The sync loop runs inside a Tokio task spawned during cell activation, executing every five seconds for the lifetime of the process.

Restore and Recovery

When a cell restarts or crash recovery is required, Replica::restore() in crates/ltx/src/replica.rs orchestrates the download sequence. The method calculates a restore plan (calc_restore_plan), downloads LTX files from the bucket in ascending TXID order, and merges them using the Litestream compactor. The reconstructed database is written to a temporary file, fsync-ed to disk, and atomically renamed to the target path, guaranteeing a consistent state even if the process terminates mid-restore.

S3-Compatible Backend Implementation

The S3 backend is implemented in crates/ltx/src/client/object_store.rs using the object_store crate, which provides a unified interface for AWS S3, MinIO, Cloudflare R2, and other compatible services.

Object Store Integration

The ObjectStoreClient struct implements the ReplicaClient trait defined in crates/ltx/src/client/mod.rs. It maps five core operations—listing, reading, writing, deleting, and checking existence—onto the object_store API. Configuration parsing occurs via ObjectStoreConfig::from_url(), which accepts standard S3 URLs:

let url = "s3://my-bucket/celld-db?region=us-west-2".parse::<celld::replica_url::ParsedReplicaUrl>()?;
let cfg = ObjectStoreConfig::from_url(&url)?;
let client = ObjectStoreClient::new(cfg).await?;

Multipart Uploads and Metadata

For objects larger than 5 MiB, the backend automatically initiates multipart uploads to optimize throughput and reliability. The implementation also supports batch delete operations (up to 1,000 keys per request) for cleanup tasks. LTX files are stored using the Litestream key naming convention:

{path}/{level:04x}/{min}-{max}.ltx

The original LTX header timestamp is preserved in the litestream-timestamp metadata header, ensuring temporal consistency across restores.

Implementation Example

To configure continuous replication, bind a Db instance to a Replica backed by an S3-compatible client and spawn the sync loop:

use celld::db::Db;
use celld::ltx::client::object_store::{ObjectStoreClient, ObjectStoreConfig};
use celld::ltx::replica::{Replica, Pos};

// Parse the replica URL and build the S3 client
let url = "s3://my-bucket/celld-db?region=us-west-2".parse().expect("valid URL");
let cfg = ObjectStoreConfig::from_url(&url).expect("valid config");
let s3_client = ObjectStoreClient::new(cfg).await.expect("S3 client created");

// Open the managed SQLite database
let db = Db::open("./watch/mycell/ltx/e1/db.sqlite")
    .await
    .expect("opened DB");

// Create the replica and seed initial position
let mut replica = Replica::new(db, s3_client);
replica.seed_pos(Pos::ZERO);

// Spawn the continuous replication loop
tokio::spawn(async move {
    loop {
        if let Err(e) = replica.sync().await {
            eprintln!("replication error: {e}");
        }
        tokio::time::sleep(std::time::Duration::from_secs(5)).await;
    }
});

To force a restore from remote storage—useful after hardware failure or when migrating a cell to new infrastructure—use the client-only replica constructor:

use celld::ltx::replica::Replica;

let mut restore_replica = Replica::new_client_only(s3_client);
let restored_path = std::path::Path::new("./restored/db.sqlite");

restore_replica.restore(&restored_path)
    .await
    .expect("restore succeeded");

This downloads all LTX files in order, compacts them, and writes the final database to the specified path.

Summary

• LTX File Format: celld writes every SQLite transaction to immutable L0 LTX files locally before uploading, creating a durable append-only log.
• Continuous Sync: A Tokio task in crates/celld/startup.rs runs Replica::sync() every five seconds, uploading new files to S3-compatible storage and advancing the TXID position.
• Fault Tolerance: Divergence errors reset the replica position automatically, while restore operations download and compact LTX files into a consistent SQLite database.
• S3 Compatibility: The object_store backend in crates/ltx/src/client/object_store.rs supports multipart uploads, batch deletes, and standard S3 URL parsing for AWS, MinIO, and R2.

Frequently Asked Questions

What happens if the sync loop crashes mid-upload?

celld treats LTX files as immutable and uploads are idempotent. If Replica::sync() fails, the replica position is not advanced for that TXID. On the next iteration, the loop retries the same file, overwriting any partial object in S3. The position tracking logic in crates/ltx/src/replica.rs ensures that only fully uploaded and acknowledged transactions are considered complete.

Can celld replicate to MinIO or Cloudflare R2 instead of AWS S3?

Yes. The ObjectStoreClient in crates/ltx/src/client/object_store.rs uses the generic object_store crate, which abstracts S3-compatible APIs. You can configure alternative endpoints via the replica URL (e.g., s3://bucket/path?endpoint=http://localhost:9000&region=us-east-1) and the ObjectStoreConfig::from_url() parser will handle the connection.

How does celld handle large database files during restore?

The restore process streams LTX files from S3 using byte-range requests, processing them through the Litestream compactor without loading the entire database into memory. The final SQLite file is reconstructed incrementally and atomically renamed into place only after all TXIDs are applied and fsync confirms durability.

What consistency guarantees does celld provide?

celld provides eventual consistency with a bounded staleness of approximately five seconds (the default sync interval). The replication model is write-once-read-many: once an LTX file is uploaded to S3, it becomes the source of truth for that TXID. Concurrent readers restoring from the bucket will see a consistent snapshot up to the highest contiguous TXID available in the object store.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}