# How celld Continuously Replicates SQLite Databases to S3-Compatible Storage

> celld continuously replicates SQLite to S3 compatible storage by capturing writes as LTX files and uploading them with a Tokio-backed sync loop for fault-tolerant, incremental replication.

- Repository: [Deno/celld](https://github.com/denoland/celld)
- Tags: how-to-guide
- Published: 2026-08-09

---

**celld captures every SQLite write as LTX (Logical Transaction) files and uploads them to S3-compatible storage using a continuous Tokio-backed sync loop, enabling fault-tolerant, incremental replication.**

The **denoland/celld** project implements a write-once-read-many replication model that streams SQLite transactions to object storage without interrupting the live database. By combining SQLite’s backup API with an LTX file format compatible with Litestream, celld ensures that any S3-compatible bucket—including AWS S3, MinIO, or Cloudflare R2—can serve as a durable backend for continuous database replication.

## The LTX Replication Architecture

celld’s replication pipeline centers on three coordinated components: a managed database handle that captures transactions, a replica client that interfaces with remote storage, and a background sync loop that bridges the two.

### Capturing Writes as LTX Files

Every write operation is persisted locally before reaching the cloud. In [`crates/celld/replication.rs`](https://github.com/denoland/celld/blob/main/crates/celld/replication.rs), the `sqlite_snapshot` function creates a transient read-only copy of the live database using SQLite’s backup API. This snapshot does not interfere with the WAL (Write-Ahead Log), allowing concurrent transactions to proceed uninterrupted.

The capture logic writes each committed transaction as an **L0-level LTX file** into a local directory structured as `<watch>/<cell>/ltx/e<epoch>/`. These files represent logical transactions (LTX) that are immutable once written. The `crate::db::Db` type manages this process, ensuring that the local LTX log grows sequentially by TXID (Transaction ID).

### The Sync Loop

The `Replica<C>` type in [`crates/ltx/src/replica.rs`](https://github.com/denoland/celld/blob/main/crates/ltx/src/replica.rs) owns both the `Db` instance and a `ReplicaClient` implementation. Its `sync()` method drives the continuous replication:

1. **List new L0 LTX files** in the local watch directory.
2. **Upload via `client.write_ltx_file()`** for each new TXID.
3. **Advance the replica position** atomically after each successful upload.

If the sync encounters errors indicating the replica has diverged from the remote state, the implementation resets the cached position. This forces the next sync iteration to recompute the correct TXID, ensuring idempotent recovery without duplicate data. The sync loop runs inside a Tokio task spawned during cell activation, executing every five seconds for the lifetime of the process.

### Restore and Recovery

When a cell restarts or crash recovery is required, `Replica::restore()` in [`crates/ltx/src/replica.rs`](https://github.com/denoland/celld/blob/main/crates/ltx/src/replica.rs) orchestrates the download sequence. The method calculates a restore plan (`calc_restore_plan`), downloads LTX files from the bucket in ascending TXID order, and merges them using the Litestream compactor. The reconstructed database is written to a temporary file, `fsync`-ed to disk, and atomically renamed to the target path, guaranteeing a consistent state even if the process terminates mid-restore.

## S3-Compatible Backend Implementation

The S3 backend is implemented in [`crates/ltx/src/client/object_store.rs`](https://github.com/denoland/celld/blob/main/crates/ltx/src/client/object_store.rs) using the **`object_store`** crate, which provides a unified interface for AWS S3, MinIO, Cloudflare R2, and other compatible services.

### Object Store Integration

The `ObjectStoreClient` struct implements the `ReplicaClient` trait defined in [`crates/ltx/src/client/mod.rs`](https://github.com/denoland/celld/blob/main/crates/ltx/src/client/mod.rs). It maps five core operations—listing, reading, writing, deleting, and checking existence—onto the `object_store` API. Configuration parsing occurs via `ObjectStoreConfig::from_url()`, which accepts standard S3 URLs:

```rust
let url = "s3://my-bucket/celld-db?region=us-west-2".parse::<celld::replica_url::ParsedReplicaUrl>()?;
let cfg = ObjectStoreConfig::from_url(&url)?;
let client = ObjectStoreClient::new(cfg).await?;

```

### Multipart Uploads and Metadata

For objects larger than 5 MiB, the backend automatically initiates multipart uploads to optimize throughput and reliability. The implementation also supports batch delete operations (up to 1,000 keys per request) for cleanup tasks. LTX files are stored using the Litestream key naming convention:

```

{path}/{level:04x}/{min}-{max}.ltx

```

The original LTX header timestamp is preserved in the `litestream-timestamp` metadata header, ensuring temporal consistency across restores.

## Implementation Example

To configure continuous replication, bind a `Db` instance to a `Replica` backed by an S3-compatible client and spawn the sync loop:

```rust
use celld::db::Db;
use celld::ltx::client::object_store::{ObjectStoreClient, ObjectStoreConfig};
use celld::ltx::replica::{Replica, Pos};

// Parse the replica URL and build the S3 client
let url = "s3://my-bucket/celld-db?region=us-west-2".parse().expect("valid URL");
let cfg = ObjectStoreConfig::from_url(&url).expect("valid config");
let s3_client = ObjectStoreClient::new(cfg).await.expect("S3 client created");

// Open the managed SQLite database
let db = Db::open("./watch/mycell/ltx/e1/db.sqlite")
    .await
    .expect("opened DB");

// Create the replica and seed initial position
let mut replica = Replica::new(db, s3_client);
replica.seed_pos(Pos::ZERO);

// Spawn the continuous replication loop
tokio::spawn(async move {
    loop {
        if let Err(e) = replica.sync().await {
            eprintln!("replication error: {e}");
        }
        tokio::time::sleep(std::time::Duration::from_secs(5)).await;
    }
});

```

To force a restore from remote storage—useful after hardware failure or when migrating a cell to new infrastructure—use the client-only replica constructor:

```rust
use celld::ltx::replica::Replica;

let mut restore_replica = Replica::new_client_only(s3_client);
let restored_path = std::path::Path::new("./restored/db.sqlite");

restore_replica.restore(&restored_path)
    .await
    .expect("restore succeeded");

```

This downloads all LTX files in order, compacts them, and writes the final database to the specified path.

## Summary

- **LTX File Format**: celld writes every SQLite transaction to immutable L0 LTX files locally before uploading, creating a durable append-only log.
- **Continuous Sync**: A Tokio task in [`crates/celld/startup.rs`](https://github.com/denoland/celld/blob/main/crates/celld/startup.rs) runs `Replica::sync()` every five seconds, uploading new files to S3-compatible storage and advancing the TXID position.
- **Fault Tolerance**: Divergence errors reset the replica position automatically, while restore operations download and compact LTX files into a consistent SQLite database.
- **S3 Compatibility**: The `object_store` backend in [`crates/ltx/src/client/object_store.rs`](https://github.com/denoland/celld/blob/main/crates/ltx/src/client/object_store.rs) supports multipart uploads, batch deletes, and standard S3 URL parsing for AWS, MinIO, and R2.

## Frequently Asked Questions

### What happens if the sync loop crashes mid-upload?

celld treats LTX files as immutable and uploads are idempotent. If `Replica::sync()` fails, the replica position is not advanced for that TXID. On the next iteration, the loop retries the same file, overwriting any partial object in S3. The position tracking logic in [`crates/ltx/src/replica.rs`](https://github.com/denoland/celld/blob/main/crates/ltx/src/replica.rs) ensures that only fully uploaded and acknowledged transactions are considered complete.

### Can celld replicate to MinIO or Cloudflare R2 instead of AWS S3?

Yes. The `ObjectStoreClient` in [`crates/ltx/src/client/object_store.rs`](https://github.com/denoland/celld/blob/main/crates/ltx/src/client/object_store.rs) uses the generic `object_store` crate, which abstracts S3-compatible APIs. You can configure alternative endpoints via the replica URL (e.g., `s3://bucket/path?endpoint=http://localhost:9000&region=us-east-1`) and the `ObjectStoreConfig::from_url()` parser will handle the connection.

### How does celld handle large database files during restore?

The restore process streams LTX files from S3 using byte-range requests, processing them through the Litestream compactor without loading the entire database into memory. The final SQLite file is reconstructed incrementally and atomically renamed into place only after all TXIDs are applied and `fsync` confirms durability.

### What consistency guarantees does celld provide?

celld provides **eventual consistency** with a bounded staleness of approximately five seconds (the default sync interval). The replication model is write-once-read-many: once an LTX file is uploaded to S3, it becomes the source of truth for that TXID. Concurrent readers restoring from the bucket will see a consistent snapshot up to the highest contiguous TXID available in the object store.