# Durability Guarantees of celld's Replication Protocol

> Explore celld's replication protocol durability guarantees. Learn how SQLite snapshots are persisted to S3 ensuring data survives node crashes and providing linearizable reads.

- Repository: [Deno/celld](https://github.com/denoland/celld)
- Tags: deep-dive
- Published: 2026-08-10

---

**celld's replication protocol guarantees that once a write is acknowledged as durable, the corresponding SQLite snapshot has been successfully persisted to an S3-compatible bucket, ensuring the write survives any node crash or restart while providing linearizable reads up to the last durable transaction ID.**

celld is a runtime for Durable Objects (cells) that leverages SQLite for state management and S3-compatible storage for persistence. Understanding the durability guarantees of celld's replication protocol is essential for building distributed applications that require strong persistence semantics and predictable failure recovery.

## Durable Write Acknowledgement

The primary durability mechanism in celld centers on explicit write acknowledgment. When your application calls `ensure_durable` or `await_durable`, the runtime blocks until the latest local commit has been successfully uploaded to the configured S3-compatible bucket.

In [`crates/celld/runtime.rs`](https://github.com/denoland/celld/blob/main/crates/celld/runtime.rs) (lines 184-211), the `ensure_durable` method delegates to `Replication::ensure_durable`, which performs the following sequence:

1. Captures the current local transaction state
2. Uploads the SQLite snapshot to the bucket
3. Returns only after confirming the object exists in remote storage

```rust
// Example: Wait for a cell's latest write to become durable
let cell = "my-cell";
let epoch = 42;

// Block until the latest local commit is present in the bucket
runtime.ensure_durable(cell, epoch).await?;

// Or retrieve the exact durable position (TXID) for a specific write
let position = 1234;
let durable_txid = runtime.await_durable(cell, epoch, position).await?;
println!("Write {} is durable at TXID {}", position, durable_txid);

```

From the moment `ensure_durable` resolves, the write is considered **strongly durable**—it will survive any subsequent node crash, restart, or ownership transfer because the bucket holds the authoritative copy.

## Linearizable Reads Up to Last Durable TXID

celld guarantees that read operations observe only state that has been persisted to the bucket. The replica exposes database state that is **≤ the last durable TXID**, ensuring no read observes a write that has not yet been persisted.

This property is explicitly verified in the test suite at [`crates/ltx/tests/faults_inject.rs`](https://github.com/denoland/celld/blob/main/crates/ltx/tests/faults_inject.rs) (lines 7-9 and 105-107). The tests confirm that:

- Reads are bounded by the last durable transaction ID
- Local writes that have not yet replicated remain invisible to other nodes
- The database remains valid at any TXID up to and including the last durable position

This creates a consistency model where reads are linearizable with respect to acknowledged durable writes, though there exists a bounded window where writes are locally visible but not yet durable.

## Fail-Over Recovery Mechanisms

When a node loses ownership of a cell—whether through crash, network partition, or graceful handoff—another node can restore the cell from the durable snapshot stored in the bucket.

The restoration logic in [`crates/celld/ltx_repl.rs`](https://github.com/denoland/celld/blob/main/crates/celld/ltx_repl.rs) (lines 202-216) handles this transition:

```rust
// Example: Restoring a cell after takeover
let activation = replication.activate(ActivationOptions {
    cell: "my-cell",
    epoch: 43,
    fresh: false,
    took_over: true,
})?;
if activation.restored {
    // The snapshot includes all durable writes up to the latest epoch
    println!("Cell restored from bucket snapshot");
}

```

The restored replica starts with the *newest durable epoch* and can immediately serve reads guaranteed to include all previously durable writes. This ensures **zero data loss** for any write that received a durability acknowledgment before the failure.

## Graceful Degradation and Configuration

The replication protocol provides configurable semantics for scenarios where durability cannot be immediately proven. If the replicator is unavailable or the S3 bucket is unreachable, celld can fall back to best-effort semantics rather than failing operations entirely.

This behavior is controlled via the `CELLD_OUTPUT_GATE` environment variable, defined in [`crates/celld/main.rs`](https://github.com/denoland/celld/blob/main/crates/celld/main.rs) (lines 512-514). When configured for graceful degradation:

- Clients may receive responses without explicit durability proof
- Operations can be retried by the application layer
- The system maintains availability at the cost of temporary durability uncertainty

This configuration allows operators to balance between strict durability requirements and availability during network partitions or storage outages.

## Durability Scope and Implementation Boundaries

It is important to understand that celld's strong durability guarantees apply only to writes that have been flushed to the bucket. The protocol does **not** guarantee immediate consistency between nodes during the replication window—there is a bounded delay between local commit and remote persistence.

Clients requiring strict ordering guarantees should always await durability confirmation before proceeding with dependent operations. The `await_durable` API (defined in [`crates/celld/runtime.rs`](https://github.com/denoland/celld/blob/main/crates/celld/runtime.rs), lines 184-221) provides fine-grained control for sequencing operations against specific durable positions.

Key implementation files supporting these guarantees include:

- [`crates/celld/replication.rs`](https://github.com/denoland/celld/blob/main/crates/celld/replication.rs) — Low-level replication primitives and snapshot upload logic
- [`crates/ltx/src/replica.rs`](https://github.com/denoland/celld/blob/main/crates/ltx/src/replica.rs) — Local LTX replica state management and durable position tracking
- [`crates/celld/ltx_repl.rs`](https://github.com/denoland/celld/blob/main/crates/celld/ltx_repl.rs) — Coordination of cell restoration from durable snapshots

## Summary

- **Durable Write Acknowledgement**: `ensure_durable` and `await_durable` in [`runtime.rs`](https://github.com/denoland/celld/blob/main/runtime.rs) block until writes are persisted to the S3-compatible bucket, guaranteeing survival through node crashes.
- **Linearizable Reads**: The protocol ensures reads never observe writes beyond the last durable TXID, verified by the [`faults_inject.rs`](https://github.com/denoland/celld/blob/main/faults_inject.rs) test suite.
- **Fail-Over Recovery**: Nodes restore from `sqlite_snapshot` in the bucket after takeover, starting from the newest durable epoch with zero data loss for acknowledged writes.
- **Graceful Degradation**: The `CELLD_OUTPUT_GATE` environment variable configures fallback to best-effort semantics when durability proof is unavailable.
- **Bounded Consistency Window**: Strong durability applies only to bucket-flushed writes; local commits remain invisible to other nodes until replication completes.

## Frequently Asked Questions

### What happens if a node crashes before durability is confirmed?

If a node crashes before `ensure_durable` resolves, the write may be lost. The system only guarantees persistence for writes that have received explicit acknowledgment. Upon restart or takeover, the new node restores from the last durable snapshot in the bucket, which contains only writes that were confirmed before the crash.

### How does celld handle network partitions to the S3 bucket?

According to the source code in [`main.rs`](https://github.com/denoland/celld/blob/main/main.rs) (lines 512-514), celld uses the `CELLD_OUTPUT_GATE` configuration to determine behavior during storage outages. The system can either block until connectivity returns or fall back to best-effort semantics, depending on the configured durability requirements and availability priorities.

### What is the consistency model for reads in celld?

celld provides linearizable reads up to the last durable transaction ID. As implemented in [`crates/ltx/tests/faults_inject.rs`](https://github.com/denoland/celld/blob/main/crates/ltx/tests/faults_inject.rs), reads are bounded by the durability frontier—clients will never see data that has not been persisted to the bucket, though they may observe slightly stale state during active replication.

### How can developers enforce stronger durability guarantees?

Developers should use the `await_durable` method from [`crates/celld/runtime.rs`](https://github.com/denoland/celld/blob/main/crates/celld/runtime.rs) to block until specific writes are confirmed in the bucket. For critical operations, awaiting durability before responding to clients ensures that subsequent fail-over or reads by other nodes will observe the written data, eliminating the bounded window of local-only visibility.