# How Cell Failover Works in celld When the Owning Node Fails

> Discover how celld cell failover works when the owning node fails. Standby nodes detect lease expiration and atomically claim ownership for seamless recovery without consensus.

- Repository: [Deno/celld](https://github.com/denoland/celld)
- Tags: internals
- Published: 2026-08-10

---

**Cell failover in celld triggers when a standby node detects an expired lease in the shared S3-compatible bucket and atomically claims ownership via a compare-and-swap operation, allowing seamless recovery without a consensus protocol.**

celld, Deno's distributed SQLite engine, achieves high availability through a lease-based ownership model that stores cell authority as records in object storage. When the node owning a cell becomes unreachable, **cell failover** occurs through TTL expiration and atomic operations rather than complex consensus protocols. This design ensures that at most one node can own a cell at any given moment while enabling rapid recovery from node crashes.

## The Lease Structure in Object Storage

Each cell's ownership is stored as a **lease** record in a shared S3-compatible bucket, acting as the single source of truth for the entire cluster. According to [`crates/celld/protocol.rs`](https://github.com/denoland/celld/blob/main/crates/celld/protocol.rs), the lease structure contains three critical fields defined in the wire protocol: the owner's `node_id`, a monotonic `generation` counter, and an `expires_ms` timestamp representing the lease's TTL. The [`crates/celld/ownership_store.rs`](https://github.com/denoland/celld/blob/main/crates/celld/ownership_store.rs) module handles the actual storage and retrieval of these records from the bucket.

The node currently holding the lease must continuously renew it by extending the expiration time. If the owning node crashes or becomes unreachable, it can no longer renew the lease, causing the record to naturally expire based on wall-clock time.

## Detecting Node Death via Lease Expiration

Standby nodes continuously monitor the bucket for lease records using the `node_record_is_dead` function defined in [`crates/logic/dead_node_reconciliation.rs`](https://github.com/denoland/celld/blob/main/crates/logic/dead_node_reconciliation.rs). This function implements the expiration check that triggers failover:

```rust
// Helper used by `node_record_is_dead` (logic/dead_node_reconciliation.rs)
pub fn node_record_is_dead(
    key_node: &str,
    record_node: &str,
    expires_ms: u64,
    now_ms: u64,
) -> bool {
    // The lease key must still refer to the same node and be past its TTL
    key_node == record_node && expires_ms <= now_ms
}

```

When `expires_ms` is less than or equal to `now_ms`, the lease is considered dead, signaling that the owning node has failed and the cell is available for takeover.

## Atomic Takeover via Compare-and-Swap

Once a standby detects an expired lease, it attempts to claim ownership through an **atomic compare-and-swap (CAS)** operation. This mechanism, implemented in [`crates/ltx/src/leaser.rs`](https://github.com/denoland/celld/blob/main/crates/ltx/src/leaser.rs), ensures that only one node can successfully acquire ownership during a failover window.

The failover sequence follows these steps:

1. **Read Current State** – The standby reads the expired lease from the bucket.
2. **Prepare New Lease** – It constructs a new lease containing its own `node_id`, an incremented `generation` counter, and a fresh `expires_ms` timestamp.
3. **Atomic CAS** – The node issues a CAS operation that succeeds only if the stored lease still matches the expired value read in step 1.
4. **Acquire Ownership** – If the CAS succeeds, the node becomes the new primary; if it fails, another standby won the race.

The following pseudocode illustrates the failover path implemented in the celld source:

```rust
// Pseudocode showing the failover path
async fn monitor_and_failover(bucket: &Bucket, cell_key: &str) {
    loop {
        let lease = bucket.get_lease(cell_key).await?;
        if lease.is_expired(now_ms()) {
            // Try to become the new owner
            let new_lease = Lease {
                node_id: my_node_id(),
                expires_ms: now_ms() + LEASE_TTL,
                generation: lease.generation + 1,
            };
            // Atomic CAS: succeed only if the lease we saw is still present
            if bucket.compare_and_swap(cell_key, lease, new_lease).await.is_ok() {
                // We now own the cell – restore its DB and resume execution
                restore_cell_from_bucket(cell_key).await?;
                break;
            }
        }
        // Back-off before next check
        tokio::time::sleep(Duration::from_millis(
            retry_delay_ms(TICK_MS, failure_count)
        )).await;
    }
}

```

Because the bucket's CAS operation is linearizable, the transition from the dead primary to the new primary is atomic, preventing split-brain scenarios where multiple nodes believe they own the same cell.

## Restoring State and Resuming Execution

After acquiring the lease, the new owner must restore the cell's state before serving traffic. The [`crates/celld/replication.rs`](https://github.com/denoland/celld/blob/main/crates/celld/replication.rs) module handles downloading the cell's SQLite database from the bucket to the local node.

Once restoration completes, the node re-instantiates the Durable Object representing the cell and re-attaches any live WebSocket connections that were proxied through the cluster. The cell resumes full operation with the new node acting as the primary, all previous state intact.

## Retry and Back-off Mechanisms

To prevent thundering herds during failover storms or network partitions, [`crates/logic/dead_node_reconciliation.rs`](https://github.com/denoland/celld/blob/main/crates/logic/dead_node_reconciliation.rs) implements intelligent back-off logic via the `retry_delay_ms` function. This adds jittered delays between retry attempts, ensuring that competing standbys don't hammer the storage backend with CAS attempts during prolonged instability.

## Summary

- **Lease-based ownership** eliminates the need for consensus protocols by using TTL-based records in S3-compatible storage.
- **Expiration detection** relies on the `node_record_is_dead` function checking `expires_ms <= now_ms` in [`crates/logic/dead_node_reconciliation.rs`](https://github.com/denoland/celld/blob/main/crates/logic/dead_node_reconciliation.rs).
- **Atomic failover** uses compare-and-swap operations in [`crates/ltx/src/leaser.rs`](https://github.com/denoland/celld/blob/main/crates/ltx/src/leaser.rs) to guarantee that only one standby can claim an expired lease.
- **State restoration** occurs through [`crates/celld/replication.rs`](https://github.com/denoland/celld/blob/main/crates/celld/replication.rs), which downloads the SQLite database before the new primary resumes execution.
- **Back-off logic** prevents tight retry loops during unstable network conditions using `retry_delay_ms` calculations.

## Frequently Asked Questions

### What prevents two nodes from taking over the same cell simultaneously?

The **atomic compare-and-swap (CAS)** operation ensures only one winner. When multiple standbys detect an expired lease, they all attempt to write new lease records, but the CAS operation in [`crates/ltx/src/leaser.rs`](https://github.com/denoland/celld/blob/main/crates/ltx/src/leaser.rs) succeeds only for the node that correctly matches the previous lease value. All other attempts fail and must retry, guaranteeing at most one primary exists at any time.

### How does celld handle split-brain scenarios without a consensus protocol?

celld avoids split-brain by treating the S3-compatible bucket as the single source of truth. Since the bucket provides linearizable CAS operations, the lease acts as a global mutex. The system does not rely on node-to-node heartbeats or membership protocols; instead, lease expiration in object storage definitively marks a node as dead, making the failover decision unambiguous.

### What specific data is stored in a cell lease record?

Each lease record contains the owning node's identifier (`node_id`), a monotonic `generation` counter that increments with each ownership transfer, and an `expires_ms` timestamp indicating when the lease becomes invalid. This structure is defined in [`crates/celld/protocol.rs`](https://github.com/denoland/celld/blob/main/crates/celld/protocol.rs) and persisted via [`crates/celld/ownership_store.rs`](https://github.com/denoland/celld/blob/main/crates/celld/ownership_store.rs).

### How long does the failover process take?

Failover latency depends primarily on the **lease TTL** configured in the system and the time required to restore the SQLite database from object storage. Once the `expires_ms` timestamp passes, detection is immediate, and the CAS operation typically completes within milliseconds. However, total downtime includes the database restoration phase handled by [`crates/celld/replication.rs`](https://github.com/denoland/celld/blob/main/crates/celld/replication.rs), which varies with database size and network bandwidth.