How Cell Failover Works in celld When the Owning Node Fails

Cell failover in celld triggers when a standby node detects an expired lease in the shared S3-compatible bucket and atomically claims ownership via a compare-and-swap operation, allowing seamless recovery without a consensus protocol.

celld, Deno's distributed SQLite engine, achieves high availability through a lease-based ownership model that stores cell authority as records in object storage. When the node owning a cell becomes unreachable, cell failover occurs through TTL expiration and atomic operations rather than complex consensus protocols. This design ensures that at most one node can own a cell at any given moment while enabling rapid recovery from node crashes.

The Lease Structure in Object Storage

Each cell's ownership is stored as a lease record in a shared S3-compatible bucket, acting as the single source of truth for the entire cluster. According to crates/celld/protocol.rs, the lease structure contains three critical fields defined in the wire protocol: the owner's node_id, a monotonic generation counter, and an expires_ms timestamp representing the lease's TTL. The crates/celld/ownership_store.rs module handles the actual storage and retrieval of these records from the bucket.

The node currently holding the lease must continuously renew it by extending the expiration time. If the owning node crashes or becomes unreachable, it can no longer renew the lease, causing the record to naturally expire based on wall-clock time.

Detecting Node Death via Lease Expiration

Standby nodes continuously monitor the bucket for lease records using the node_record_is_dead function defined in crates/logic/dead_node_reconciliation.rs. This function implements the expiration check that triggers failover:

// Helper used by `node_record_is_dead` (logic/dead_node_reconciliation.rs)
pub fn node_record_is_dead(
    key_node: &str,
    record_node: &str,
    expires_ms: u64,
    now_ms: u64,
) -> bool {
    // The lease key must still refer to the same node and be past its TTL
    key_node == record_node && expires_ms <= now_ms
}

When expires_ms is less than or equal to now_ms, the lease is considered dead, signaling that the owning node has failed and the cell is available for takeover.

Atomic Takeover via Compare-and-Swap

Once a standby detects an expired lease, it attempts to claim ownership through an atomic compare-and-swap (CAS) operation. This mechanism, implemented in crates/ltx/src/leaser.rs, ensures that only one node can successfully acquire ownership during a failover window.

The failover sequence follows these steps:

1. Read Current State – The standby reads the expired lease from the bucket.
2. Prepare New Lease – It constructs a new lease containing its own node_id, an incremented generation counter, and a fresh expires_ms timestamp.
3. Atomic CAS – The node issues a CAS operation that succeeds only if the stored lease still matches the expired value read in step 1.
4. Acquire Ownership – If the CAS succeeds, the node becomes the new primary; if it fails, another standby won the race.

The following pseudocode illustrates the failover path implemented in the celld source:

// Pseudocode showing the failover path
async fn monitor_and_failover(bucket: &Bucket, cell_key: &str) {
    loop {
        let lease = bucket.get_lease(cell_key).await?;
        if lease.is_expired(now_ms()) {
            // Try to become the new owner
            let new_lease = Lease {
                node_id: my_node_id(),
                expires_ms: now_ms() + LEASE_TTL,
                generation: lease.generation + 1,
            };
            // Atomic CAS: succeed only if the lease we saw is still present
            if bucket.compare_and_swap(cell_key, lease, new_lease).await.is_ok() {
                // We now own the cell – restore its DB and resume execution
                restore_cell_from_bucket(cell_key).await?;
                break;
            }
        }
        // Back-off before next check
        tokio::time::sleep(Duration::from_millis(
            retry_delay_ms(TICK_MS, failure_count)
        )).await;
    }
}

Because the bucket's CAS operation is linearizable, the transition from the dead primary to the new primary is atomic, preventing split-brain scenarios where multiple nodes believe they own the same cell.

Restoring State and Resuming Execution

After acquiring the lease, the new owner must restore the cell's state before serving traffic. The crates/celld/replication.rs module handles downloading the cell's SQLite database from the bucket to the local node.

Once restoration completes, the node re-instantiates the Durable Object representing the cell and re-attaches any live WebSocket connections that were proxied through the cluster. The cell resumes full operation with the new node acting as the primary, all previous state intact.

Retry and Back-off Mechanisms

To prevent thundering herds during failover storms or network partitions, crates/logic/dead_node_reconciliation.rs implements intelligent back-off logic via the retry_delay_ms function. This adds jittered delays between retry attempts, ensuring that competing standbys don't hammer the storage backend with CAS attempts during prolonged instability.

Summary

• Lease-based ownership eliminates the need for consensus protocols by using TTL-based records in S3-compatible storage.
• Expiration detection relies on the node_record_is_dead function checking expires_ms <= now_ms in crates/logic/dead_node_reconciliation.rs.
• Atomic failover uses compare-and-swap operations in crates/ltx/src/leaser.rs to guarantee that only one standby can claim an expired lease.
• State restoration occurs through crates/celld/replication.rs, which downloads the SQLite database before the new primary resumes execution.
• Back-off logic prevents tight retry loops during unstable network conditions using retry_delay_ms calculations.

Frequently Asked Questions

What prevents two nodes from taking over the same cell simultaneously?

The atomic compare-and-swap (CAS) operation ensures only one winner. When multiple standbys detect an expired lease, they all attempt to write new lease records, but the CAS operation in crates/ltx/src/leaser.rs succeeds only for the node that correctly matches the previous lease value. All other attempts fail and must retry, guaranteeing at most one primary exists at any time.

How does celld handle split-brain scenarios without a consensus protocol?

celld avoids split-brain by treating the S3-compatible bucket as the single source of truth. Since the bucket provides linearizable CAS operations, the lease acts as a global mutex. The system does not rely on node-to-node heartbeats or membership protocols; instead, lease expiration in object storage definitively marks a node as dead, making the failover decision unambiguous.

What specific data is stored in a cell lease record?

Each lease record contains the owning node's identifier (node_id), a monotonic generation counter that increments with each ownership transfer, and an expires_ms timestamp indicating when the lease becomes invalid. This structure is defined in crates/celld/protocol.rs and persisted via crates/celld/ownership_store.rs.

How long does the failover process take?

Failover latency depends primarily on the lease TTL configured in the system and the time required to restore the SQLite database from object storage. Once the expires_ms timestamp passes, detection is immediate, and the CAS operation typically completes within milliseconds. However, total downtime includes the database restoration phase handled by crates/celld/replication.rs, which varies with database size and network bandwidth.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}