How celld Detects Node Failures and Handles Lease Expirations

Celld detects node failures by scanning fleet bucket records for expired leases and safely removes dead nodes using conditional writes, while the fleet-waker role manages lease renewals to coordinate garbage collection.

The denoland/celld distributed process scheduler tracks cluster membership using time-bound leases stored in a shared fleet bucket. Each active node periodically writes a JSON record containing an expires_ms timestamp; when a lease expires without renewal, celld marks the node as dead and triggers garbage collection. This article explains the exact mechanisms for celld node failure detection and lease expiration handling, referencing the production Rust source code.

Node Failure Detection Mechanism

Celld stores a node record for every active process under the key pattern nodes/<node>.json in the shared fleet bucket. Each record contains the node identifier, a generation string, and the critical expires_ms timestamp representing wall-clock lease expiry.

Identifying Dead Nodes with node_record_is_dead

The background DeadNodeGc task periodically executes run_pass to list all objects under nodes/ and read each record via read_node. For every record, it calls celld_logic::dead_node_reconciliation::node_record_is_dead:

// crates/logic/dead_node_reconciliation.rs
pub fn node_record_is_dead(
    key_node: &str,
    record_node: &str,
    expires_ms: u64,
    now_ms: u64,
) -> bool {
    key_node == record_node && expires_ms <= now_ms
}

The function returns true only when the node ID in the filename matches the ID stored inside the file and the expires_ms timestamp is less than or equal to the current time. When both conditions hold, the node is added to the dead list for cleanup (see crates/celld/dead_node_gc.rs, lines 99-102).

Lease Expiration and the Fleet-Waker Role

Only one celld instance acts as the fleet-waker at any time. This elected instance coordinates dead-node garbage collection by holding a renewable lease that must be actively maintained.

Acquiring the Waker Lease with try_hold_waker

The waker acquires its role by calling crate::wake::try_hold_waker from crates/celld/wake.rs at the start of each GC pass:

// crates/celld/dead_node_gc.rs – lease acquisition
let lease_ttl_ms = tick_ms.saturating_mul(3).min(i64::MAX as u64);
if !crate::wake::try_hold_waker(
    bucket,
    node,
    crate::ownership_store::now_ms() as i64,
    lease_ttl_ms as i64,
).await {
    return;
}

If try_hold_waker returns false, the GC pass aborts immediately, ensuring that only a valid waker with an active lease can perform node decommissioning.

Renewal Strategy and TTL Calculation

The lease TTL is calculated as three times the GC tick interval using tick_ms.saturating_mul(3).min(i64::MAX as u64) to prevent overflow. A renewal timer set to one-third of the TTL triggers periodic re-acquisition (lines 74-78). If renewal fails, the GC pass cancels (lines 84-92), preventing split-brain scenarios where multiple instances might simultaneously attempt cleanup.

Safe Cleanup of Dead Nodes

Once a node is confirmed dead, retire_dead_node executes a two-phase deletion to prevent race conditions where a concurrent writer might resurrect the node after lease expiry.

Tombstone Writes and CAS Operations

The cleanup writes a tombstone record with expires_ms: 0 using a conditional CAS write (bucket.put_cas), then deletes the object (lines 107-122 of crates/celld/dead_node_gc.rs):

// Conceptual implementation from dead_node_gc.rs
// Write tombstone with expires_ms: 0 using conditional CAS
bucket.put_cas(&key, &tombstone_data, expected_generation).await?;
// Safe to delete after tombstone confirms state
bucket.delete(&key).await?;

This guarantees that a concurrent writer cannot resurrect the node after the lease has expired, as the compare-and-swap operation validates the record generation before permitting the tombstone write.

Summary

• Node records stored at nodes/<node>.json contain expires_ms timestamps that act as time-bound leases.
• Failure detection relies on node_record_is_dead in crates/logic/dead_node_reconciliation.rs, which verifies ID consistency and timestamp expiry.
• Fleet-waker election uses try_hold_waker in crates/celld/wake.rs to ensure only one instance manages dead-node GC at a time.
• Lease TTL is calculated as three times the GC tick interval with saturation arithmetic, with renewals occurring at one-third intervals.
• Safe deletion employs tombstone writes with CAS operations in retire_dead_node to prevent race conditions during cleanup.

Frequently Asked Questions

How does celld determine if a node is dead?

Celld scans the fleet bucket for node records and calls node_record_is_dead, which verifies that the filename matches the internal node ID and that the expires_ms timestamp has passed the current wall-clock time. If both conditions are true, the node is flagged for garbage collection in crates/celld/dead_node_gc.rs.

What happens if the fleet-waker lease expires?

If the waker fails to renew its lease via try_hold_waker, the DeadNodeGc pass aborts immediately (lines 84-92). This prevents multiple celld instances from simultaneously attempting to clean up nodes, avoiding split-brain scenarios in the distributed system.

How does celld prevent resurrecting dead nodes during cleanup?

The retire_dead_node function writes a tombstone with expires_ms: 0 using a conditional put_cas operation before deletion. This ensures that any concurrent write attempting to update the node record will fail the compare-and-swap check, maintaining consistency and preventing zombie node resurrection.

What is the relationship between GC tick interval and lease TTL?

The lease TTL is set to three times the GC tick interval (tick_ms * 3) using saturating multiplication, with renewals occurring every one-third of the TTL. This provides a buffer window for network latency while ensuring timely detection of waker failures.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}