# How celld Detects Node Failures and Handles Lease Expirations

> Learn how celld detects node failures by scanning for expired leases and safely removes dead nodes. Discover how lease renewals are managed for efficient garbage collection.

- Repository: [Deno/celld](https://github.com/denoland/celld)
- Tags: internals
- Published: 2026-08-09

---

**Celld detects node failures by scanning fleet bucket records for expired leases and safely removes dead nodes using conditional writes, while the fleet-waker role manages lease renewals to coordinate garbage collection.**

The `denoland/celld` distributed process scheduler tracks cluster membership using time-bound leases stored in a shared fleet bucket. Each active node periodically writes a JSON record containing an `expires_ms` timestamp; when a lease expires without renewal, celld marks the node as dead and triggers garbage collection. This article explains the exact mechanisms for **celld node failure detection and lease expiration** handling, referencing the production Rust source code.

## Node Failure Detection Mechanism

Celld stores a *node record* for every active process under the key pattern `nodes/<node>.json` in the shared fleet bucket. Each record contains the node identifier, a generation string, and the critical `expires_ms` timestamp representing wall-clock lease expiry.

### Identifying Dead Nodes with node_record_is_dead

The background `DeadNodeGc` task periodically executes `run_pass` to list all objects under `nodes/` and read each record via `read_node`. For every record, it calls `celld_logic::dead_node_reconciliation::node_record_is_dead`:

```rust
// crates/logic/dead_node_reconciliation.rs
pub fn node_record_is_dead(
    key_node: &str,
    record_node: &str,
    expires_ms: u64,
    now_ms: u64,
) -> bool {
    key_node == record_node && expires_ms <= now_ms
}

```

The function returns `true` only when the node ID in the filename matches the ID stored inside the file **and** the `expires_ms` timestamp is less than or equal to the current time. When both conditions hold, the node is added to the `dead` list for cleanup (see [`crates/celld/dead_node_gc.rs`](https://github.com/denoland/celld/blob/main/crates/celld/dead_node_gc.rs), lines 99-102).

## Lease Expiration and the Fleet-Waker Role

Only one celld instance acts as the *fleet-waker* at any time. This elected instance coordinates dead-node garbage collection by holding a renewable lease that must be actively maintained.

### Acquiring the Waker Lease with try_hold_waker

The waker acquires its role by calling `crate::wake::try_hold_waker` from [`crates/celld/wake.rs`](https://github.com/denoland/celld/blob/main/crates/celld/wake.rs) at the start of each GC pass:

```rust
// crates/celld/dead_node_gc.rs – lease acquisition
let lease_ttl_ms = tick_ms.saturating_mul(3).min(i64::MAX as u64);
if !crate::wake::try_hold_waker(
    bucket,
    node,
    crate::ownership_store::now_ms() as i64,
    lease_ttl_ms as i64,
).await {
    return;
}

```

If `try_hold_waker` returns `false`, the GC pass aborts immediately, ensuring that only a valid waker with an active lease can perform node decommissioning.

### Renewal Strategy and TTL Calculation

The lease TTL is calculated as three times the GC tick interval using `tick_ms.saturating_mul(3).min(i64::MAX as u64)` to prevent overflow. A renewal timer set to one-third of the TTL triggers periodic re-acquisition (lines 74-78). If renewal fails, the GC pass cancels (lines 84-92), preventing split-brain scenarios where multiple instances might simultaneously attempt cleanup.

## Safe Cleanup of Dead Nodes

Once a node is confirmed dead, `retire_dead_node` executes a two-phase deletion to prevent race conditions where a concurrent writer might resurrect the node after lease expiry.

### Tombstone Writes and CAS Operations

The cleanup writes a tombstone record with `expires_ms: 0` using a conditional CAS write (`bucket.put_cas`), then deletes the object (lines 107-122 of [`crates/celld/dead_node_gc.rs`](https://github.com/denoland/celld/blob/main/crates/celld/dead_node_gc.rs)):

```rust
// Conceptual implementation from dead_node_gc.rs
// Write tombstone with expires_ms: 0 using conditional CAS
bucket.put_cas(&key, &tombstone_data, expected_generation).await?;
// Safe to delete after tombstone confirms state
bucket.delete(&key).await?;

```

This guarantees that a concurrent writer cannot resurrect the node after the lease has expired, as the compare-and-swap operation validates the record generation before permitting the tombstone write.

## Summary

- **Node records** stored at `nodes/<node>.json` contain `expires_ms` timestamps that act as time-bound leases.
- **Failure detection** relies on `node_record_is_dead` in [`crates/logic/dead_node_reconciliation.rs`](https://github.com/denoland/celld/blob/main/crates/logic/dead_node_reconciliation.rs), which verifies ID consistency and timestamp expiry.
- **Fleet-waker election** uses `try_hold_waker` in [`crates/celld/wake.rs`](https://github.com/denoland/celld/blob/main/crates/celld/wake.rs) to ensure only one instance manages dead-node GC at a time.
- **Lease TTL** is calculated as three times the GC tick interval with saturation arithmetic, with renewals occurring at one-third intervals.
- **Safe deletion** employs tombstone writes with CAS operations in `retire_dead_node` to prevent race conditions during cleanup.

## Frequently Asked Questions

### How does celld determine if a node is dead?

Celld scans the fleet bucket for node records and calls `node_record_is_dead`, which verifies that the filename matches the internal node ID and that the `expires_ms` timestamp has passed the current wall-clock time. If both conditions are true, the node is flagged for garbage collection in [`crates/celld/dead_node_gc.rs`](https://github.com/denoland/celld/blob/main/crates/celld/dead_node_gc.rs).

### What happens if the fleet-waker lease expires?

If the waker fails to renew its lease via `try_hold_waker`, the `DeadNodeGc` pass aborts immediately (lines 84-92). This prevents multiple celld instances from simultaneously attempting to clean up nodes, avoiding split-brain scenarios in the distributed system.

### How does celld prevent resurrecting dead nodes during cleanup?

The `retire_dead_node` function writes a tombstone with `expires_ms: 0` using a conditional `put_cas` operation before deletion. This ensures that any concurrent write attempting to update the node record will fail the compare-and-swap check, maintaining consistency and preventing zombie node resurrection.

### What is the relationship between GC tick interval and lease TTL?

The lease TTL is set to three times the GC tick interval (`tick_ms * 3`) using saturating multiplication, with renewals occurring every one-third of the TTL. This provides a buffer window for network latency while ensuring timely detection of waker failures.