# How Celld Handles Dead Node Reconciliation and Garbage Collection

> Learn how celld manages dead node reconciliation and garbage collection. Discover its efficient strategies for cleaning up stale records and orphaned ownership markers.

- Repository: [Deno/celld](https://github.com/denoland/celld)
- Tags: internals
- Published: 2026-08-15

---

**Celld handles dead node reconciliation and garbage collection by periodically scanning a shared fleet-wide object store for expired node records, cleaning up orphaned `node-cells` ownership markers, and retiring stale node session records using conditional updates with exponential back-off retry logic.**

Celld is a distributed cell execution engine that coordinates state through a shared object store with other services in the fleet. When a Celld process terminates unexpectedly, its lease and ownership-index markers persist as stale data that can block new processes from acquiring ownership of cells. The system solves this through a **compatibility garbage-collection loop** that ensures cluster health without manual intervention.

## Detecting Dead Nodes in the Shared Store

The garbage collection process begins with node discovery. In [`crates/celld/dead_node_gc.rs`](https://github.com/denoland/celld/blob/main/crates/celld/dead_node_gc.rs), the `dead_nodes` function lists all objects under the `nodes/` prefix and evaluates each node's JSON record against the current wall-clock time.

The helper `node_record_is_dead` performs the expiration check at lines 97–104, comparing the record's `expires_ms` field to determine if the node session has timed out.

```rust
// Conceptual flow based on dead_node_gc.rs#L97-L104
fn node_record_is_dead(record: &NodeRecord, now_ms: u64) -> bool {
    record.expires_ms < now_ms
}

```

Nodes that fail this check are queued for cleanup. This detection runs while the process holds the fleet-wide **waker lease**, ensuring only one node performs reconciliation at any time.

## Parsing and Indexing Node-Cell Markers

Dead nodes leave behind ownership markers in the `node-cells/` prefix. These markers follow the pattern `node-cells/<node>/<generation>/<cell>`, where the generation component is versioning metadata that the GC ignores.

In [`crates/logic/dead_node_reconciliation.rs`](https://github.com/denoland/celld/blob/main/crates/logic/dead_node_reconciliation.rs), the `parse_marker_key` function at lines 26–34 extracts the node and cell identifiers:

```rust
// Based on dead_node_reconciliation.rs#L26-L34
fn parse_marker_key(key: &str) -> Option<(String, String)> {
    // key format: "node-cells/{node}/{generation}/{cell}"
    let parts: Vec<&str> = key.split('/').collect();
    if parts.len() >= 4 {
        Some((parts[1].to_string(), parts[3].to_string()))
    } else {
        None
    }
}

```

The GC then builds a per-node index of markers to clean. In [`dead_node_gc.rs`](https://github.com/denoland/celld/blob/main/dead_node_gc.rs) at lines 35–52, `cells_indexed_by_nodes` aggregates all `node-cells` objects belonging to each dead node, enabling batched deletion.

## Concurrent Marker Deletion with Retry Logic

Marker deletion proceeds concurrently with bounded parallelism. The `gc_markers` function in [`dead_node_gc.rs`](https://github.com/denoland/celld/blob/main/dead_node_gc.rs) (lines 56–90) issues up to **64 concurrent `bucket.delete` calls**, tracking successes and failures separately.

Failed deletions trigger exponential back-off computed by `retry_delay_ms` in [`dead_node_reconciliation.rs`](https://github.com/denoland/celld/blob/main/dead_node_reconciliation.rs) at lines 15–19. The delay grows as **1× → 2× → 4× … up to 64× the tick interval**, preventing tight retry loops when the object store experiences degraded availability.

```rust
// Based on dead_node_reconciliation.rs#L15-L19
fn retry_delay_ms(attempt: u32, tick_ms: u64) -> u64 {
    let multiplier = (1u64 << attempt.min(6)).min(64);
    tick_ms * multiplier
}

```

## Retiring Dead Node Records Safely

After all markers are cleared, the GC must remove the dead node's session record without creating a window where the record appears alive. The `retire_dead_node` function in [`dead_node_gc.rs`](https://github.com/denoland/celld/blob/main/dead_node_gc.rs) (lines 94–121) implements a **two-step fence**:

1. **Conditional update (CAS)** writes a tombstone with `expires_ms = 0`
2. **Unconditional delete** removes the object entirely

This design guarantees crash safety: if the process fails between steps, the tombstone record still appears dead on the next GC pass and will be retried.

## State Management and Periodic Execution

The `DeadNodeGc` struct maintains local state to optimize repeated passes. As defined at lines 46–55 in [`dead_node_gc.rs`](https://github.com/denoland/celld/blob/main/dead_node_gc.rs), it tracks:

- **`swept`**: Nodes already cleaned in previous passes (avoid redundant work)
- **`retries`**: Per-node failure counters to enforce back-off

The `run_elected_pass` function (lines 58–95) integrates with Celld's waker system. It:
- Renews the waker lease during execution
- Cancels the pass immediately if lease renewal fails
- Returns control to the caller for the next tick interval

## Example: Running the Dead Node GC

Below is a minimal Rust example showing how to instantiate and run the garbage collector as part of a waker loop:

```rust
use celld::dead_node_gc::DeadNodeGc;
use celld::bucket::Bucket;
use std::time::Duration;
use tokio::time;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Assume `bucket` is a configured Bucket pointing at the shared fleet store.
    let bucket: Bucket = Bucket::new("s3", "celld-fleet-bucket")?;

    // Identifier of the current node (the waker participant)
    let node_id = "node-1234";

    // How often the waker ticks (e.g., 5 seconds)
    let tick_ms = 5_000;

    // The GC is stateful – we keep it around across passes.
    let mut gc = DeadNodeGc::default();

    // Run the GC as part of the regular waker loop.
    let mut interval = time::interval(Duration::from_millis(tick_ms));
    loop {
        interval.tick().await;
        // `run_elected_pass` will early-return if this node has lost the
        // waker lease, otherwise it performs one full dead-node reconciliation.
        gc.run_elected_pass(&bucket, node_id, tick_ms).await?;
    }
}

```

This pattern demonstrates the intended integration: create a single `DeadNodeGc` instance, then invoke `run_elected_pass` on each waker tick. Lease renewal and cancellation handling occur internally—no additional coordination code is required.

## Key Source Files

- **[`crates/logic/dead_node_reconciliation.rs`](https://github.com/denoland/celld/blob/main/crates/logic/dead_node_reconciliation.rs)** — Core I/O-free utilities: `parse_marker_key`, `node_record_is_dead`, `retry_delay_ms`
- **[`crates/celld/dead_node_gc.rs`](https://github.com/denoland/celld/blob/main/crates/celld/dead_node_gc.rs)** — Main GC implementation: `DeadNodeGc`, `run_elected_pass`, marker scanning, deletion, and node retirement
- **[`crates/celld/bucket.rs`](https://github.com/denoland/celld/blob/main/crates/celld/bucket.rs)** — Object store abstraction for list, get, put-CAS, and delete operations
- **[`crates/celld/wake.rs`](https://github.com/denoland/celld/blob/main/crates/celld/wake.rs)** — Waker lease management; gates GC execution to the elected leader
- **[`crates/celld/replication.rs`](https://github.com/denoland/celld/blob/main/crates/celld/replication.rs)** — Coordination layer integrating waker and GC passes

## Summary

- **Dead node detection** checks `expires_ms` against wall-clock time in `dead_nodes`
- **Marker parsing** extracts node/cell identifiers while discarding generation suffixes via `parse_marker_key`
- **Concurrent deletion** runs up to 64 operations in parallel with exponential back-off on failure
- **Safe retirement** uses a two-step CAS-then-delete fence to prevent phantom alive records
- **State tracking** avoids redundant work and limits retry storms through per-node counters
- **Leader-only execution** ties GC passes to the waker lease, ensuring single-node coordination

## Frequently Asked Questions

### What triggers dead node garbage collection in Celld?

The GC runs periodically as part of the waker loop while a node holds the fleet-wide waker lease. Each `tick_ms` interval, `run_elected_pass` performs one reconciliation sweep if the node remains the elected leader. The lease itself is renewed during GC execution, with automatic cancellation if renewal fails.

### How does Celld prevent multiple nodes from running GC simultaneously?

The waker lease mechanism in [`crates/celld/wake.rs`](https://github.com/denoland/celld/blob/main/crates/celld/wake.rs) ensures mutual exclusion. Only the node holding the lease invokes `run_elected_pass`; if lease renewal fails mid-pass, the GC cancels immediately. This integrates GC execution with Celld's existing leader election rather than requiring separate distributed locking.

### Why does marker deletion use exponential back-off?

The `retry_delay_ms` function prevents thundering-herd retry storms when the object store is degraded. Delays scale from 1× to 64× the tick interval, capping maximum backoff. This protects both Celld and the underlying storage from cascading failure during outages.

### What happens if the GC crashes while retiring a node record?

The two-step retirement fence in `retire_dead_node` ensures crash safety. First, a CAS operation writes `expires_ms = 0` as a tombstone. Only then does unconditional deletion occur. If the process crashes between steps, the tombstone remains visible to subsequent GC passes, which will retry the full retirement sequence.