How Celld Handles Dead Node Reconciliation and Garbage Collection

Celld handles dead node reconciliation and garbage collection by periodically scanning a shared fleet-wide object store for expired node records, cleaning up orphaned node-cells ownership markers, and retiring stale node session records using conditional updates with exponential back-off retry logic.

Celld is a distributed cell execution engine that coordinates state through a shared object store with other services in the fleet. When a Celld process terminates unexpectedly, its lease and ownership-index markers persist as stale data that can block new processes from acquiring ownership of cells. The system solves this through a compatibility garbage-collection loop that ensures cluster health without manual intervention.

Detecting Dead Nodes in the Shared Store

The garbage collection process begins with node discovery. In crates/celld/dead_node_gc.rs, the dead_nodes function lists all objects under the nodes/ prefix and evaluates each node's JSON record against the current wall-clock time.

The helper node_record_is_dead performs the expiration check at lines 97–104, comparing the record's expires_ms field to determine if the node session has timed out.

// Conceptual flow based on dead_node_gc.rs#L97-L104
fn node_record_is_dead(record: &NodeRecord, now_ms: u64) -> bool {
    record.expires_ms < now_ms
}

Nodes that fail this check are queued for cleanup. This detection runs while the process holds the fleet-wide waker lease, ensuring only one node performs reconciliation at any time.

Parsing and Indexing Node-Cell Markers

Dead nodes leave behind ownership markers in the node-cells/ prefix. These markers follow the pattern node-cells/<node>/<generation>/<cell>, where the generation component is versioning metadata that the GC ignores.

In crates/logic/dead_node_reconciliation.rs, the parse_marker_key function at lines 26–34 extracts the node and cell identifiers:

// Based on dead_node_reconciliation.rs#L26-L34
fn parse_marker_key(key: &str) -> Option<(String, String)> {
    // key format: "node-cells/{node}/{generation}/{cell}"
    let parts: Vec<&str> = key.split('/').collect();
    if parts.len() >= 4 {
        Some((parts[1].to_string(), parts[3].to_string()))
    } else {
        None
    }
}

The GC then builds a per-node index of markers to clean. In dead_node_gc.rs at lines 35–52, cells_indexed_by_nodes aggregates all node-cells objects belonging to each dead node, enabling batched deletion.

Concurrent Marker Deletion with Retry Logic

Marker deletion proceeds concurrently with bounded parallelism. The gc_markers function in dead_node_gc.rs (lines 56–90) issues up to 64 concurrent bucket.delete calls, tracking successes and failures separately.

Failed deletions trigger exponential back-off computed by retry_delay_ms in dead_node_reconciliation.rs at lines 15–19. The delay grows as 1× → 2× → 4× … up to 64× the tick interval, preventing tight retry loops when the object store experiences degraded availability.

// Based on dead_node_reconciliation.rs#L15-L19
fn retry_delay_ms(attempt: u32, tick_ms: u64) -> u64 {
    let multiplier = (1u64 << attempt.min(6)).min(64);
    tick_ms * multiplier
}

Retiring Dead Node Records Safely

After all markers are cleared, the GC must remove the dead node's session record without creating a window where the record appears alive. The retire_dead_node function in dead_node_gc.rs (lines 94–121) implements a two-step fence:

1. Conditional update (CAS) writes a tombstone with expires_ms = 0
2. Unconditional delete removes the object entirely

This design guarantees crash safety: if the process fails between steps, the tombstone record still appears dead on the next GC pass and will be retried.

State Management and Periodic Execution

The DeadNodeGc struct maintains local state to optimize repeated passes. As defined at lines 46–55 in dead_node_gc.rs, it tracks:

• swept: Nodes already cleaned in previous passes (avoid redundant work)
• retries: Per-node failure counters to enforce back-off

The run_elected_pass function (lines 58–95) integrates with Celld's waker system. It:

• Renews the waker lease during execution
• Cancels the pass immediately if lease renewal fails
• Returns control to the caller for the next tick interval

Example: Running the Dead Node GC

Below is a minimal Rust example showing how to instantiate and run the garbage collector as part of a waker loop:

use celld::dead_node_gc::DeadNodeGc;
use celld::bucket::Bucket;
use std::time::Duration;
use tokio::time;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Assume `bucket` is a configured Bucket pointing at the shared fleet store.
    let bucket: Bucket = Bucket::new("s3", "celld-fleet-bucket")?;

    // Identifier of the current node (the waker participant)
    let node_id = "node-1234";

    // How often the waker ticks (e.g., 5 seconds)
    let tick_ms = 5_000;

    // The GC is stateful – we keep it around across passes.
    let mut gc = DeadNodeGc::default();

    // Run the GC as part of the regular waker loop.
    let mut interval = time::interval(Duration::from_millis(tick_ms));
    loop {
        interval.tick().await;
        // `run_elected_pass` will early-return if this node has lost the
        // waker lease, otherwise it performs one full dead-node reconciliation.
        gc.run_elected_pass(&bucket, node_id, tick_ms).await?;
    }
}

This pattern demonstrates the intended integration: create a single DeadNodeGc instance, then invoke run_elected_pass on each waker tick. Lease renewal and cancellation handling occur internally—no additional coordination code is required.

Key Source Files

• crates/logic/dead_node_reconciliation.rs — Core I/O-free utilities: parse_marker_key, node_record_is_dead, retry_delay_ms
• crates/celld/dead_node_gc.rs — Main GC implementation: DeadNodeGc, run_elected_pass, marker scanning, deletion, and node retirement
• crates/celld/bucket.rs — Object store abstraction for list, get, put-CAS, and delete operations
• crates/celld/wake.rs — Waker lease management; gates GC execution to the elected leader
• crates/celld/replication.rs — Coordination layer integrating waker and GC passes

Summary

• Dead node detection checks expires_ms against wall-clock time in dead_nodes
• Marker parsing extracts node/cell identifiers while discarding generation suffixes via parse_marker_key
• Concurrent deletion runs up to 64 operations in parallel with exponential back-off on failure
• Safe retirement uses a two-step CAS-then-delete fence to prevent phantom alive records
• State tracking avoids redundant work and limits retry storms through per-node counters
• Leader-only execution ties GC passes to the waker lease, ensuring single-node coordination

Frequently Asked Questions

What triggers dead node garbage collection in Celld?

The GC runs periodically as part of the waker loop while a node holds the fleet-wide waker lease. Each tick_ms interval, run_elected_pass performs one reconciliation sweep if the node remains the elected leader. The lease itself is renewed during GC execution, with automatic cancellation if renewal fails.

How does Celld prevent multiple nodes from running GC simultaneously?

The waker lease mechanism in crates/celld/wake.rs ensures mutual exclusion. Only the node holding the lease invokes run_elected_pass; if lease renewal fails mid-pass, the GC cancels immediately. This integrates GC execution with Celld's existing leader election rather than requiring separate distributed locking.

Why does marker deletion use exponential back-off?

The retry_delay_ms function prevents thundering-herd retry storms when the object store is degraded. Delays scale from 1× to 64× the tick interval, capping maximum backoff. This protects both Celld and the underlying storage from cascading failure during outages.

What happens if the GC crashes while retiring a node record?

The two-step retirement fence in retire_dead_node ensures crash safety. First, a CAS operation writes expires_ms = 0 as a tombstone. Only then does unconditional deletion occur. If the process crashes between steps, the tombstone remains visible to subsequent GC passes, which will retry the full retirement sequence.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}