What Happens When a Node Fails or Is Evicted from a celld Fleet: A Technical Deep Dive

When a celld node fails or is evicted, the fleet's background Dead-Node GC reclaims its lease and cell markers from object storage, while voluntary eviction publishes cells as unowned for other nodes to acquire—no separate consensus layer required.

The celld project implements a bucket-coordinated distributed system where nodes share state through a single S3-compatible or GCS bucket rather than direct communication. Understanding node failure and eviction behavior is critical for operating production deployments. This article examines the exact mechanisms implemented in denoland/celld for handling node lifecycle transitions.

How celld Tracks Node Health with Leases

Every active node in a celld fleet maintains a node lease stored at nodes/<node>.json in the shared bucket. This lease contains an expiration timestamp that serves as the node's proof of life.

The lease mechanism enables failure detection without heartbeats between nodes. Each node periodically refreshes its own lease; if a node crashes or becomes partitioned, its lease expires and becomes dead—still present in the bucket but no longer valid.

Node Failure Detection and Dead-Node GC

When a node fails, the Dead-Node GC task handles cleanup. This background process is implemented in crates/celld/dead_node_gc.rs.

The GC scans the nodes/ prefix and calls node_record_is_dead to identify expired leases. Upon finding a dead node, it executes two critical operations:

1. Deletes the stale lease record from nodes/<node>.json
2. Reclaims all cell markers stored under node-cells/<node>/… (the marker GC step)

The completion of this process produces a log entry: "dead-node marker GC complete", as found at lines 55-62 of dead_node_gc.rs.

You can manually trigger this detection using the diagnostic command:

celld diagnose --bucket s3://my-cells-bucket

This command enumerates all node leases, displays their TTL status, and identifies dead nodes. When dead nodes are found, the tool triggers the dead-node GC to reclaim their cells.

Voluntary Cell Eviction and the Hand-Off Protocol

Nodes may voluntarily evict cells during memory pressure or planned maintenance. This process is implemented in crates/logic/lib.rs through the begin_eviction function.

The eviction flow works as follows:

• The node records its eviction intent in internal state
• The cell is inserted into eviction_permits (lines 350-357)
• The cell is published as unowned (or ownership is released based on ownership_on_evict configuration)

Once published, any node in the fleet may acquire the cell by writing a new lease record. This bucket-driven ownership protocol eliminates the need for leader election or consensus algorithms.

For testing or operational purposes, you can force eviction via CLI:

celld evict my-app::my-cell-id --bucket s3://my-cells-bucket

This invokes the same begin_eviction path used internally, making the cell immediately available for acquisition.

Graceful Node Shutdown and Draining

When a node initiates graceful shutdown—triggered by signals, configuration changes, or CELLD_MAX_RESIDENT_CELLS thresholds—it enters a draining state.

The draining logic, found in crates/logic/lib.rs (lines 336-338), implements the following behavior:

• Sets the internal draining flag (nodes "never go back to serving" once draining)
• Stops accepting new work
• Finishes in-flight requests
• Releases all owned cells through the same eviction path

This ensures clean hand-off of workload without dropped requests. Configure drain behavior with environment variables:

CELLD_MAX_RESIDENT_CELLS=0 CELLD_SHUTDOWN_DRAIN_MS=30000 \
celld --bucket s3://my-cells-bucket --listen 0.0.0.0:8080 \
      --internal-listen 10.0.0.12:8081 --advertise node-a.internal:8081

The CELLD_SHUTDOWN_DRAIN_MS value controls maximum time to wait for in-flight work completion before forced exit.

Resulting System State After Node Transitions

After failure or eviction processing completes, the celld fleet reaches a consistent state:

Scenario	Bucket State	Cell Availability
Dead node GC completes	Lease deleted, markers removed	All cells unowned, claimable by any node
Voluntary eviction	Original ownership cleared	Specific cell unowned, immediately claimable
Graceful drain	Node lease expires naturally	All cells released progressively

Key Source Files and Structures

The ownership and eviction semantics rely on structures defined across these locations:

• crates/logic/types.rs: Defines node: Option<NodeId> in lease structures, eviction configuration, and ownership_on_evict semantics
• crates/logic/lib.rs: Core eviction logic, eviction_permits maintenance, and draining flag handling
• crates/celld/dead_node_gc.rs: Dead node detection and marker reclamation
• README.md (How it works section): Documents the bucket-only coordination model

Summary

• Node failure is detected through lease expiration in shared object storage, not inter-node heartbeats
• Dead-Node GC (dead_node_gc.rs) automatically reclaims failed nodes' cells by deleting stale leases and cleaning node-cells/ markers
• Voluntary eviction (begin_eviction in logic/lib.rs) publishes cells as unowned for immediate acquisition by healthy nodes
• Graceful shutdown uses the draining flag to finish in-flight work before releasing all cells
• All state transitions use the bucket as source of truth, giving celld self-healing properties without external coordination

Frequently Asked Questions

How long does celld wait before considering a node dead?

The exact timeout depends on the node's lease TTL configuration. The Dead-Node GC scans continuously and detects death immediately when node_record_is_dead finds an expired timestamp. No additional grace period is applied—expired leases are treated as dead on the next GC cycle.

Can two nodes simultaneously claim the same cell during eviction?

No. Cell acquisition depends on atomic writes to object storage. The bucket's consistency model ensures that only one node succeeds in writing the ownership record. Failed writers receive a conflict and retry, discovering the new owner.

What happens to in-flight requests when a node is evicted?

Requests targeting an evicted cell are routed based on the latest bucket state. If the cell is unowned, the request triggers acquisition by the receiving node. If ownership transferred to another node, the request is forwarded. The draining flag ensures nodes complete in-flight work before releasing cells, minimizing disruption.

Is there a risk of data loss during node failure?

celld's Durable Object cells are backed by persistent storage in the shared bucket. Cell state is not stored solely on the node—the node only holds a lease and potentially cached data. When the Dead-Node GC reclaims markers, the authoritative cell data remains in bucket storage for the next owner to access.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}