# What Happens When a Node Fails or Is Evicted from a celld Fleet: A Technical Deep Dive

> Discover what happens when a celld node fails or is evicted. Learn how the fleet reclaims resources and handles cell ownership without a consensus layer.

- Repository: [Deno/celld](https://github.com/denoland/celld)
- Tags: deep-dive
- Published: 2026-08-15

---

**When a celld node fails or is evicted, the fleet's background Dead-Node GC reclaims its lease and cell markers from object storage, while voluntary eviction publishes cells as unowned for other nodes to acquire—no separate consensus layer required.**

The celld project implements a **bucket-coordinated distributed system** where nodes share state through a single S3-compatible or GCS bucket rather than direct communication. Understanding node failure and eviction behavior is critical for operating production deployments. This article examines the exact mechanisms implemented in `denoland/celld` for handling node lifecycle transitions.

## How celld Tracks Node Health with Leases

Every active node in a celld fleet maintains a **node lease** stored at `nodes/<node>.json` in the shared bucket. This lease contains an expiration timestamp that serves as the node's proof of life.

The lease mechanism enables failure detection without heartbeats between nodes. Each node periodically refreshes its own lease; if a node crashes or becomes partitioned, its lease expires and becomes **dead**—still present in the bucket but no longer valid.

## Node Failure Detection and Dead-Node GC

When a node fails, the **Dead-Node GC** task handles cleanup. This background process is implemented in [`crates/celld/dead_node_gc.rs`](https://github.com/denoland/celld/blob/main/crates/celld/dead_node_gc.rs).

The GC scans the `nodes/` prefix and calls `node_record_is_dead` to identify expired leases. Upon finding a dead node, it executes two critical operations:

1. **Deletes the stale lease record** from `nodes/<node>.json`
2. **Reclaims all cell markers** stored under `node-cells/<node>/…` (the marker GC step)

The completion of this process produces a log entry: `"dead-node marker GC complete"`, as found at lines 55-62 of [`dead_node_gc.rs`](https://github.com/denoland/celld/blob/main/dead_node_gc.rs).

You can manually trigger this detection using the diagnostic command:

```bash
celld diagnose --bucket s3://my-cells-bucket

```

This command enumerates all node leases, displays their TTL status, and identifies dead nodes. When dead nodes are found, the tool triggers the dead-node GC to reclaim their cells.

## Voluntary Cell Eviction and the Hand-Off Protocol

Nodes may **voluntarily evict** cells during memory pressure or planned maintenance. This process is implemented in [`crates/logic/lib.rs`](https://github.com/denoland/celld/blob/main/crates/logic/lib.rs) through the `begin_eviction` function.

The eviction flow works as follows:

- The node records its **eviction intent** in internal state
- The cell is inserted into `eviction_permits` (lines 350-357)
- The cell is **published as unowned** (or ownership is released based on `ownership_on_evict` configuration)

Once published, any node in the fleet may acquire the cell by writing a new lease record. This **bucket-driven ownership protocol** eliminates the need for leader election or consensus algorithms.

For testing or operational purposes, you can force eviction via CLI:

```bash
celld evict my-app::my-cell-id --bucket s3://my-cells-bucket

```

This invokes the same `begin_eviction` path used internally, making the cell immediately available for acquisition.

## Graceful Node Shutdown and Draining

When a node initiates **graceful shutdown**—triggered by signals, configuration changes, or `CELLD_MAX_RESIDENT_CELLS` thresholds—it enters a **draining** state.

The draining logic, found in [`crates/logic/lib.rs`](https://github.com/denoland/celld/blob/main/crates/logic/lib.rs) (lines 336-338), implements the following behavior:

- Sets the internal `draining` flag (nodes "never go back to serving" once draining)
- Stops accepting new work
- Finishes in-flight requests
- **Releases all owned cells** through the same eviction path

This ensures clean hand-off of workload without dropped requests. Configure drain behavior with environment variables:

```bash
CELLD_MAX_RESIDENT_CELLS=0 CELLD_SHUTDOWN_DRAIN_MS=30000 \
celld --bucket s3://my-cells-bucket --listen 0.0.0.0:8080 \
      --internal-listen 10.0.0.12:8081 --advertise node-a.internal:8081

```

The `CELLD_SHUTDOWN_DRAIN_MS` value controls maximum time to wait for in-flight work completion before forced exit.

## Resulting System State After Node Transitions

After failure or eviction processing completes, the celld fleet reaches a consistent state:

| Scenario | Bucket State | Cell Availability |
|----------|-----------|-----------------|
| Dead node GC completes | Lease deleted, markers removed | All cells unowned, claimable by any node |
| Voluntary eviction | Original ownership cleared | Specific cell unowned, immediately claimable |
| Graceful drain | Node lease expires naturally | All cells released progressively |

## Key Source Files and Structures

The ownership and eviction semantics rely on structures defined across these locations:

- **[`crates/logic/types.rs`](https://github.com/denoland/celld/blob/main/crates/logic/types.rs)**: Defines `node: Option<NodeId>` in lease structures, eviction configuration, and `ownership_on_evict` semantics
- **[`crates/logic/lib.rs`](https://github.com/denoland/celld/blob/main/crates/logic/lib.rs)**: Core eviction logic, `eviction_permits` maintenance, and `draining` flag handling
- **[`crates/celld/dead_node_gc.rs`](https://github.com/denoland/celld/blob/main/crates/celld/dead_node_gc.rs)**: Dead node detection and marker reclamation
- **[`README.md`](https://github.com/denoland/celld/blob/main/README.md)** (How it works section): Documents the bucket-only coordination model

## Summary

- **Node failure** is detected through **lease expiration** in shared object storage, not inter-node heartbeats
- **Dead-Node GC** ([`dead_node_gc.rs`](https://github.com/denoland/celld/blob/main/dead_node_gc.rs)) automatically reclaims failed nodes' cells by deleting stale leases and cleaning `node-cells/` markers
- **Voluntary eviction** (`begin_eviction` in [`logic/lib.rs`](https://github.com/denoland/celld/blob/main/logic/lib.rs)) publishes cells as unowned for immediate acquisition by healthy nodes
- **Graceful shutdown** uses the **draining flag** to finish in-flight work before releasing all cells
- All state transitions use the **bucket as source of truth**, giving celld self-healing properties without external coordination

## Frequently Asked Questions

### How long does celld wait before considering a node dead?

The exact timeout depends on the node's lease TTL configuration. The Dead-Node GC scans continuously and detects death immediately when `node_record_is_dead` finds an expired timestamp. No additional grace period is applied—expired leases are treated as dead on the next GC cycle.

### Can two nodes simultaneously claim the same cell during eviction?

No. Cell acquisition depends on **atomic writes to object storage**. The bucket's consistency model ensures that only one node succeeds in writing the ownership record. Failed writers receive a conflict and retry, discovering the new owner.

### What happens to in-flight requests when a node is evicted?

Requests targeting an evicted cell are routed based on the **latest bucket state**. If the cell is unowned, the request triggers acquisition by the receiving node. If ownership transferred to another node, the request is forwarded. The `draining` flag ensures nodes complete in-flight work before releasing cells, minimizing disruption.

### Is there a risk of data loss during node failure?

celld's **Durable Object cells** are backed by persistent storage in the shared bucket. Cell state is not stored solely on the node—the node only holds a **lease** and potentially cached data. When the Dead-Node GC reclaims markers, the authoritative cell data remains in bucket storage for the next owner to access.