# How celld Handles Network Partitions and S3 Bucket Availability Issues

> celld tackles network partitions and S3 availability issues using CAS semantics, lease expiration, and signed peer probes for automatic failover. Learn how celld ensures data consistency and reliability.

- Repository: [Deno/celld](https://github.com/denoland/celld)
- Tags: how-to-guide
- Published: 2026-08-10

---

**celld handles network partitions and S3 bucket availability issues by using a single S3-compatible bucket as the sole source of truth, implementing compare-and-swap (CAS) semantics for all coordination data, and relying on lease expiration and signed peer probes to detect failures and trigger automatic failovers.**

The `denoland/celld` distributed system coordinates nodes through an S3-centric architecture that treats a single object bucket as the **sole source of truth** for all cluster state. By storing ownership records, node leases, and placement information as JSON objects and enforcing strict **compare-and-swap (CAS)** semantics on every write, celld maintains consistency even when facing **network partitions and S3 bucket availability** disruptions. This design allows nodes to safely lose and regain connectivity without data loss or split-brain scenarios.

## Detecting and Recovering from Network Partitions

### Signed Peer Probes for Reachability Testing

Each node publishes a `probe_public_key` in its lease record within the S3 bucket. Other nodes perform signed HTTP probes to the `/__celld/probe` endpoint to verify peer reachability. The probing logic resides in **[peer_probe.rs](https://github.com/denoland/celld/blob/main/crates/celld/peer_probe.rs)**. If a probe fails due to HTTP 5xx errors or missing challenge responses, the target node is marked unreachable for that check cycle, triggering the partition handling logic.

### Lease Expiration as a Failover Mechanism

Every node lease contains an `expires_ms` timestamp that the core logic (`celld_logic`) monitors continuously. When network partitions isolate a node, its lease naturally expires once the current time exceeds `expires_ms`. Other nodes then become eligible to acquire the cell by writing a new lease with a fresh epoch. The lease lifecycle is managed in **[ownership_store.rs](https://github.com/denoland/celld/blob/main/crates/celld/ownership_store.rs)** through functions like `read_self_node_lease`, `cas_node_lease`, and `release_owner`.

### Filtering Stale Capacity Records

To prevent acting on data written before a partition occurred, celld filters peer records by recency. The `read_capacity_peers` function discards any record whose last-modified timestamp is older than three lease-TTL windows (with a minimum floor of 60 seconds). This logic is implemented in the `capacity_record_is_recent` helper within **[ownership_store.rs](https://github.com/denoland/celld/blob/main/crates/celld/ownership_store.rs)**, ensuring nodes only coordinate with recently active peers.

### Deterministic State Restoration

Because every cell's SQLite database is continuously replicated to the S3 bucket, a node that regains connectivity after a partition can restore the latest snapshot from object storage. The `restore` module in **[logic/restore.rs](https://github.com/denoland/celld/blob/main/crates/logic/restore.rs)** handles this reconciliation, guaranteeing that no writes are lost even if temporary ownership loss occurred during the network split.

## Managing S3 Bucket Availability Issues

### Atomic Conditional Writes Against Split-Brain

All writes to the bucket use the `put_cas` API provided by the `Bucket` abstraction. If the underlying S3 service returns a **412 Precondition Failed** error (indicating an ETag mismatch), celld treats this as a normal rejection rather than a fatal error. The core logic either retries or falls back to a no-op. This behavior is documented in **[ownership_store.rs](https://github.com/denoland/celld/blob/main/crates/celld/ownership_store.rs)** lines 78-90 (`release_owner`) and lines 98-111 (`cas_owner`).

### Graceful Degradation on Transient Failures

When S3 operations fail for reasons other than 412 errors—such as network timeouts or 5xx responses—the error propagates as an `anyhow::Error`. The calling logic interprets any non-412 error as **"ambiguous"**, meaning the operation's effect is unknown and must be retried later. For example, `cas_node_lease` (lines 19-23) explicitly verifies that a probe key exists before attempting the write, ensuring nodes never publish incomplete lease records if the bucket is temporarily unavailable.

### Recency Checks for Read Operations

The `read_capacity_peers` function (lines 13-30) implements read-through caching with strict recency filters, listing only objects that meet the freshness criteria. This prevents nodes from relying on partially written or corrupted bucket state that might exist during an S3 outage window.

### Exponential Backoff via object_store

Low-level S3 operations in **[bucket.rs](https://github.com/denoland/celld/blob/main/crates/celld/bucket.rs)** (`Bucket::get`, `Bucket::put_cas`, `Bucket::list`) leverage the `object_store` crate, which implements exponential backoff and automatic retries for transient network errors. This ensures temporary connectivity glitches do not cause permanent coordination data loss.

### Operational Fallback During Extended Outages

If the S3 bucket is completely unreachable during startup, celld fails fast via `S3Ownership::from_environment` and aborts the run. This prevents nodes from operating with an inconsistent view of the cluster. Once connectivity restores, the node automatically rejoins the fleet by reconciling its state against the latest bucket contents.

## Failure Scenario Guarantees

| Failure Scenario | celld Response |
|------------------|----------------|
| **Peer unreachable (network partition)** | Probe fails → lease expires → other nodes may acquire the cell. |
| **S3 write conflict (ETag mismatch)** | Treated as normal rejection; core retries later. |
| **Transient S3/network error (5xx, timeout)** | Propagated as ambiguous error → operation retried with backoff. |
| **Extended S3 outage** | Node stays offline until bucket recovers; fails fast on startup. |
| **Stale bucket records** | Filtered by `capacity_record_is_recent` recency checks. |
| **Node restart** | Reads latest lease and snapshot from bucket; restores ownership. |

## Implementation Code Examples

The following Rust snippets demonstrate how celld's API handles coordination failures:

```rust
// Attempt to acquire ownership of a cell (CAS write)
let guard = CasGuard::Absent; // no previous lease
let outcome = s3_ownership.cas_owner("my-cell", guard, new_epoch).await?;
match outcome {
    CasOutcome::Applied => println!("Ownership acquired."),
    CasOutcome::Rejected => println!("Another node beat us – retry later."),
}

```

```rust
// Publish a node lease with a signed probe key
let lease = NodeLeaseRecord {
    node: my_node.clone(),
    addr: my_addr.clone(),
    expires_ms: now_ms() + lease_ttl,
    peer_protocol: PROTOCOL_VERSION,
    generation: s3_ownership.process_generation().unwrap_or("").into(),
    // … other fields omitted …
};
let guard = CasGuard::Match(previous_etag);
let lease_outcome = s3_ownership.cas_node_lease(guard, &lease).await?;
assert!(matches!(lease_outcome, LeaseCasOutcome::Applied { .. }));

```

## Key Source Files

| File | Role |
|------|------|
| **[peer_probe.rs](https://github.com/denoland/celld/blob/main/crates/celld/peer_probe.rs)** | Implements signed HTTP probes for peer reachability testing. |
| **[ownership_store.rs](https://github.com/denoland/celld/blob/main/crates/celld/ownership_store.rs)** | Manages S3-backed lease reads, ownership records, and CAS writes. |
| **[bucket.rs](https://github.com/denoland/celld/blob/main/crates/celld/bucket.rs)** | Thin wrapper around `object_store` for S3 operations with retry logic. |
| **[logic/restore.rs](https://github.com/denoland/celld/blob/main/crates/logic/restore.rs)** | Handles restoration of SQLite snapshots from the bucket after partitions. |
| **[protocol.rs](https://github.com/denoland/celld/blob/main/crates/celld/protocol.rs)** | Defines JSON structures for leases and ownership records stored in S3. |

## Summary

- celld uses **compare-and-swap (CAS)** semantics on a single S3 bucket to prevent split-brain during network partitions.
- **Lease expiration** (`expires_ms`) and **signed peer probes** (`/__celld/probe`) automatically detect failed nodes and trigger failover.
- The `capacity_record_is_recent` filter ensures nodes ignore stale data that predates a partition.
- **412 Precondition Failed** responses from S3 are handled as normal rejections, while other errors trigger ambiguous-state retries.
- The `object_store` crate provides exponential backoff for transient S3 availability issues.
- Nodes fail fast on startup if the bucket is unreachable, ensuring no operation proceeds with inconsistent state.

## Frequently Asked Questions

### What happens if two nodes try to acquire the same cell during a network partition?

The S3 bucket acts as the arbiter through **compare-and-swap** operations. Only one node will successfully write the new lease with a matching ETag; the other receives a 412 Precondition Failed error and must retry. This prevents split-brain scenarios regardless of network topology.

### How does celld distinguish between a dead node and a partitioned node?

It doesn't distinguish explicitly. Instead, celld relies on **lease expiration** (`expires_ms`). If a node fails to renew its lease before expiration due to a partition, other nodes treat it as dead and can acquire its cells. If the partitioned node rejoins, it discovers its expired lease and enters recovery mode.

### Can celld operate if the S3 bucket is completely unavailable?

No. celld requires the S3 bucket as the sole source of truth. If the bucket is unreachable during startup, the node fails fast via `S3Ownership::from_environment`. During runtime, transient outages trigger retries with backoff, but extended outages cause the node to remain offline until connectivity restores.

### How are SQLite databases recovered after a prolonged network partition?

Each cell's SQLite database is continuously replicated to the S3 bucket. When a node restarts or rejoins after a partition, the `restore` module in **logic/restore.rs** fetches the latest snapshot from the bucket, ensuring the node resumes with the most recent state regardless of local data loss.