# How celld’s S3 Compare-and-Swap Ownership Protocol Prevents Split-Brain Scenarios

> celld's S3 compare-and-swap ownership protocol prevents split-brain by using ETag conditional writes. Only one node atomically claims a cell, ensuring lease validity and epoch consistency.

- Repository: [Deno/celld](https://github.com/denoland/celld)
- Tags: internals
- Published: 2026-08-10

---

**celld's S3 compare-and-swap ownership protocol prevents split-brain scenarios by using ETag-based conditional writes that ensure only one node can atomically claim a cell at any time, rejecting concurrent attempts with `CasOutcome::Rejected` while maintaining lease validity and epoch consistency.**

The denoland/celld project implements distributed cell ownership using an S3-compatible object store as its source of truth. Understanding how celld's S3 compare-and-swap ownership protocol prevents split-brain scenarios requires examining its atomic conditional write mechanisms and lease validation strategies that guarantee single-node ownership even during network partitions.

## The Split-Brain Problem in Distributed Ownership

In distributed systems, split-brain occurs when two nodes simultaneously believe they own the same resource, leading to conflicting writes and data corruption. Traditional locking mechanisms often fail across network partitions, requiring an external source of truth with strong consistency guarantees.

## ETag-Based Conditional Writes

The core protection mechanism resides in `S3Ownership::cas_owner` within [`crates/celld/ownership_store.rs`](https://github.com/denoland/celld/blob/main/crates/celld/ownership_store.rs). This method implements a compare-and-swap operation using S3's conditional PUT semantics based on ETag versioning.

When a node attempts to claim ownership, it first reads the current record to obtain the ETag, then issues a conditional write that succeeds only if the ETag remains unchanged:

```rust
// S3Ownership::cas_owner – conditional write (lines 99-112)
let etag = match &guard {
    CasGuard::Absent => None,
    CasGuard::Match(etag) => Some(etag.as_str()),
};
match self.bucket.put_cas(&key, body, etag).await? {
    Some(_) => Ok(CasOutcome::Applied),
    None => Ok(CasOutcome::Rejected),
}

```

The `CasGuard` enum distinguishes between creating new records (`Absent`) or updating existing ones (`Match`), while `put_cas` in [`crates/celld/bucket.rs`](https://github.com/denoland/celld/blob/main/crates/celld/bucket.rs) wraps the S3 `If-Match` header logic.

## Atomicity Guarantees and Race Condition Prevention

S3 services guarantee that conditional PUT operations are atomic. If two nodes race to write the same ownership record, only one will observe the matching ETag; the other receives `None`, resulting in `CasOutcome::Rejected`.

This atomic rejection prevents the split-brain scenario where both nodes simultaneously believe they acquired the cell:

```rust
let guard = CasGuard::Match(current_etag); // obtained from read_owner()
let outcome = s3_ownership.cas_owner("cell123", guard, new_epoch).await?;
match outcome {
    CasOutcome::Applied => println!("Ownership acquired"),
    CasOutcome::Rejected => println!("Another node won the race"),
}

```

## Lease-Based Liveness Validation

Beyond atomic writes, celld enforces a lease mechanism through `NodeLeaseRecord` defined in [`crates/logic/lib.rs`](https://github.com/denoland/celld/blob/main/crates/logic/lib.rs). Ownership changes are only permitted while the requesting node holds a valid lease.

The `cas_node_lease` method ensures that expired leases automatically invalidate ownership attempts from stale nodes. If a node's lease expires during a network partition, its subsequent ownership claims are rejected regardless of ETag matching, preventing zombie nodes from reclaiming resources after recovering from isolation.

## Epoch Preservation and Safe Release

When releasing ownership, celld maintains strict consistency through `release_owner` (lines 78-85 in [`crates/celld/ownership_store.rs`](https://github.com/denoland/celld/blob/main/crates/celld/ownership_store.rs)). This method preserves the epoch value but clears the owner field only if the current record still matches the releasing node's identity and epoch.

If another node has already claimed the cell via a successful CAS operation, the release fails with `CasOutcome::Rejected`, ensuring the new owner's claim remains intact:

```rust
let result = s3_ownership.release_owner("cell123", current_epoch).await?;
assert_eq!(result, CasOutcome::Applied); // or Rejected if taken elsewhere

```

## Implementation Architecture

The protocol coordination resides in [`crates/celld/protocol.rs`](https://github.com/denoland/celld/blob/main/crates/celld/protocol.rs), which drives ownership requests by invoking the CAS methods. This separation between high-level protocol logic and low-level storage operations in [`ownership_store.rs`](https://github.com/denoland/celld/blob/main/ownership_store.rs) ensures that all ownership transitions pass through the same conditional write validation.

## Summary

- **ETag conditional writes** in `cas_owner` ensure atomic updates that fail if another node modified the record concurrently.
- **CasOutcome::Rejected** provides explicit signaling when race conditions occur, preventing dual ownership.
- **Lease validation** through `NodeLeaseRecord` blocks stale nodes from reclaiming ownership after partitions heal.
- **Epoch preservation** in `release_owner` maintains consistency during ownership transfers, rejecting releases when the cell has changed hands.

## Frequently Asked Questions

### What happens if two nodes attempt to claim the same cell simultaneously?

The S3 service processes only one conditional PUT successfully. The node whose request arrives first receives `CasOutcome::Applied`, while the second receives `CasOutcome::Rejected` because its ETag no longer matches the current object version.

### How does celld handle node failures during ownership acquisition?

If a node fails after reading the ETag but before writing, its lease will eventually expire. Other nodes can then claim the cell using fresh ETags, while the failed node's stale write attempts are rejected due to lease invalidation.

### Why does the release_owner method check the epoch?

The epoch check in `release_owner` ensures that a node only releases a cell it actually owns. If another node has already claimed the cell (changing the epoch), the release is rejected with `CasOutcome::Rejected`, preventing accidental overwriting of valid ownership records.

### Where are the core ownership types defined?

The `CasOutcome` enum, `CasGuard` enum, and `OwnerRecord` struct are defined in [`crates/logic/lib.rs`](https://github.com/denoland/celld/blob/main/crates/logic/lib.rs), while the S3-specific implementation resides in [`crates/celld/ownership_store.rs`](https://github.com/denoland/celld/blob/main/crates/celld/ownership_store.rs) and the bucket abstraction is in [`crates/celld/bucket.rs`](https://github.com/denoland/celld/blob/main/crates/celld/bucket.rs).