How celld’s S3 Compare-and-Swap Ownership Protocol Prevents Split-Brain Scenarios

celld's S3 compare-and-swap ownership protocol prevents split-brain scenarios by using ETag-based conditional writes that ensure only one node can atomically claim a cell at any time, rejecting concurrent attempts with CasOutcome::Rejected while maintaining lease validity and epoch consistency.

The denoland/celld project implements distributed cell ownership using an S3-compatible object store as its source of truth. Understanding how celld's S3 compare-and-swap ownership protocol prevents split-brain scenarios requires examining its atomic conditional write mechanisms and lease validation strategies that guarantee single-node ownership even during network partitions.

The Split-Brain Problem in Distributed Ownership

In distributed systems, split-brain occurs when two nodes simultaneously believe they own the same resource, leading to conflicting writes and data corruption. Traditional locking mechanisms often fail across network partitions, requiring an external source of truth with strong consistency guarantees.

ETag-Based Conditional Writes

The core protection mechanism resides in S3Ownership::cas_owner within crates/celld/ownership_store.rs. This method implements a compare-and-swap operation using S3's conditional PUT semantics based on ETag versioning.

When a node attempts to claim ownership, it first reads the current record to obtain the ETag, then issues a conditional write that succeeds only if the ETag remains unchanged:

// S3Ownership::cas_owner – conditional write (lines 99-112)
let etag = match &guard {
    CasGuard::Absent => None,
    CasGuard::Match(etag) => Some(etag.as_str()),
};
match self.bucket.put_cas(&key, body, etag).await? {
    Some(_) => Ok(CasOutcome::Applied),
    None => Ok(CasOutcome::Rejected),
}

The CasGuard enum distinguishes between creating new records (Absent) or updating existing ones (Match), while put_cas in crates/celld/bucket.rs wraps the S3 If-Match header logic.

Atomicity Guarantees and Race Condition Prevention

S3 services guarantee that conditional PUT operations are atomic. If two nodes race to write the same ownership record, only one will observe the matching ETag; the other receives None, resulting in CasOutcome::Rejected.

This atomic rejection prevents the split-brain scenario where both nodes simultaneously believe they acquired the cell:

let guard = CasGuard::Match(current_etag); // obtained from read_owner()
let outcome = s3_ownership.cas_owner("cell123", guard, new_epoch).await?;
match outcome {
    CasOutcome::Applied => println!("Ownership acquired"),
    CasOutcome::Rejected => println!("Another node won the race"),
}

Lease-Based Liveness Validation

Beyond atomic writes, celld enforces a lease mechanism through NodeLeaseRecord defined in crates/logic/lib.rs. Ownership changes are only permitted while the requesting node holds a valid lease.

The cas_node_lease method ensures that expired leases automatically invalidate ownership attempts from stale nodes. If a node's lease expires during a network partition, its subsequent ownership claims are rejected regardless of ETag matching, preventing zombie nodes from reclaiming resources after recovering from isolation.

Epoch Preservation and Safe Release

When releasing ownership, celld maintains strict consistency through release_owner (lines 78-85 in crates/celld/ownership_store.rs). This method preserves the epoch value but clears the owner field only if the current record still matches the releasing node's identity and epoch.

If another node has already claimed the cell via a successful CAS operation, the release fails with CasOutcome::Rejected, ensuring the new owner's claim remains intact:

let result = s3_ownership.release_owner("cell123", current_epoch).await?;
assert_eq!(result, CasOutcome::Applied); // or Rejected if taken elsewhere

Implementation Architecture

The protocol coordination resides in crates/celld/protocol.rs, which drives ownership requests by invoking the CAS methods. This separation between high-level protocol logic and low-level storage operations in ownership_store.rs ensures that all ownership transitions pass through the same conditional write validation.

Summary

• ETag conditional writes in cas_owner ensure atomic updates that fail if another node modified the record concurrently.
• CasOutcome::Rejected provides explicit signaling when race conditions occur, preventing dual ownership.
• Lease validation through NodeLeaseRecord blocks stale nodes from reclaiming ownership after partitions heal.
• Epoch preservation in release_owner maintains consistency during ownership transfers, rejecting releases when the cell has changed hands.

Frequently Asked Questions

What happens if two nodes attempt to claim the same cell simultaneously?

The S3 service processes only one conditional PUT successfully. The node whose request arrives first receives CasOutcome::Applied, while the second receives CasOutcome::Rejected because its ETag no longer matches the current object version.

How does celld handle node failures during ownership acquisition?

If a node fails after reading the ETag but before writing, its lease will eventually expire. Other nodes can then claim the cell using fresh ETags, while the failed node's stale write attempts are rejected due to lease invalidation.

Why does the release_owner method check the epoch?

The epoch check in release_owner ensures that a node only releases a cell it actually owns. If another node has already claimed the cell (changing the epoch), the release is rejected with CasOutcome::Rejected, preventing accidental overwriting of valid ownership records.

Where are the core ownership types defined?

The CasOutcome enum, CasGuard enum, and OwnerRecord struct are defined in crates/logic/lib.rs, while the S3-specific implementation resides in crates/celld/ownership_store.rs and the bucket abstraction is in crates/celld/bucket.rs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}