How Node Lease Fencing and Epoch Tracking Ensure Cell Ownership Integrity in celld

celld guarantees that exactly one node owns a cell at any given moment by combining S3-based node lease fencing with monotonic ownership epoch tracking to reject writes from non-owners or stale epochs.

The celld distributed cell runtime from Deno implements a strict ownership model to prevent split-brain scenarios and data corruption in multi-node deployments. By leveraging conditional S3 operations for node lease fencing and maintaining ownership epoch counters in cell metadata, the system ensures that only the rightful owner may modify cell state. This design, implemented primarily in crates/celld/ownership_store.rs, creates a two-layer defense that protects both the process and the persistent data.

The Two-Layer Defense Mechanism

celld enforces cell ownership integrity through complementary mechanisms that guard different layers of the system. The lease protects the node process, while the epoch protects the cell state itself.

Node Lease Fencing via S3 Conditional Writes

Before a node may write LTX data for any cell, it must hold a live lease stored as a JSON object (lock.json) in an S3-compatible bucket. The acquisition happens through compare-and-swap (CAS) operations using ETag-based conditional requests. In crates/celld/ownership_store.rs, the cas_node_lease function (lines [14‑38]‑[14‑41]) attempts to write the lease record only if the supplied CasGuard matches the current ETag—either Absent for a fresh lease or Match(etag) for a renewal. If another node has acquired the lease in the interim, the ETag changes and the operation returns LeaseCasOutcome::Rejected, immediately fencing the previous owner from further writes.

Ownership Epoch Tracking

Each cell maintains an epoch—a monotonically increasing integer—in cells/<cell>/own.json. This value is updated only when a node successfully acquires the lease and writes a new owner record via the cas_owner method (lines [30‑99]‑[30‑107]). The write is conditional on the current ETag, ensuring that any attempt to write with an out-of-date epoch fails. When nodes read cell ownership via read_owner (lines [16‑76]‑[16‑78]), they obtain both the epoch and its corresponding ETag, establishing the authoritative version for subsequent operations.

How Lease Acquisition and Epoch Updates Work Together

The coordination between lease fencing and epoch tracking follows a strict sequence enforced by the implementation in ownership_store.rs:

1. Acquire the node lease. Upon startup or renewal, the node calls cas_node_lease with the current CasGuard. The S3 bucket only accepts the write if the ETag matches, ensuring single-writer semantics for the lease object.

2. Publish ownership with a new epoch. After securing the lease, the node increments the epoch and calls cas_owner, passing the previous ETag as a guard. This conditional write succeeds only if no other node has taken ownership and advanced the epoch.

3. Maintain liveness through renewal. Periodic renewals call cas_node_lease again to extend the expires_ms timestamp. If renewal fails—returning Rejected—the node knows it has lost the lease and must stop writing.

4. Enforce epoch validation. Every write operation validates the epoch. A node that lost its lease but attempts to write with a stale epoch will be rejected because the epoch guard fails, even if the lease check were somehow bypassed.

Safety Guarantees in the Implementation

According to the celld source code and docs/security.md (lines [9‑12] and [29‑31]), this dual-fence design provides several critical safety properties:

• Atomic single-writer semantics. Conditional S3 writes (put_cas) eliminate race conditions where two nodes might simultaneously believe they own the same cell.
• Monotonic epoch progression. Epoch numbers strictly increase during hand-offs. When a new owner increments the epoch, the previous owner’s epoch becomes permanently invalid.
• TTL-enforced liveness. The lease requires periodic renewal; a missed deadline causes expiration, which in turn prevents epoch updates from that node.
• Decentralized coordination. celld relies solely on these S3-based fences rather than an external placement service, eliminating single points of failure that could accidentally double-assign cells.

Practical Implementation Examples

The following patterns from crates/celld/ownership_store.rs demonstrate the lease and epoch mechanics in Rust.

Acquiring a Node Lease

let guard = CasGuard::Absent; // Use Match(etag) for renewals
let record = NodeLeaseRecord {
    node: my_node.clone(),
    expires_ms: now_ms() + lease_ttl,
    addr: my_addr.clone(),
    peer_protocol: 1,
    generation: "gen-1".into(),
};
let outcome = ownership.cas_node_lease(guard, &record).await?;
assert!(matches!(outcome, LeaseCasOutcome::Applied { .. }));

This corresponds to S3Ownership::cas_node_lease (lines [14‑38]‑[14‑41]) and establishes the process-level fence.

Writing Ownership with Epoch Protection

let epoch = current_epoch + 1; // Increment for hand-off or initial claim
let guard = CasGuard::Match(current_etag);
let outcome = ownership.cas_owner(cell_name, guard, epoch).await?;
assert_eq!(outcome, CasOutcome::Applied);

Implemented in S3Ownership::cas_owner (lines [30‑99]‑[30‑107]), this ensures the cell state is updated only when the epoch matches expectations.

Handling Lease Loss Gracefully

match ownership.cas_node_lease(CasGuard::Match(old_etag), &record).await? {
    LeaseCasOutcome::Rejected => {
        // Lease expired or stolen—immediately stop writes
        stop_writing();
    }
    LeaseCasOutcome::Applied { etag } => {
        // Lease renewed—update etag and continue
        current_etag = etag;
    }
}

This pattern leverages the same cas_node_lease function used for initial acquisition, ensuring consistent fencing behavior across the node lifecycle.

Summary

• Node lease fencing in crates/celld/ownership_store.rs uses S3 conditional writes to ensure only one node holds the lease for a cell at any time.
• Ownership epoch tracking maintains a monotonic counter in cells/<cell>/own.json that increments only on successful ownership transfers via cas_owner.
• The two-layer fence combines process protection (lease) with data protection (epoch), rejecting writes from nodes that lack current leases or present stale epochs.
• Conditional CAS operations on ETags provide atomic guarantees without requiring external coordination services.
• The design is documented in docs/security.md and supports the lease abstractions defined in crates/ltx/src/leaser.rs.

Frequently Asked Questions

What happens if two nodes attempt to acquire the same lease simultaneously?

Only one node succeeds. The S3 put_cas operation in cas_node_lease is atomic; the first request with a matching ETag succeeds, while the second receives an ETag mismatch and returns LeaseCasOutcome::Rejected. The rejected node is immediately fenced and cannot write to the cell.

How does celld prevent a node from writing after it loses its lease?

The ownership epoch acts as a second fence. Even if a node somehow bypasses lease validation, it must provide the current epoch when calling cas_owner. Since the epoch increments in the own.json record whenever a new owner takes over, the stale epoch from the old owner causes an immediate rejection.

Where is the lease TTL enforced and what happens on expiration?

The TTL is enforced by the expires_ms field within the lock.json lease object. Other nodes monitoring the lease can observe when expires_ms has passed. Additionally, the crates/ltx/src/leaser.rs implementation tracks lease duration; once expired, the node can no longer renew via cas_node_lease, effectively removing its write privileges.

Why does celld use both leases and epochs instead of just one mechanism?

The lease protects the process—it ensures only one node is active at a time. The epoch protects the state—it persists across lease transitions to ensure that even if lease cleanup is delayed or messages are delayed in transit, no stale node can overwrite current data. This redundancy ensures cell ownership integrity survives network partitions and clock skew.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}