# How Node Lease Fencing and Epoch Tracking Ensure Cell Ownership Integrity in celld

> Learn how celld uses node lease fencing and epoch tracking with S3 to ensure single node cell ownership, preventing stale writes and maintaining data integrity. Read more!

- Repository: [Deno/celld](https://github.com/denoland/celld)
- Tags: internals
- Published: 2026-08-10

---

**celld guarantees that exactly one node owns a cell at any given moment by combining S3-based node lease fencing with monotonic ownership epoch tracking to reject writes from non-owners or stale epochs.**

The celld distributed cell runtime from Deno implements a strict ownership model to prevent split-brain scenarios and data corruption in multi-node deployments. By leveraging conditional S3 operations for **node lease fencing** and maintaining **ownership epoch counters** in cell metadata, the system ensures that only the rightful owner may modify cell state. This design, implemented primarily in [`crates/celld/ownership_store.rs`](https://github.com/denoland/celld/blob/main/crates/celld/ownership_store.rs), creates a two-layer defense that protects both the process and the persistent data.

## The Two-Layer Defense Mechanism

celld enforces **cell ownership integrity** through complementary mechanisms that guard different layers of the system. The lease protects the node process, while the epoch protects the cell state itself.

### Node Lease Fencing via S3 Conditional Writes

Before a node may write LTX data for any cell, it must hold a live lease stored as a JSON object ([`lock.json`](https://github.com/denoland/celld/blob/main/lock.json)) in an S3-compatible bucket. The acquisition happens through **compare-and-swap (CAS)** operations using ETag-based conditional requests. In [`crates/celld/ownership_store.rs`](https://github.com/denoland/celld/blob/main/crates/celld/ownership_store.rs), the `cas_node_lease` function (lines [14‑38]‑[14‑41]) attempts to write the lease record only if the supplied `CasGuard` matches the current ETag—either `Absent` for a fresh lease or `Match(etag)` for a renewal. If another node has acquired the lease in the interim, the ETag changes and the operation returns `LeaseCasOutcome::Rejected`, immediately fencing the previous owner from further writes.

### Ownership Epoch Tracking

Each cell maintains an **epoch**—a monotonically increasing integer—in `cells/<cell>/own.json`. This value is updated only when a node successfully acquires the lease and writes a new owner record via the `cas_owner` method (lines [30‑99]‑[30‑107]). The write is conditional on the current ETag, ensuring that any attempt to write with an out-of-date epoch fails. When nodes read cell ownership via `read_owner` (lines [16‑76]‑[16‑78]), they obtain both the epoch and its corresponding ETag, establishing the authoritative version for subsequent operations.

## How Lease Acquisition and Epoch Updates Work Together

The coordination between lease fencing and epoch tracking follows a strict sequence enforced by the implementation in [`ownership_store.rs`](https://github.com/denoland/celld/blob/main/ownership_store.rs):

1. **Acquire the node lease.** Upon startup or renewal, the node calls `cas_node_lease` with the current `CasGuard`. The S3 bucket only accepts the write if the ETag matches, ensuring single-writer semantics for the lease object.

2. **Publish ownership with a new epoch.** After securing the lease, the node increments the epoch and calls `cas_owner`, passing the previous ETag as a guard. This conditional write succeeds only if no other node has taken ownership and advanced the epoch.

3. **Maintain liveness through renewal.** Periodic renewals call `cas_node_lease` again to extend the `expires_ms` timestamp. If renewal fails—returning `Rejected`—the node knows it has lost the lease and must stop writing.

4. **Enforce epoch validation.** Every write operation validates the epoch. A node that lost its lease but attempts to write with a stale epoch will be rejected because the epoch guard fails, even if the lease check were somehow bypassed.

## Safety Guarantees in the Implementation

According to the celld source code and [`docs/security.md`](https://github.com/denoland/celld/blob/main/docs/security.md) (lines [9‑12] and [29‑31]), this dual-fence design provides several critical safety properties:

- **Atomic single-writer semantics.** Conditional S3 writes (`put_cas`) eliminate race conditions where two nodes might simultaneously believe they own the same cell.
- **Monotonic epoch progression.** Epoch numbers strictly increase during hand-offs. When a new owner increments the epoch, the previous owner’s epoch becomes permanently invalid.
- **TTL-enforced liveness.** The lease requires periodic renewal; a missed deadline causes expiration, which in turn prevents epoch updates from that node.
- **Decentralized coordination.** celld relies solely on these S3-based fences rather than an external placement service, eliminating single points of failure that could accidentally double-assign cells.

## Practical Implementation Examples

The following patterns from [`crates/celld/ownership_store.rs`](https://github.com/denoland/celld/blob/main/crates/celld/ownership_store.rs) demonstrate the lease and epoch mechanics in Rust.

### Acquiring a Node Lease

```rust
let guard = CasGuard::Absent; // Use Match(etag) for renewals
let record = NodeLeaseRecord {
    node: my_node.clone(),
    expires_ms: now_ms() + lease_ttl,
    addr: my_addr.clone(),
    peer_protocol: 1,
    generation: "gen-1".into(),
};
let outcome = ownership.cas_node_lease(guard, &record).await?;
assert!(matches!(outcome, LeaseCasOutcome::Applied { .. }));

```

This corresponds to `S3Ownership::cas_node_lease` (lines [14‑38]‑[14‑41]) and establishes the process-level fence.

### Writing Ownership with Epoch Protection

```rust
let epoch = current_epoch + 1; // Increment for hand-off or initial claim
let guard = CasGuard::Match(current_etag);
let outcome = ownership.cas_owner(cell_name, guard, epoch).await?;
assert_eq!(outcome, CasOutcome::Applied);

```

Implemented in `S3Ownership::cas_owner` (lines [30‑99]‑[30‑107]), this ensures the cell state is updated only when the epoch matches expectations.

### Handling Lease Loss Gracefully

```rust
match ownership.cas_node_lease(CasGuard::Match(old_etag), &record).await? {
    LeaseCasOutcome::Rejected => {
        // Lease expired or stolen—immediately stop writes
        stop_writing();
    }
    LeaseCasOutcome::Applied { etag } => {
        // Lease renewed—update etag and continue
        current_etag = etag;
    }
}

```

This pattern leverages the same `cas_node_lease` function used for initial acquisition, ensuring consistent fencing behavior across the node lifecycle.

## Summary

- **Node lease fencing** in [`crates/celld/ownership_store.rs`](https://github.com/denoland/celld/blob/main/crates/celld/ownership_store.rs) uses S3 conditional writes to ensure only one node holds the lease for a cell at any time.
- **Ownership epoch tracking** maintains a monotonic counter in `cells/<cell>/own.json` that increments only on successful ownership transfers via `cas_owner`.
- The **two-layer fence** combines process protection (lease) with data protection (epoch), rejecting writes from nodes that lack current leases or present stale epochs.
- **Conditional CAS operations** on ETags provide atomic guarantees without requiring external coordination services.
- The design is documented in [`docs/security.md`](https://github.com/denoland/celld/blob/main/docs/security.md) and supports the lease abstractions defined in [`crates/ltx/src/leaser.rs`](https://github.com/denoland/celld/blob/main/crates/ltx/src/leaser.rs).

## Frequently Asked Questions

### What happens if two nodes attempt to acquire the same lease simultaneously?

Only one node succeeds. The S3 `put_cas` operation in `cas_node_lease` is atomic; the first request with a matching ETag succeeds, while the second receives an ETag mismatch and returns `LeaseCasOutcome::Rejected`. The rejected node is immediately fenced and cannot write to the cell.

### How does celld prevent a node from writing after it loses its lease?

The **ownership epoch** acts as a second fence. Even if a node somehow bypasses lease validation, it must provide the current epoch when calling `cas_owner`. Since the epoch increments in the [`own.json`](https://github.com/denoland/celld/blob/main/own.json) record whenever a new owner takes over, the stale epoch from the old owner causes an immediate rejection.

### Where is the lease TTL enforced and what happens on expiration?

The TTL is enforced by the `expires_ms` field within the [`lock.json`](https://github.com/denoland/celld/blob/main/lock.json) lease object. Other nodes monitoring the lease can observe when `expires_ms` has passed. Additionally, the [`crates/ltx/src/leaser.rs`](https://github.com/denoland/celld/blob/main/crates/ltx/src/leaser.rs) implementation tracks lease duration; once expired, the node can no longer renew via `cas_node_lease`, effectively removing its write privileges.

### Why does celld use both leases and epochs instead of just one mechanism?

The **lease** protects the *process*—it ensures only one node is active at a time. The **epoch** protects the *state*—it persists across lease transitions to ensure that even if lease cleanup is delayed or messages are delayed in transit, no stale node can overwrite current data. This redundancy ensures **cell ownership integrity** survives network partitions and clock skew.