How celld Prevents Split‑Brain Scenarios During Network Partitioning

celld prevents split‑brain scenarios by implementing a lease‑based detector that uses monotonically increasing version counters and atomic compare‑and‑swap operations on a shared durable store, ensuring only one node can hold leadership even during network partitions.

The denoland/celld distributed storage system is designed to maintain strict consistency and availability across clustered nodes. By leveraging a lease‑based split‑brain detector backed by atomic operations on shared storage, celld guarantees that network partitions cannot result in divergent leadership states that compromise data integrity.

The Lease‑Based Split‑Brain Detector

According to the security documentation in docs/security.md, celld employs a lease‑based split‑brain detector that relies on a monotonically increasing lease counter. Each node maintains a local lease version and persists lease state to a shared, durable key‑value store (such as S3, MinIO, or a replicated local filesystem directory) that is accessible to all cluster members.

When a node starts, it reads the current lease value from this shared store. If the stored version exceeds the node’s local version, the node immediately relinquishes leadership and transitions to a follower state. This ensures that stale nodes cannot assert authority over the cluster.

Atomic Compare‑and‑Set Operations

To prevent race conditions during leader election, the system utilizes the underlying storage's atomic compare‑and‑swap (CAS) semantics. In crates/celld/src/replication.rs, the try_acquire_lease method attempts to increment the lease version only if the current stored version matches the node's expected value. This atomic guarantee ensures that when two nodes simultaneously attempt to acquire leadership, exactly one will succeed, eliminating the possibility of dual leadership.

Handling Network Partitions

During a network partition, the cluster may split into majority and minority segments. The lease mechanism ensures safe degradation:

Minority Partition Behavior

Nodes on the minority side of a partition will fail to renew their lease within the configured lease_timeout_ms (default: 5000ms). Upon expiration, these nodes automatically step down and become read‑only replicas. They continue to serve read requests from local cache but reject all write operations until they successfully reacquire the lease, preventing divergent writes during the partition.

Recovery and State Synchronization

When the partition heals, minority nodes detect the newer lease version held by the majority partition. As implemented in the try_acquire_lease method, these nodes read the updated lease value, recognize their local version is stale, and synchronize any divergent state before resuming normal write operations. This automatic reconciliation ensures linearizable consistency without manual intervention.

Implementation Details in crates/celld/src/replication.rs

The core logic resides in the Replication struct, which manages lease acquisition and renewal through two primary methods.

The try_acquire_lease method implements the CAS logic:

pub async fn try_acquire_lease(&mut self) -> bool {
    // Read the current lease version from the shared store.
    let current = self.lease_store.read_lease().await.unwrap_or(0);
    if current > self.lease_version {
        // Another node has a newer lease – step down.
        self.lease_version = current;
        self.lease_expiry = self.runtime.now() + Duration::from_secs(5);
        return false;
    }
    // Increment our version and attempt an atomic write (CAS).
    let new_version = self.lease_version + 1;
    if self.lease_store.compare_and_set_lease(current, new_version).await {
        self.lease_version = new_version;
        self.lease_expiry = self.runtime.now() + Duration::from_secs(5);
        true
    } else {
        // Lost the race – another node won.
        false
    }
}

The lease_renew_task method handles periodic renewal:

pub async fn lease_renew_task(mut self) {
    loop {
        self.runtime.sleep(self.ping_interval).await;
        if self.runtime.now() >= self.lease_expiry {
            // Lease expired – try to reacquire.
            let acquired = self.try_acquire_lease().await;
            if acquired {
                log::info!("Acquired lease version {}", self.lease_version);
            } else {
                log::warn!("Failed to acquire lease");
            }
        }
    }
}

Verification Through Integration Testing

The split‑brain prevention logic is validated in crates/ltx/tests/integration_resilience.rs. The test_split_brain_prevention test simulates two nodes racing to acquire the same lease:

#[tokio::test]
async fn test_split_brain_prevention() {
    // Simulate two nodes with a shared lease store.
    let store = OwnershipStore::new_temp();
    let rt1 = Runtime::new();
    let rt2 = Runtime::new();
    let mut rep1 = Replication::new(rt1.handle(), store.clone());
    let mut rep2 = Replication::new(rt2.handle(), store.clone());

    // Both nodes attempt to acquire the lease simultaneously.
    let f1 = tokio::spawn(async move { rep1.try_acquire_lease().await });
    let f2 = tokio::spawn(async move { rep2.try_acquire_lease().await });
    let (a1, a2) = tokio::join!(f1, f2);

    // Exactly one node should become the leader.
    assert!(a1.unwrap() ^ a2.unwrap(), "Only one leader allowed");
}

This test confirms that the CAS mechanism guarantees mutually exclusive leadership, preventing split‑brain scenarios even under race conditions.

Configuration Parameters

Operators can tune lease behavior through three key settings defined in the security documentation:

• lease_timeout_ms (default: 5000): The maximum duration a leader can operate without renewing its lease. Shorter values increase safety but require more frequent renewals.
• lease_renew_interval_ms (default: 1000): The frequency at which the leader attempts to renew its lease. This should be significantly shorter than the timeout to account for network jitter.
• lease_store: The URI or path specifying the shared storage backend (e.g., s3://bucket/lease, /mnt/shared/lease).

These parameters allow deployment‑specific balancing between availability and partition tolerance.

Summary

• celld prevents split‑brain scenarios through a lease‑based detector using monotonic version counters stored in a shared durable store.
• Atomic compare‑and‑swap operations in crates/celld/src/replication.rs ensure that only one node can successfully acquire or renew the lease at any time.
• During network partitions, minority nodes automatically step down when their leases expire, rejecting writes until connectivity is restored.
• The system provides linearizable consistency and automatic recovery without manual intervention when partitions heal.
• Integration tests in crates/ltx/tests/integration_resilience.rs verify that simultaneous lease acquisition attempts result in exactly one leader.

Frequently Asked Questions

What storage backends are supported for the lease store?

celld supports any durable key‑value store that provides atomic compare‑and‑swap semantics. Commonly used backends include Amazon S3, MinIO, and replicated local filesystem directories. The OwnershipStore abstraction in the source code handles the underlying storage interface.

How does celld handle clock skew between nodes?

The lease mechanism relies on monotonic version counters rather than absolute timestamps for correctness. While lease_timeout_ms uses wall‑clock time for expiration checks locally, the authoritative lease state is determined by the version number stored in the shared backend, making the system resilient to moderate clock skew between nodes.

Can the lease timeout be adjusted for high‑latency networks?

Yes. The lease_timeout_ms and lease_renew_interval_ms parameters can be tuned based on network conditions. For high‑latency environments, increasing the timeout prevents unnecessary failovers, while decreasing it improves detection speed of failed nodes. The renewal interval should typically be set to one‑fifth or less of the timeout value.

What happens if the shared lease store becomes unavailable?

If the shared lease store becomes unreachable, nodes will be unable to read or renew leases. Current leaders will step down when their leases expire, and the cluster will enter a read‑only state until storage connectivity is restored. This design prioritizes consistency over availability during storage failures, preventing potential split‑brain conditions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}