How Celld Handles Cell Migration and Ownership Transfer Between Nodes

Celld uses an epoch-based compare-and-swap (CAS) lease mechanism stored in a cloud bucket to atomically transfer cell ownership between nodes, ensuring that only one node can write to a SQLite database replica at any time while supporting graceful handoffs and automatic failover.

In the denoland/celld distributed database system, each cell is a lightweight SQLite database file that must be served by exactly one node at any given moment. The architecture implements a robust ownership transfer protocol using versioned leases and atomic bucket operations to enable seamless cell migration between nodes without data loss or split-brain scenarios.

The Ownership Record and Epoch-Based Leases

Every cell in celld maintains an ownership record in a cloud-compatible bucket that acts as the source of truth for which node holds the active lease. This record is versioned by a monotonically increasing epoch number (a u64), creating a linear history of ownership changes.

A node acquires responsibility for a cell by performing three core operations defined in crates/celld/ownership_store.rs:

• read_owner – Reads the current ownership record and epoch from the bucket.
• cas_owner – Attempts a compare-and-swap to claim the next epoch for the cell.
• release_owner – Explicitly releases the lease so another node can acquire the next epoch.

When a node successfully executes cas_owner, it gains an exclusive right to write to the cell for that epoch. The node also maintains a node-lease via read_self_node_lease to prove liveness before the epoch expires, preventing stale nodes from holding locks indefinitely.

The Cell Migration Workflow

Cell migration follows a strict five-phase protocol that ensures durability and consistency across the distributed fleet.

Detecting the Need to Migrate

Migration triggers fall into three categories: graceful shutdowns, load-shedding decisions, or automated failover. The dead_node_gc.rs module continuously monitors node leases and their expiration timestamps (checked against now_ms()). When a lease expires, the garbage collector identifies the stranded cells and initiates a takeover by a healthy node.

Releasing the Current Lease

During a graceful handoff, the current owner calls ownership.release_owner(&cell, epoch) in main.rs to surrender its claim. This operation updates the bucket record to indicate the epoch is complete, allowing the next owner to begin acquisition. In failure scenarios, the lease simply expires without an explicit release.

Acquiring the New Epoch

The new owner candidate reads the current state via read_owner, then attempts to claim the next epoch using cas_owner. If the CAS succeeds, the node atomically becomes the new owner; if another node has already incremented the epoch, the CAS fails and the candidate must retry with the updated state. This operation is defined in main.rs as ownership.cas_owner(&cell, guard, epoch).

Restoring the Database Replica

Upon successful lease acquisition, the new owner must establish a local SQLite instance. In replication.rs, the sqlite_snapshot function creates a consistent point-in-time snapshot of the database without disrupting the WAL. The snapshot is retrieved from the bucket path cells/<cell>/ltx/e<epoch>/db.sqlite, where the epoch prefix ensures the node reads the latest committed state from the previous owner.

Publishing the New Epoch

Finally, the new owner writes its replica back to the bucket under the new epoch prefix at cells/<cell>/ltx/e<epoch>/. This epoch-embedded path structure acts as a data-path fence, guaranteeing that a stale owner cannot overwrite newer data even if it attempts to write after a network partition heals.

The ActivationOptions Struct and Takeover Semantics

The ActivationOptions struct in replication.rs controls how a node initializes a cell replica after acquiring a lease:

pub struct ActivationOptions {
    pub cell: String,
    pub epoch: u64,
    pub fresh: bool,
    pub took_over: bool,
}

The took_over boolean is critical for distinguishing migration scenarios. When set to true, it indicates the node is taking over from a different node (a true migration), requiring a full snapshot restore. When false, the activation represents a process restart where the previous lease still names the same node (epoch-1), allowing the system to skip unnecessary writes if the local state remains authoritative.

Safety Guarantees and Failure Handling

Celld’s ownership transfer mechanism provides three core guarantees essential for distributed correctness:

• Atomicity – All ownership changes are performed via CAS operations on the bucket, ensuring that at most one node can own a cell for any given epoch, eliminating split-brain scenarios.
• Durability – By embedding the epoch in the storage path (ltx/e<epoch>/), the system creates an immutable lineage of cell states. A node holding an expired lease cannot corrupt newer data because its write targets a superseded path.
• Fail-over – Node leases include expiration timestamps. If a node crashes while holding a cell, its lease eventually expires, allowing the dead node garbage collector in dead_node_gc.rs to safely trigger a takeover by another member of the fleet. The fleet.rs module manages node-lease traffic independently to ensure renewal operations do not block ordinary ownership checks.

Code Example: Implementing a Manual Takeover

The following Rust example demonstrates the essential API for migrating a cell between nodes, combining ownership operations with activation:

use celld::ownership_store::Ownership;
use celld::replication::{ActivationOptions, ActivationResult};

async fn migrate_cell(
    ownership: &Ownership,
    cell: &str,
    new_epoch: u64,
) -> anyhow::Result<ActivationResult> {
    // 1️⃣ Read current owner & epoch
    let current = ownership.read_owner(cell).await?;
    
    // 2️⃣ Attempt to acquire lease for the next epoch
    let guard = ownership.cas_owner(cell, current.guard, new_epoch).await?;
    
    // 3️⃣ If we got the lease, activate the cell (marking it as a takeover)
    let activation = ActivationOptions {
        cell: cell.to_string(),
        epoch: new_epoch,
        fresh: false,
        took_over: true, // Indicates migration from a dead node
    };
    
    // 4️⃣ Perform activation – restores snapshot if needed
    let result = ownership.activate(activation, guard).await?;
    Ok(result)
}

This snippet follows the exact pattern used internally: reading the owner, performing a CAS to obtain the lease, and activating the cell with took_over = true to trigger the migration path.

Summary

• Celld stores ownership records in a cloud bucket with epoch versioning, using CAS operations to guarantee atomic ownership transfers.
• Migration requires releasing the current lease (or waiting for expiration), acquiring the next epoch via cas_owner, and restoring a snapshot from cells/<cell>/ltx/e<epoch>/db.sqlite.
• The took_over flag in ActivationOptions distinguishes between process restarts and true inter-node migrations, optimizing for the common case of same-node recovery.
• Safety is ensured through lease expirations (enforced by dead_node_gc.rs), epoch-prefixed storage paths, and the atomicity guarantees of the underlying bucket CAS operations.

Frequently Asked Questions

What happens if a node crashes during an active cell migration?

If the original owner crashes after releasing its lease but before the new owner completes activation, the system remains consistent because the lease has been surrendered. If the new owner crashes mid-migration, its partial activation state is discarded; the dead node garbage collector in dead_node_gc.rs will eventually detect the expired node lease and trigger a new takeover attempt by a healthy node, starting the CAS process fresh from the last committed epoch.

How does celld prevent two nodes from simultaneously believing they own the same cell?

The compare-and-swap mechanism in ownership_store.rs ensures that only one node can successfully increment the epoch for a given cell. Because all nodes must perform cas_owner to claim ownership, and because bucket storage provides atomic CAS semantics, concurrent attempts result in exactly one winner. The losing node receives a CAS failure and must retry, at which point it discovers the new epoch and owner.

What is the difference between a fresh activation and a takeover?

A fresh activation occurs when a cell is being instantiated for the first time or after all data has been explicitly cleared; the fresh flag is true. A takeover (migration) occurs when a node acquires ownership of an existing cell from a previous owner, either gracefully or via failure recovery; the took_over flag is true. This distinction allows the replication layer to optimize the activation path—avoiding unnecessary snapshot restores when a node is simply restarting and its local SQLite file remains authoritative.

How does the dead node garbage collector decide when to trigger ownership transfers?

The dead_node_gc.rs module monitors node leases stored in the bucket and compares their expiration timestamps against the current time (now_ms()). When a lease expires without renewal, the module identifies all cells owned by that node and initiates takeover procedures. This process runs independently of the main request path to ensure that temporary latency spikes do not block ownership operations managed by fleet.rs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}