How Cell Activation and Migration Work When Ownership Transfers Between Nodes in celld

celld manages cell lifecycle and ownership transfer through a Phase state machine that queues activation requests, enforces concurrency limits via permits, and preserves epochs during handoffs to enable seamless migration between nodes.

The denoland/celld repository implements a distributed cell runtime where ownership can migrate between nodes without service interruption. Understanding how activation and migration interact requires examining the core state machine, the permit-based admission control, and the handoff protocol that preserves cell identity across nodes.

The Phase State Machine

Every cell in celld exists in one of several Phase states defined in crates/logic/lib.rs. The runtime transitions cells through these phases based on incoming requests, resource pressure, and ownership changes.

Key phases include:

• WaitingActivation — A request arrived but the cell is not yet resident; awaiting a permit to begin activation.
• Restoring — The cell is being restored from a transferred ownership record, either after eviction or handoff.
• Resident — The cell is active and owned by the current node.
• Dormant — The cell was evicted locally but ownership may persist elsewhere.
• Remote — Ownership has definitively moved to another node.

The state machine drives all subsequent I/O: publishing changes, durably proving state, and cleaning up resources.

Cell Activation (Cold-Start Path)

When a request targets a cell that is not resident, celld enters the activation flow. This path is rate-limited by max_activations to prevent thundering herds.

Queuing and Permit Acquisition

The admission logic resides in crates/logic/lib.rs:1268-1275:

//logic.admit_or_queue_activation(id, &mut cell, ColdStart::ReadOwner, effects);
// From main logic/lib.rs:
self.admit_or_queue_activation(id, &mut cell, ColdStart::ReadOwner, effects);
// Afterwards: cell.waiting_activation = Some(start);

Here's a practical example of starting a cold activation:

// Start a cold activation if the cell is inactive
let cell_id = "my-cell".to_string();
let start = std::time::Instant::now();
logic.admit_or_queue_activation(
    &cell_id,
    &mut cell,
    ColdStart::ReadOwner,
    &mut effects,
);

The method admit_or_queue_activation performs three critical operations:

1. Records the activation start timestamp in cell.waiting_activation.
2. If permits are available (activation_permits.len() < config.max_activations), immediately grants one.
3. Otherwise, enqueues the request in activation_waiters for later processing.

Pumping Pending Activations

When permits become free, celld drains the queue at logic/lib.rs:1287-1290:

while self.activation_permits.len() < self.config.max_activations {
    if let Some(waiter) = self.activation_waiters.pop_front() {
        self.activation_permits.insert(waiter.id);
        // ... trigger activation
    }
}

Beginning Activation

Once a permit is obtained, begin_activation at logic/lib.rs:1436-1439 initiates the actual work:

self.begin_activation(id, cell, activation, effects);

The Activation enum specifies the activation mode:

• Activation::Claim — Create a fresh epoch for a new cell.
• Activation::Restore — Resume an existing epoch using RestoreSpec.

RestoreSpec Semantics

The RestoreSpec struct at logic/lib.rs:40-54 encodes how the epoch should be treated:

// Example: Restoring a cell after ownership handoff
let restore = RestoreSpec {
    epoch: current_epoch,
    fresh: false,      // Not a fresh activation
    took_over: true,   // Taken from another node
    resume_local: true,// Can resume on this node
};
logic.begin_activation(cell_id, &mut cell, Activation::Restore(restore), &mut effects);

The resume_local flag is particularly important for migration: it indicates whether the target node can continue the existing epoch without a full cold start.

Ownership Transfer and Migration Handoff

When a node shuts down or sheds load, celld performs a capacity handoff that transfers ownership while preserving the cell's epoch.

Detecting Handoff Requests

The routing layer in crates/celld/main.rs:298-308 identifies handoff traffic:

if capacity_handoff {
    // Handle as capacity acquisition, not normal activation
}

A handoff is signaled by handoff=preserve in the query string, detected at main.rs:3817-3819:

query.split('&').any(|part| part == "handoff=preserve")

Graceful Shutdown Handoff

To perform a graceful handoff, the source node releases ownership records while keeping epochs intact. Example usage:

// Perform a graceful shutdown handoff
let mut request = Request::new(...);
request.query.push_str("&handoff=preserve");
runtime.handle_request(request).await?; // triggers capacity_handoff logic

The release happens at main.rs:1899-1900:

ownership.release_owner(&cell, epoch).await;

This preserves the epoch so the receiving node can acquire it without triggering a fresh activation.

Draining Resident Cells

The handoff loop at logic/lib.rs:334-340 processes pending work in batches:

// "hand-off" loop drains handfuls until no resident cells remain
while let Some(handful) = self.handfuls.pop() {
    // Process batch of cells for migration
}

Final shutdown coordination occurs at main.rs:3232-3240, where the capacity_handoff flag determines cleanup behavior.

Target Node Restoration

The receiving node reads the transferred ownership record:

ownership.read_owner(&cell).await

Based on the record state, it either:

• Restores using RestoreSpec::resume_local — continuing the existing epoch.
• Claims — if the source already released ownership, treating it as a standard activation with inherited epoch.

Interaction Between Activation and Migration

The same infrastructure handles both cold starts and migration-driven restores:

Scenario	Source Node State	Target Node Action
Resident cell handoff	Resident, releases ownership	Reads record, resume_local=true, no cold start
Evicted cell handoff	Dormant (evicted but owned)	Enters Restoring phase with RestoreSpec
Fresh activation	N/A (no prior owner)	Claim with new epoch, subject to max_activations

The activation_waiters queue and activation_permits limit apply universally. Whether from cold start or handoff, concurrent activations cannot exceed max_activations. This prevents handoff storms from overwhelming a receiving node.

Key Source Files and Their Roles

File	Responsibility
crates/logic/lib.rs	Core state machine (Phase), activation queue, handoff drainage, begin_activation
crates/logic/types.rs	RestoreSpec, Activation, Claim definitions
crates/celld/main.rs	Request routing, capacity_handoff detection, ownership release
crates/celld/replication.rs	Low-level handoff implementation for ownership record transfer
crates/celld/storage.rs	Ownership record persistence and safe transfer semantics

Summary

• celld uses a Phase state machine to track cell lifecycle across nodes, with explicit WaitingActivation and Restoring phases for startup and migration.
• Activation permits (max_activations) enforce concurrency limits shared between cold starts and handoff-driven restores.
• Capacity handoff preserves epochs through the handoff=preserve signal and ownership.release_owner, allowing target nodes to resume_local without cold-start penalty.
• RestoreSpec encodes transition semantics: fresh, took_over, and resume_local flags determine how the target node treats the inherited epoch.
• The same queue infrastructure (activation_waiters) handles both organic load spikes and orchestrated migrations, ensuring predictable resource consumption.

Frequently Asked Questions

What happens to in-flight requests during a node shutdown handoff?

Requests are queued in activation_waiters while the handoff loop at logic/lib.rs:334-340 drains cells in batches. Once ownership records are released via ownership.release_owner, subsequent requests route to the new owner node transparently. The source node completes processing of already-resident cells before finalizing shutdown.

How does celld prevent a receiving node from being overwhelmed during mass migration?

The max_activations configuration applies to both cold starts and handoff restores. The permit system at logic/lib.rs:1287-1290 ensures concurrent activations never exceed this ceiling, regardless of whether the activation is organic or migration-driven. Excess handoff requests queue in activation_waiters identical to cold-start requests.

What is the difference between Claim and Restore activation types?

Activation::Claim creates a fresh epoch for a cell with no prior ownership on any node. Activation::Restore resumes an existing epoch using RestoreSpec, which encodes whether the restore is fresh, took_over from another node, or should resume_local. The latter is essential for zero-downtime ownership transfer between nodes.

When does a cell enter the Restoring phase versus WaitingActivation?

WaitingActivation indicates no prior ownership record exists—the cell is starting cold. Restoring indicates the cell has an existing epoch that must be resumed, either from local eviction (Dormant with retained ownership) or from a handoff (ownership transferred from another node). The RestoreSpec determines the specific restoration semantics.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}