How Alarm Schedules Persist Across Cell Hibernation and Node Failover in Celld

Celld persists alarm schedules in each cell’s SQLite database, restoring them after hibernation or failover by checking for existing durable wake entries and re-arming only when necessary to ensure at-least-once delivery.

The denoland/celld repository implements a durable alarm system that survives process teardowns and machine crashes. When a cell hibernates or its hosting node fails, the alarm schedule is not lost—it is written to disk and reconstructed by any subsequent node that restores the cell. This article examines the exact persistence mechanism, from SQLite writes to distributed coverage checks.

The Alarm Persistence Architecture

Celld treats alarms as first-class citizens of a cell’s state. Rather than storing timers in ephemeral memory, the system writes alarm metadata—specifically the Unix timestamp in milliseconds (at_ms)—into the cell’s own SQLite file. This design ensures that alarm schedules survive both cell hibernation and node failover because the database file is the single source of truth that moves with the cell.

The architecture separates concerns between storage (SQLite operations), logic (coverage determination), and the JavaScript runtime (re-arming). This separation allows any node in the cluster to resume alarm handling by reading the same on-disk state.

Writing Alarm Schedules to Disk

The set_alarm Implementation

When a Worker script calls setAlarm(), the runtime invokes storage::set_alarm in crates/celld/storage.rs. This function performs an upsert operation on the alarms table, storing the target time, retry counters, and a random generation identifier.

// crates/celld/storage.rs (line 1955)
pub fn set_alarm(&self, at_ms: i64) -> anyhow::Result<()> {
    self.conn.execute(
        "INSERT INTO alarms(scope,at_ms,retry,counted_retry,generation) \
         VALUES(?1,?2,0,0,random()) \
         ON CONFLICT(scope) DO UPDATE SET at_ms=excluded.at_ms, retry=0",
        rusqlite::params![self.scope, at_ms],
    )?;
    Ok(())
}

The ON CONFLICT clause ensures that setting a new alarm overwrites any previous schedule for the same scope, maintaining exactly one active alarm row per cell.

Restoring Alarms After Hibernate or Failover

Reading Persisted State with persisted_alarm

When a cell wakes from hibernation or migrates to a new node, the runtime calls storage::persisted_alarm (lines 2257–2275 in crates/celld/storage.rs). This function queries the SQLite file to retrieve the scheduled time, generation, and retry state.

// crates/celld/storage.rs
let (at_ms, ..) = crate::storage::persisted_alarm(&path.to_string_lossy(), cell)?;

If no alarm exists, the function returns a negative value. Otherwise, it provides the wall-clock time at which the alarm should fire.

The RestoredAlarm Struct

The runtime wraps the retrieved data in a RestoredAlarm struct defined in crates/logic/lib.rs (lines 90–97). This struct carries two critical pieces of information: the at_ms timestamp and a covered boolean indicating whether a durable wake entry already exists for this alarm.

// crates/logic/lib.rs
let restored = (at_ms >= 0).then(|| celld_logic::RestoredAlarm {
    at_ms,
    covered: self.alarm_covered(cell, Some(at_ms)),
});

The covered flag prevents duplicate wake entries across the distributed system, ensuring that only one node owns the alarm at any given time.

Preventing Duplicate Alarms with Coverage Checks

Determining If a Wake Entry Exists

Before re-arming an alarm, Celld must verify whether the replication layer already holds a valid wake entry created by a previous node. The celld_logic::wake::covered method (lines 227–229 in crates/logic/wake.rs) checks the flusher’s state for an existing entry matching the cell and timestamp.

// crates/logic/wake.rs
pub fn covered(&self, cell: &str, next_alarm_ms: Ms) -> bool {
    // Returns true if a durable entry already exists for this alarm
    self.entry_exists(cell, entry_key(next_alarm_ms, cell))
}

If covered returns true, the runtime skips re-arming because the distributed wake system already guarantees delivery.

Re-arming When Necessary

When covered is false, the runtime creates a new durable wake entry. The adopt_wake_entry function in crates/celld/js.rs (line 148) bridges the storage layer to the replication flusher, while spawn_arm_gate (line 182) issues the actual PUT operation to the distributed store.

// crates/celld/js.rs
pub fn adopt_wake_entry(cell: &str, at_ms: i64) {
    gate.flusher.adopt(cell, at_ms);
}

// Later, during the arm operation
let Some(celld_logic::wake::Op::Put { key, due_ms }) = 
    gate.flusher.arm_op(cell, at_ms) else { … };

This two-phase process ensures that alarm schedules are replicated durably before the cell resumes execution, protecting against immediate subsequent crashes.

The Failover Flow Across Nodes

When a node crashes and a different node acquires the cell, the new owner executes the same restoration sequence:

1. Read the alarm row from the SQLite file via persisted_alarm
2. Construct a RestoredAlarm with the retrieved timestamp
3. Check coverage by calling alarm_covered (lines 801–804 in crates/celld/runtime.rs)
4. Adopt or re-arm based on the covered flag

Because the alarm data lives in the cell’s database file—which is accessible to any node that mounts the cell’s storage—the schedule transfers seamlessly. The covered check ensures that even if multiple nodes attempt to restore the cell simultaneously, only one creates a wake entry, avoiding spurious alarm firings.

Summary

• Alarms are stored in SQLite: The set_alarm function writes to the alarms table in the cell’s database file, making schedules durable across hibernation.
• Restoration reads from disk: The persisted_alarm function retrieves the schedule when a cell wakes or fails over to a new node.
• Coverage prevents duplicates: The covered method in crates/logic/wake.rs checks whether a wake entry already exists in the replicated store before re-arming.
• Conditional re-arming: Only uncovered alarms trigger adopt_wake_entry, ensuring exactly-once ownership of wake entries while guaranteeing at-least-once delivery of alarm events.

Frequently Asked Questions

Where are alarm schedules stored in celld?

Alarm schedules are stored in each cell’s individual SQLite database file. Specifically, the alarms table in crates/celld/storage.rs holds the at_ms timestamp, retry counters, and generation ID. This persists the alarm schedule to disk, allowing it to survive process restarts and node failures.

How does celld prevent duplicate alarms after a node failover?

Celld uses a coverage check mechanism. When a cell restores on a new node, the runtime calls covered in crates/logic/wake.rs to verify whether the distributed wake flusher already holds an entry for the alarm timestamp. If an entry exists (covered: true), the runtime does not re-arm the alarm. If no entry exists, it creates one via adopt_wake_entry, ensuring only one node owns the alarm at any time.

What happens to active alarms when a cell hibernates?

When a cell hibernates, its isolate is torn down, but the alarm schedule remains in the SQLite file. Upon restoration, the runtime reads the persisted alarm via persisted_alarm, wraps it in a RestoredAlarm struct, and either adopts the existing wake entry or creates a new one. The alarm fires according to the original schedule unless it expired during hibernation, in which case it triggers immediately upon wake.

Which source files control alarm persistence and restoration?

The primary files involved are:

• crates/celld/storage.rs — Contains set_alarm for writing and persisted_alarm for reading alarm rows
• crates/celld/runtime.rs — Orchestrates restoration logic and alarm_covered checks (lines 801–804)
• crates/logic/wake.rs — Implements the covered method for duplicate detection (lines 227–229)
• crates/celld/js.rs — Bridges to the flusher via adopt_wake_entry and spawn_arm_gate (lines 148, 182)

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}