How Celld Manages WebSocket Connections in Durable Objects During Ownership Transfer

Celld preserves WebSocket connections during ownership transfers by storing connection metadata in per-cell BTreeMap structures, re-registering outbound sockets via Message::WebSocketOpened events, and re-attaching hibernatable inbound sockets through runtime handlers, ensuring zero-downtime migration across the distributed cluster.

Celld implements a distributed Durable Objects (DO) runtime where WebSocket longevity is directly coupled to cell ownership. When a cell migrates between nodes due to lease expiration or node failure, the system must either migrate active connections to the new owner or terminate them gracefully. This article examines the specific mechanisms in the denoland/celld repository that handle WebSocket state during these ownership transitions, referencing the actual Rust implementation in crates/logic/lib.rs and crates/celld/runtime.rs.

WebSocket State Architecture in Celld

Each cell maintains a websockets field containing a BTreeMap<WebSocketId, WebSocketKind>. Defined in crates/logic/lib.rs, this map tracks three distinct connection types: Regular, Outbound, and Hibernatable. The WebSocketId serves as the unique key, while WebSocketKind determines the migration strategy. Before any transfer operation, the helper function holds_websocket(id, ws_id) validates whether a specific WebSocket belongs to the cell.

Detecting Ownership Changes

Ownership is tracked in an S3-compatible bucket via crates/celld/ownership_store.rs. Nodes periodically invoke read_self_node_lease() to verify their authority. When a lease expires or a new epoch begins, the runtime triggers the ownership-resolution path in crates/logic/lib.rs. This detection mechanism initiates the WebSocket transfer protocol before the cell state migrates to the new node.

Transferring WebSocket Ownership During Cell Migration

The core handoff logic differentiates between connection types to minimize disruption.

Re-registering Outbound WebSockets

Outbound WebSockets (WebSocketKind::Outbound) are pinned to the cell that created them. During transfer, crates/celld/runtime.rs invokes handle_ownership_move (around line 3000). The function attempts to re-register the socket by calling route_websocket_open with the new owner scope. If the target node is unreachable, the runtime emits Effect::CloseWebSocket and propagates Message::WebSocketClosed through the replication layer to inform the former owner.

Re-attaching Hibernatable Inbound Sockets

Hibernatable inbound sockets survive cell hibernation via WebSocketPair storage linked to the cell's persistent state. When ownership moves, the runtime calls app.websocket_opened(scope, websocket, WebSocketKind::Hibernatable) (lines 2740-2745 in runtime.rs). This re-binds the TCP connection to the new isolate's FastWebSockets implementation without dropping the underlying connection or interrupting message flow.

Consistency Validation and Cleanup

Post-transfer, the runtime audits the cell's WebSocketKind entries. Connections not owned by the current node generate Effect::CloseWebSocket events, while valid outbound sockets are retained under the new owner. This audit prevents ghost connections and ensures the distributed cluster maintains a consistent view of active sockets.

Event-Driven Consistency Guarantees

All WebSocket lifecycle events flow through the event-driven core defined by the Message enum in crates/celld/main.rs. This architecture processes ownership updates, WebSocketOpened, and WebSocketClosed messages in total order, eliminating race conditions between hand-off completion and inbound message processing.

Practical Implementation Examples

Opening an Outbound WebSocket

// examples/wsclient/index.js
const socket = new WebSocket(target, protocol ? [protocol] : []);
socket.addEventListener('open', () => console.log('connected'));
socket.addEventListener('message', e => console.log('msg', e.data));

The ws_client.rs implementation creates a FastWebSockets client via WebSocket::after_handshake and registers it with the core as WebSocketKind::Outbound (see lines 104-115).

Handling Hibernatable Server-Side WebSockets

// examples/wsecho/index.js
const pair = new WebSocketPair();
this.state.acceptWebSocket(pair[1]);   // keep one end live
pair[0].addEventListener('message', e => {
  pair[0].send(e.data);                // echo back
});

The state.acceptWebSocket call stores the socket as WebSocketKind::Hibernatable in the cell’s websockets map (runtime.rs, line 2740).

Ownership Handoff Error Handling

// crates/celld/runtime.rs – around line 3000
if let Err(e) = self.route_websocket_open(scope.clone(), websocket, WebSocketKind::Outbound) {
    // owner moved, close the socket
    self.tx.send(Message::WebSocketClosed { cell, websocket }).ok();
}

Summary

• WebSocket state is stored per-cell in crates/logic/lib.rs as a BTreeMap<WebSocketId, WebSocketKind>.
• Ownership changes are detected via read_self_node_lease() in crates/celld/ownership_store.rs.
• Outbound sockets are re-registered with new owners through Message::WebSocketOpened events.
• Hibernatable inbound sockets are re-attached via websocket_opened calls in crates/celld/runtime.rs.
• The Message enum in crates/celld/main.rs ensures ordered event processing to prevent race conditions.

Frequently Asked Questions

What happens to WebSocket connections when a celld node fails?

When a node fails, its cell leases expire in the ownership store. The new owner node detects this via read_self_node_lease(), triggers ownership resolution, and either re-registers outbound sockets or re-attaches hibernatable inbound sockets. If re-registration fails, the system emits Effect::CloseWebSocket to gracefully terminate the connection and notify the peer.

How does celld differentiate between outbound and hibernatable WebSocket types?

The system uses the WebSocketKind enum defined in crates/logic/lib.rs. Outbound represents client-initiated connections pinned to the creating cell, while Hibernatable represents server-side connections that persist through cell hibernation via WebSocketPair storage in the cell's state.

Where is the WebSocket ownership transfer logic implemented?

The core transfer logic resides in crates/celld/runtime.rs, specifically in the handle_ownership_move function (around line 3000) and the route_websocket_open method. Ownership detection occurs in crates/celld/ownership_store.rs, while state structures and the WebSocketKind enum live in crates/logic/lib.rs.

How does celld ensure WebSocket messages aren't lost during ownership transfer?

The runtime uses an event-driven core with a Message enum (defined in crates/celld/main.rs) that processes ownership updates and socket events in total order. This guarantees that ownership hand-off completes before new messages are routed, preventing race conditions and ensuring message delivery consistency across the distributed cluster.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}