Celld Peer-to-Peer Communication Protocol: How Deno's Edge Runtime Nodes Sync

Celld nodes communicate using a custom, versioned JSON protocol serialized through serde and transmitted over WebSocket connections, with authentication and replay protection enforced before any message is processed.

This article explains the peer-to-peer communication protocol that powers denoland/celld, the edge runtime designed for distributed JavaScript and TypeScript deployments. The protocol is deliberately minimal—everything fits in a single file, crates/celld/protocol.rs, making the system auditable, versionable, and easy to reason about.

Core Protocol Structures in protocol.rs

The entire inter-node vocabulary lives in crates/celld/protocol.rs. These are plain data structures with derive macros for Serialize and Deserialize:

Struct	Purpose
Manifest	Describes a complete deployment: script name, version, modules, assets, feature flags, and raw metadata
DeployPointer	The fleet-wide pointer read on node start-up; drives convergence across all nodes
Rollout	Expresses rollout percentage (e.g., 75% traffic to version v2)
ModuleRef	Individual module metadata: name, size, SHA-256 hash, optional kind
AssetManifestRef / AssetIndex / AssetEntry	Immutable references to the asset blob store used by workers
RunWorkerFirst	Encodes "run-worker-first" semantics as bool or route list

No other payload types are exchanged. Everything—from deployment coordination to asset synchronization—uses these structures.

Serialization and Versioning

All structures use serde with JSON as the wire format. This choice is explicit: JSON provides human readability for debugging, ubiquitous library support, and straightforward versioning through additive field changes.

// Serialize a DeployPointer to send to a peer
use celld::protocol::{DeployPointer, Rollout};
use serde_json::to_string;

// Create a pointer that tells the fleet to roll out 75% of version "v2"
let pointer = DeployPointer {
    script_name: Some("my-app".into()),
    version: "v2".into(),
    prefix: "deploy/my-app/v2".into(),
    rollout: Rollout { percent: 75 },
};

let json = to_string(&pointer).unwrap();
// `json` can now be sent over a WebSocket to a peer node

The prefix field in DeployPointer serves as the storage key prefix, enabling multiple versions to coexist while the pointer atomically shifts traffic.

WebSocket Transport Layer

Celld nodes establish WebSocket connections for all peer-to-peer traffic. The implementation in crates/celld/main/websocket.rs follows a thin-wrapper philosophy:

1. Read a complete text frame from the socket
2. Deserialize into the appropriate protocol struct
3. Execute the operation
4. Optionally serialize and send a response

This design keeps transport concerns separate from protocol semantics. The WebSocket layer does not parse message contents—it merely delivers JSON payloads to the handler logic.

// Peer side – receive and deserialize a Manifest
use celld::protocol::Manifest;
use serde_json::from_str;

fn handle_message(msg: &str) {
    // Try to interpret the incoming JSON as a Manifest
    if let Ok(manifest) = from_str::<Manifest>(msg) {
        // Validate required features before applying
        if let Err(e) = celld::protocol::validate_required_features(&manifest.required_features) {
            eprintln!("Unsupported feature: {}", e);
            return;
        }
        // Proceed with deployment logic…
    }
}

Authentication and Replay Protection

Before any JSON payload reaches application logic, the node enforces security through crates/logic/peer_auth.rs and crates/logic/peer.rs. These modules contain pure predicates—functions without side effects that return boolean pass/fail results.

Identity Validation

The valid_identity function in crates/logic/peer.rs restricts peer identifiers to ASCII alphanumerics plus hyphen, underscore, and period:

// Authentication check (peer_auth)
use logic::peer_auth::verify;
use logic::peer::{valid_identity, within_clock_window};

let identity = "node-01";
assert!(valid_identity(identity));

Clock Window and Replay Defense

Messages must arrive within a configurable time window, and nonces are tracked to prevent replay attacks:

let now_ms = 1_700_000_000u64;
let ts_ms = now_ms - 30_000; // 30 seconds ago
assert!(within_clock_window(now_ms, ts_ms, 60_000));

The replay_entry_expired function handles cache eviction for seen nonces. These checks run in peer_auth::verify before the message is accepted.

Replication Protocol

State synchronization between nodes uses additional message types defined in crates/celld/replication.rs. These extend the base protocol with replication-specific structures while maintaining the same serde-JSON contract.

The replication flow is orchestrated by two modules:

• crates/celld/pool.rs — Manages peer discovery and connection pools
• crates/celld/peer_probe.rs — Handles health-checking and probing of peer nodes

Together they maintain the mesh topology, detect failures, and ensure that DeployPointer and Manifest updates propagate across the fleet. The replication logic reuses the same WebSocket transport and authentication pipeline—no separate channel is established.

Protocol Design Philosophy

The celld peer-to-peer communication protocol reflects several deliberate constraints:

• Single source of truth: All durable structures in one file (protocol.rs)
• No hidden state: JSON payloads are self-describing and loggable
• Transport agnostic: While WebSocket is current, the serde contract could adapt
• Fail-closed: Authentication predicates reject on any ambiguity

Key Implementation Files

File	Responsibility
crates/celld/protocol.rs	Core JSON contract (Manifest, DeployPointer, Rollout, etc.)
crates/logic/peer_auth.rs	Signature verification and replay protection
crates/logic/peer.rs	Identity and clock-window validation helpers
crates/celld/replication.rs	Replication-specific message definitions
crates/celld/main/websocket.rs	WebSocket transport implementation
crates/celld/pool.rs	Peer connection pool management
crates/celld/peer_probe.rs	Health checking and peer discovery

Summary

• Celld nodes use a custom JSON protocol defined entirely in crates/celld/protocol.rs
• Messages serialize via serde and travel over WebSocket connections
• Authentication predicates in peer_auth.rs and peer.rs gate all message processing
• Replication messages extend the base protocol in replication.rs, managed by pool.rs and peer_probe.rs
• The design prioritizes auditability, versioning, and fail-closed security

Frequently Asked Questions

What format does celld use for peer-to-peer messages?

Celld uses JSON as the wire format, serialized through Rust's serde library. All protocol structures derive Serialize and Deserialize, making the payload human-readable and easy to version. This is implemented in crates/celld/protocol.rs.

How does celld prevent replay attacks between nodes?

The peer_auth::verify function in crates/logic/peer_auth.rs enforces replay protection through nonce tracking and timestamp validation. The replay_entry_expired helper in crates/logic/peer.rs manages cache eviction, while within_clock_window rejects stale messages outside a configurable tolerance.

What transport protocol carries celld's peer-to-peer communication?

WebSocket is the transport layer, implemented in crates/celld/main/websocket.rs. The WebSocket handler is intentionally thin—it reads text frames, deserializes JSON, and passes structures to application logic without parsing message contents.

Where is the celld replication protocol defined?

Replication-specific messages reside in crates/celld/replication.rs, using the same serde-JSON contract as the base protocol. The replication orchestration lives in crates/celld/pool.rs (peer discovery and connection management) and crates/celld/peer_probe.rs (health checking).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}