# Celld Peer-to-Peer Communication Protocol: How Deno's Edge Runtime Nodes Sync

> Discover Celld's peer-to-peer communication protocol. Learn how Deno edge runtime nodes sync using a versioned JSON protocol over WebSockets with built-in security.

- Repository: [Deno/celld](https://github.com/denoland/celld)
- Tags: internals
- Published: 2026-08-15

---

**Celld nodes communicate using a custom, versioned JSON protocol serialized through serde and transmitted over WebSocket connections, with authentication and replay protection enforced before any message is processed.**

This article explains the peer-to-peer communication protocol that powers `denoland/celld`, the edge runtime designed for distributed JavaScript and TypeScript deployments. The protocol is deliberately minimal—everything fits in a single file, [`crates/celld/protocol.rs`](https://github.com/denoland/celld/blob/main/crates/celld/protocol.rs), making the system auditable, versionable, and easy to reason about.

## Core Protocol Structures in protocol.rs

The entire inter-node vocabulary lives in [`crates/celld/protocol.rs`](https://github.com/denoland/celld/blob/main/crates/celld/protocol.rs). These are plain data structures with derive macros for `Serialize` and `Deserialize`:

| Struct | Purpose |
|--------|---------|
| **`Manifest`** | Describes a complete deployment: script name, version, modules, assets, feature flags, and raw metadata |
| **`DeployPointer`** | The fleet-wide pointer read on node start-up; drives convergence across all nodes |
| **`Rollout`** | Expresses rollout percentage (e.g., 75% traffic to version v2) |
| **`ModuleRef`** | Individual module metadata: name, size, SHA-256 hash, optional kind |
| **`AssetManifestRef` / `AssetIndex` / `AssetEntry`** | Immutable references to the asset blob store used by workers |
| **`RunWorkerFirst`** | Encodes "run-worker-first" semantics as bool or route list |

No other payload types are exchanged. Everything—from deployment coordination to asset synchronization—uses these structures.

### Serialization and Versioning

All structures use **serde** with JSON as the wire format. This choice is explicit: JSON provides human readability for debugging, ubiquitous library support, and straightforward versioning through additive field changes.

```rust
// Serialize a DeployPointer to send to a peer
use celld::protocol::{DeployPointer, Rollout};
use serde_json::to_string;

// Create a pointer that tells the fleet to roll out 75% of version "v2"
let pointer = DeployPointer {
    script_name: Some("my-app".into()),
    version: "v2".into(),
    prefix: "deploy/my-app/v2".into(),
    rollout: Rollout { percent: 75 },
};

let json = to_string(&pointer).unwrap();
// `json` can now be sent over a WebSocket to a peer node

```

The `prefix` field in `DeployPointer` serves as the storage key prefix, enabling multiple versions to coexist while the pointer atomically shifts traffic.

## WebSocket Transport Layer

Celld nodes establish **WebSocket** connections for all peer-to-peer traffic. The implementation in [`crates/celld/main/websocket.rs`](https://github.com/denoland/celld/blob/main/crates/celld/main/websocket.rs) follows a thin-wrapper philosophy:

1. Read a complete text frame from the socket
2. Deserialize into the appropriate protocol struct
3. Execute the operation
4. Optionally serialize and send a response

This design keeps transport concerns separate from protocol semantics. The WebSocket layer does not parse message contents—it merely delivers JSON payloads to the handler logic.

```rust
// Peer side – receive and deserialize a Manifest
use celld::protocol::Manifest;
use serde_json::from_str;

fn handle_message(msg: &str) {
    // Try to interpret the incoming JSON as a Manifest
    if let Ok(manifest) = from_str::<Manifest>(msg) {
        // Validate required features before applying
        if let Err(e) = celld::protocol::validate_required_features(&manifest.required_features) {
            eprintln!("Unsupported feature: {}", e);
            return;
        }
        // Proceed with deployment logic…
    }
}

```

## Authentication and Replay Protection

Before any JSON payload reaches application logic, the node enforces security through [`crates/logic/peer_auth.rs`](https://github.com/denoland/celld/blob/main/crates/logic/peer_auth.rs) and [`crates/logic/peer.rs`](https://github.com/denoland/celld/blob/main/crates/logic/peer.rs). These modules contain **pure predicates**—functions without side effects that return boolean pass/fail results.

### Identity Validation

The `valid_identity` function in [`crates/logic/peer.rs`](https://github.com/denoland/celld/blob/main/crates/logic/peer.rs) restricts peer identifiers to ASCII alphanumerics plus hyphen, underscore, and period:

```rust
// Authentication check (peer_auth)
use logic::peer_auth::verify;
use logic::peer::{valid_identity, within_clock_window};

let identity = "node-01";
assert!(valid_identity(identity));

```

### Clock Window and Replay Defense

Messages must arrive within a configurable time window, and nonces are tracked to prevent replay attacks:

```rust
let now_ms = 1_700_000_000u64;
let ts_ms = now_ms - 30_000; // 30 seconds ago
assert!(within_clock_window(now_ms, ts_ms, 60_000));

```

The `replay_entry_expired` function handles cache eviction for seen nonces. These checks run in `peer_auth::verify` before the message is accepted.

## Replication Protocol

State synchronization between nodes uses additional message types defined in [`crates/celld/replication.rs`](https://github.com/denoland/celld/blob/main/crates/celld/replication.rs). These extend the base protocol with replication-specific structures while maintaining the same serde-JSON contract.

The replication flow is orchestrated by two modules:

- **[`crates/celld/pool.rs`](https://github.com/denoland/celld/blob/main/crates/celld/pool.rs)** — Manages peer discovery and connection pools
- **[`crates/celld/peer_probe.rs`](https://github.com/denoland/celld/blob/main/crates/celld/peer_probe.rs)** — Handles health-checking and probing of peer nodes

Together they maintain the mesh topology, detect failures, and ensure that `DeployPointer` and `Manifest` updates propagate across the fleet. The replication logic reuses the same WebSocket transport and authentication pipeline—no separate channel is established.

## Protocol Design Philosophy

The celld peer-to-peer communication protocol reflects several deliberate constraints:

- **Single source of truth**: All durable structures in one file ([`protocol.rs`](https://github.com/denoland/celld/blob/main/protocol.rs))
- **No hidden state**: JSON payloads are self-describing and loggable
- **Transport agnostic**: While WebSocket is current, the serde contract could adapt
- **Fail-closed**: Authentication predicates reject on any ambiguity

## Key Implementation Files

| File | Responsibility |
|------|---------------|
| [`crates/celld/protocol.rs`](https://github.com/denoland/celld/blob/main/crates/celld/protocol.rs) | Core JSON contract (`Manifest`, `DeployPointer`, `Rollout`, etc.) |
| [`crates/logic/peer_auth.rs`](https://github.com/denoland/celld/blob/main/crates/logic/peer_auth.rs) | Signature verification and replay protection |
| [`crates/logic/peer.rs`](https://github.com/denoland/celld/blob/main/crates/logic/peer.rs) | Identity and clock-window validation helpers |
| [`crates/celld/replication.rs`](https://github.com/denoland/celld/blob/main/crates/celld/replication.rs) | Replication-specific message definitions |
| [`crates/celld/main/websocket.rs`](https://github.com/denoland/celld/blob/main/crates/celld/main/websocket.rs) | WebSocket transport implementation |
| [`crates/celld/pool.rs`](https://github.com/denoland/celld/blob/main/crates/celld/pool.rs) | Peer connection pool management |
| [`crates/celld/peer_probe.rs`](https://github.com/denoland/celld/blob/main/crates/celld/peer_probe.rs) | Health checking and peer discovery |

## Summary

- Celld nodes use a **custom JSON protocol** defined entirely in [`crates/celld/protocol.rs`](https://github.com/denoland/celld/blob/main/crates/celld/protocol.rs)
- Messages serialize via **serde** and travel over **WebSocket** connections
- **Authentication predicates** in [`peer_auth.rs`](https://github.com/denoland/celld/blob/main/peer_auth.rs) and [`peer.rs`](https://github.com/denoland/celld/blob/main/peer.rs) gate all message processing
- **Replication messages** extend the base protocol in [`replication.rs`](https://github.com/denoland/celld/blob/main/replication.rs), managed by [`pool.rs`](https://github.com/denoland/celld/blob/main/pool.rs) and [`peer_probe.rs`](https://github.com/denoland/celld/blob/main/peer_probe.rs)
- The design prioritizes **auditability, versioning, and fail-closed security**

## Frequently Asked Questions

### What format does celld use for peer-to-peer messages?

Celld uses **JSON** as the wire format, serialized through Rust's serde library. All protocol structures derive `Serialize` and `Deserialize`, making the payload human-readable and easy to version. This is implemented in [`crates/celld/protocol.rs`](https://github.com/denoland/celld/blob/main/crates/celld/protocol.rs).

### How does celld prevent replay attacks between nodes?

The `peer_auth::verify` function in [`crates/logic/peer_auth.rs`](https://github.com/denoland/celld/blob/main/crates/logic/peer_auth.rs) enforces **replay protection** through nonce tracking and timestamp validation. The `replay_entry_expired` helper in [`crates/logic/peer.rs`](https://github.com/denoland/celld/blob/main/crates/logic/peer.rs) manages cache eviction, while `within_clock_window` rejects stale messages outside a configurable tolerance.

### What transport protocol carries celld's peer-to-peer communication?

**WebSocket** is the transport layer, implemented in [`crates/celld/main/websocket.rs`](https://github.com/denoland/celld/blob/main/crates/celld/main/websocket.rs). The WebSocket handler is intentionally thin—it reads text frames, deserializes JSON, and passes structures to application logic without parsing message contents.

### Where is the celld replication protocol defined?

Replication-specific messages reside in [`crates/celld/replication.rs`](https://github.com/denoland/celld/blob/main/crates/celld/replication.rs), using the same serde-JSON contract as the base protocol. The replication **orchestration** lives in [`crates/celld/pool.rs`](https://github.com/denoland/celld/blob/main/crates/celld/pool.rs) (peer discovery and connection management) and [`crates/celld/peer_probe.rs`](https://github.com/denoland/celld/blob/main/crates/celld/peer_probe.rs) (health checking).