# Understanding the Security Model for Peer-to-Peer HTTP Communication in celld

> Explore the celld security model for peer-to-peer HTTP communication. Learn how HMAC-SHA256 signing, timestamp validation, and replay protection ensure secure data exchange without transport encryption.

- Repository: [Deno/celld](https://github.com/denoland/celld)
- Tags: deep-dive
- Published: 2026-08-09

---

**The celld peer-to-peer HTTP protocol authenticates requests using shared-secret HMAC-SHA256 signing with strict timestamp validation and replay protection, while deliberately omitting transport-level encryption to be provided by the underlying network infrastructure.**

The celld project implements a distributed systems runtime where nodes communicate directly over HTTP. Understanding the security model for peer-to-peer HTTP communication in celld is essential for operators deploying production fleets, as the protocol delegates transport security to external network layers while enforcing cryptographic request authentication through a lightweight, stateful verification mechanism.

## Root of Trust and Shared Secret Distribution

All nodes in a celld fleet derive their security identity from a **single 32-byte shared secret** stored in an S3-compatible object storage bucket. According to the source code in [`docs/security.md`](https://github.com/denoland/celld/blob/main/docs/security.md), this secret resides under the key [`fleet/peer-auth.json`](https://github.com/denoland/celld/blob/main/fleet/peer-auth.json) and functions as the ultimate authority for the entire fleet.

Ownership of this bucket constitutes administrative privileges. Anyone with credentials to access or modify [`fleet/peer-auth.json`](https://github.com/denoland/celld/blob/main/fleet/peer-auth.json) can create, rotate, or compromise the shared secret, making bucket access control the foundational security boundary for the deployment.

## HMAC-Based Request Authentication

Each peer-to-peer HTTP request carries authentication metadata through custom `x-cells-peer-*` headers. The implementation in [`crates/celld/peer_auth.rs`](https://github.com/denoland/celld/blob/main/crates/celld/peer_auth.rs) provides the `PeerAuth::signed_headers` method, which constructs a **canonical string** containing:

- HTTP method and path
- SHA-256 hash of the request body
- Source and target node identities
- Unix timestamp (milliseconds)
- 16-byte random nonce

This canonical string is then MAC-ed using **HMAC-SHA256** with the shared secret. The resulting signature is transmitted in the `x-cells-peer-signature` header, allowing the receiving node to verify both the authenticity and integrity of the request without session state.

## Time-Bounded Replay Protection

The security model incorporates two temporal defense mechanisms defined in [`crates/celld/peer_auth.rs`](https://github.com/denoland/celld/blob/main/crates/celld/peer_auth.rs) and [`crates/logic/peer.rs`](https://github.com/denoland/celld/blob/main/crates/logic/peer.rs):

**Clock Window Validation**: The `celld_logic::peer::within_clock_window` function enforces that request timestamps must fall within **±30 seconds** (`CLOCK_WINDOW_MS`) of the receiving node's system clock. Requests outside this window are rejected immediately to prevent delayed replay attacks.

**Nonce-Based Replay Cache**: Each request includes a unique 16-byte nonce tracked in an in-memory cache for at least **twice the clock window** (`REPLAY_RETENTION_MS`). If a nonce reappears while still cached, the request returns HTTP 409 Conflict. The cache is capped at **1,000,000 entries** to prevent memory exhaustion.

## Transport Layer Considerations: No Built-in TLS

Unlike traditional HTTPS-based systems, celld's peer protocol operates exclusively over **plain HTTP**. As documented in [`docs/security.md`](https://github.com/denoland/celld/blob/main/docs/security.md) and [`docs/limitations.md`](https://github.com/denoland/celld/blob/main/docs/limitations.md), the project deliberately avoids TLS termination to minimize complexity and binary size.

Operators must provide encryption through alternative means:

- **Private subnets** or VPC peering
- **WireGuard** or **Tailscale** overlays
- **External TLS-terminating ingress** proxies

This design choice places the confidentiality guarantee entirely within the operator's network infrastructure rather than the application layer.

## Request Verification Flow

When a node receives a peer request, the `PeerAuth::verify` method in [`crates/celld/peer_auth.rs`](https://github.com/denoland/celld/blob/main/crates/celld/peer_auth.rs) executes a strict validation pipeline:

1. **Protocol Version Check**: Validates the `x-cells-peer-version` header
2. **Identity Validation**: Verifies source and target identities via `valid_identity`
3. **Timestamp Verification**: Ensures the request falls within `CLOCK_WINDOW_MS`
4. **Body Integrity**: Compares the provided body hash against the actual payload
5. **Signature Verification**: Re-computes the HMAC and compares against `x-cells-peer-signature`
6. **Target Matching**: Confirms the target identity matches the receiving node
7. **Replay Detection**: Queries the nonce cache to reject duplicate signatures

Failure at any stage returns specific HTTP status codes: **401 Unauthorized** for authentication failures, **409 Conflict** for replay attempts, **426 Upgrade Required** for version mismatches, or **503 Service Unavailable** for internal errors.

## Implementation Examples

The following patterns demonstrate the complete lifecycle of peer authentication in celld.

### Initializing Peer Authentication

During node startup, load or generate the shared secret using `PeerAuth::load_or_create`:

```rust
use celld::peer_auth::PeerAuth;
use celld::bucket::Bucket;

// Configure your S3-compatible bucket
let bucket = Bucket::new("fleet-bucket", "us-east-1").await?;
let secret = PeerAuth::load_or_create(&bucket).await?;
let auth = PeerAuth::new(secret, "node-01")?;

```

### Signing Outbound Requests

Use `PeerAuth::sign` to attach authentication headers before transmission:

```rust
let client = reqwest::Client::new();
let body = b"{\"msg\":\"hello\"}";
let req = client.post("http://peer-node-02/api/echo")
    .body(body);

let signed = auth.sign(req, "POST", "/api/echo", body, "node-02")?;
let response = signed.send().await?;

```

### Verifying Inbound Requests

Within an Axum handler, validate peer requests using `PeerAuth::verify`:

```rust
use axum::{
    extract::{Request, State},
    http::StatusCode,
};

async fn handler(
    State(auth): State<PeerAuth>,
    req: Request,
) -> Result<String, (StatusCode, String)> {
    let body = hyper::body::to_bytes(req.body())
        .await
        .map_err(|e| (StatusCode::BAD_REQUEST, e.to_string()))?;
    
    auth.verify(
        req.method(),
        req.uri().path(),
        req.headers(),
        &body,
        "node-01",  // expected target identity
    ).map_err(|e| (e.status(), e.message().into()))?;
    
    Ok(String::from_utf8(body.to_vec()).unwrap())
}

```

## Summary

- **Shared-secret HMAC**: All peer nodes authenticate using a 32-byte secret from S3-compatible storage ([`fleet/peer-auth.json`](https://github.com/denoland/celld/blob/main/fleet/peer-auth.json)), with requests signed via HMAC-SHA256 in [`crates/celld/peer_auth.rs`](https://github.com/denoland/celld/blob/main/crates/celld/peer_auth.rs).
- **Temporal safeguards**: The ±30-second clock window (`CLOCK_WINDOW_MS`) and nonce replay cache (1,000,000 entries) prevent delayed and replay attacks without requiring persistent session storage.
- **No transport encryption**: The protocol runs over plain HTTP, requiring operators to secure the network layer through VPNs, private subnets, or TLS-terminating proxies.
- **Strict verification**: The `PeerAuth::verify` method enforces canonical signing, body integrity, identity matching, and nonce uniqueness before processing requests.

## Frequently Asked Questions

### Does celld use TLS for peer-to-peer communication?

No. According to [`docs/security.md`](https://github.com/denoland/celld/blob/main/docs/security.md) and [`docs/limitations.md`](https://github.com/denoland/celld/blob/main/docs/limitations.md), celld intentionally omits TLS termination from the peer protocol to reduce complexity. Encryption and confidentiality must be provided by the underlying network infrastructure, such as WireGuard tunnels, Tailscale networks, or private cloud subnets.

### How does celld prevent replay attacks between nodes?

The implementation uses a combination of **strict clock windows** and **nonce caching**. Each request must arrive within ±30 seconds of the receiver's clock (`CLOCK_WINDOW_MS`), and each 16-byte nonce is tracked in an in-memory cache for at least twice that duration (`REPLAY_RETENTION_MS`). Duplicate nonces trigger an HTTP 409 Conflict response.

### What happens if a node's clock drifts outside the 30-second window?

The `celld_logic::peer::within_clock_window` function in [`crates/logic/peer.rs`](https://github.com/denoland/celld/blob/main/crates/logic/peer.rs) rejects requests with timestamps outside the configured window, returning HTTP 401 Unauthorized. Operators must ensure NTP synchronization across the fleet to prevent legitimate requests from failing validation due to clock skew.

### Where is the shared secret stored in a celld deployment?

The 32-byte shared secret is stored in an S3-compatible object storage bucket under the key [`fleet/peer-auth.json`](https://github.com/denoland/celld/blob/main/fleet/peer-auth.json). The `PeerAuth::load_or_create` method in [`crates/celld/peer_auth.rs`](https://github.com/denoland/celld/blob/main/crates/celld/peer_auth.rs) retrieves this secret during node startup. Access to this bucket constitutes administrative control over the entire fleet's security boundary.