Understanding the Security Model for Peer-to-Peer HTTP Communication in celld

The celld peer-to-peer HTTP protocol authenticates requests using shared-secret HMAC-SHA256 signing with strict timestamp validation and replay protection, while deliberately omitting transport-level encryption to be provided by the underlying network infrastructure.

The celld project implements a distributed systems runtime where nodes communicate directly over HTTP. Understanding the security model for peer-to-peer HTTP communication in celld is essential for operators deploying production fleets, as the protocol delegates transport security to external network layers while enforcing cryptographic request authentication through a lightweight, stateful verification mechanism.

Root of Trust and Shared Secret Distribution

All nodes in a celld fleet derive their security identity from a single 32-byte shared secret stored in an S3-compatible object storage bucket. According to the source code in docs/security.md, this secret resides under the key fleet/peer-auth.json and functions as the ultimate authority for the entire fleet.

Ownership of this bucket constitutes administrative privileges. Anyone with credentials to access or modify fleet/peer-auth.json can create, rotate, or compromise the shared secret, making bucket access control the foundational security boundary for the deployment.

HMAC-Based Request Authentication

Each peer-to-peer HTTP request carries authentication metadata through custom x-cells-peer-* headers. The implementation in crates/celld/peer_auth.rs provides the PeerAuth::signed_headers method, which constructs a canonical string containing:

• HTTP method and path
• SHA-256 hash of the request body
• Source and target node identities
• Unix timestamp (milliseconds)
• 16-byte random nonce

This canonical string is then MAC-ed using HMAC-SHA256 with the shared secret. The resulting signature is transmitted in the x-cells-peer-signature header, allowing the receiving node to verify both the authenticity and integrity of the request without session state.

Time-Bounded Replay Protection

The security model incorporates two temporal defense mechanisms defined in crates/celld/peer_auth.rs and crates/logic/peer.rs:

Clock Window Validation: The celld_logic::peer::within_clock_window function enforces that request timestamps must fall within ±30 seconds (CLOCK_WINDOW_MS) of the receiving node's system clock. Requests outside this window are rejected immediately to prevent delayed replay attacks.

Nonce-Based Replay Cache: Each request includes a unique 16-byte nonce tracked in an in-memory cache for at least twice the clock window (REPLAY_RETENTION_MS). If a nonce reappears while still cached, the request returns HTTP 409 Conflict. The cache is capped at 1,000,000 entries to prevent memory exhaustion.

Transport Layer Considerations: No Built-in TLS

Unlike traditional HTTPS-based systems, celld's peer protocol operates exclusively over plain HTTP. As documented in docs/security.md and docs/limitations.md, the project deliberately avoids TLS termination to minimize complexity and binary size.

Operators must provide encryption through alternative means:

• Private subnets or VPC peering
• WireGuard or Tailscale overlays
• External TLS-terminating ingress proxies

This design choice places the confidentiality guarantee entirely within the operator's network infrastructure rather than the application layer.

Request Verification Flow

When a node receives a peer request, the PeerAuth::verify method in crates/celld/peer_auth.rs executes a strict validation pipeline:

1. Protocol Version Check: Validates the x-cells-peer-version header
2. Identity Validation: Verifies source and target identities via valid_identity
3. Timestamp Verification: Ensures the request falls within CLOCK_WINDOW_MS
4. Body Integrity: Compares the provided body hash against the actual payload
5. Signature Verification: Re-computes the HMAC and compares against x-cells-peer-signature
6. Target Matching: Confirms the target identity matches the receiving node
7. Replay Detection: Queries the nonce cache to reject duplicate signatures

Failure at any stage returns specific HTTP status codes: 401 Unauthorized for authentication failures, 409 Conflict for replay attempts, 426 Upgrade Required for version mismatches, or 503 Service Unavailable for internal errors.

Implementation Examples

The following patterns demonstrate the complete lifecycle of peer authentication in celld.

Initializing Peer Authentication

During node startup, load or generate the shared secret using PeerAuth::load_or_create:

use celld::peer_auth::PeerAuth;
use celld::bucket::Bucket;

// Configure your S3-compatible bucket
let bucket = Bucket::new("fleet-bucket", "us-east-1").await?;
let secret = PeerAuth::load_or_create(&bucket).await?;
let auth = PeerAuth::new(secret, "node-01")?;

Signing Outbound Requests

Use PeerAuth::sign to attach authentication headers before transmission:

let client = reqwest::Client::new();
let body = b"{\"msg\":\"hello\"}";
let req = client.post("http://peer-node-02/api/echo")
    .body(body);

let signed = auth.sign(req, "POST", "/api/echo", body, "node-02")?;
let response = signed.send().await?;

Verifying Inbound Requests

Within an Axum handler, validate peer requests using PeerAuth::verify:

use axum::{
    extract::{Request, State},
    http::StatusCode,
};

async fn handler(
    State(auth): State<PeerAuth>,
    req: Request,
) -> Result<String, (StatusCode, String)> {
    let body = hyper::body::to_bytes(req.body())
        .await
        .map_err(|e| (StatusCode::BAD_REQUEST, e.to_string()))?;
    
    auth.verify(
        req.method(),
        req.uri().path(),
        req.headers(),
        &body,
        "node-01",  // expected target identity
    ).map_err(|e| (e.status(), e.message().into()))?;
    
    Ok(String::from_utf8(body.to_vec()).unwrap())
}

Summary

• Shared-secret HMAC: All peer nodes authenticate using a 32-byte secret from S3-compatible storage (fleet/peer-auth.json), with requests signed via HMAC-SHA256 in crates/celld/peer_auth.rs.
• Temporal safeguards: The ±30-second clock window (CLOCK_WINDOW_MS) and nonce replay cache (1,000,000 entries) prevent delayed and replay attacks without requiring persistent session storage.
• No transport encryption: The protocol runs over plain HTTP, requiring operators to secure the network layer through VPNs, private subnets, or TLS-terminating proxies.
• Strict verification: The PeerAuth::verify method enforces canonical signing, body integrity, identity matching, and nonce uniqueness before processing requests.

Frequently Asked Questions

Does celld use TLS for peer-to-peer communication?

No. According to docs/security.md and docs/limitations.md, celld intentionally omits TLS termination from the peer protocol to reduce complexity. Encryption and confidentiality must be provided by the underlying network infrastructure, such as WireGuard tunnels, Tailscale networks, or private cloud subnets.

How does celld prevent replay attacks between nodes?

The implementation uses a combination of strict clock windows and nonce caching. Each request must arrive within ±30 seconds of the receiver's clock (CLOCK_WINDOW_MS), and each 16-byte nonce is tracked in an in-memory cache for at least twice that duration (REPLAY_RETENTION_MS). Duplicate nonces trigger an HTTP 409 Conflict response.

What happens if a node's clock drifts outside the 30-second window?

The celld_logic::peer::within_clock_window function in crates/logic/peer.rs rejects requests with timestamps outside the configured window, returning HTTP 401 Unauthorized. Operators must ensure NTP synchronization across the fleet to prevent legitimate requests from failing validation due to clock skew.

Where is the shared secret stored in a celld deployment?

The 32-byte shared secret is stored in an S3-compatible object storage bucket under the key fleet/peer-auth.json. The PeerAuth::load_or_create method in crates/celld/peer_auth.rs retrieves this secret during node startup. Access to this bucket constitutes administrative control over the entire fleet's security boundary.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}