How celld Implements HMAC-Based Peer-to-Peer Authentication with Clock-Bounded Replay Protection

celld authenticates every inter-node HTTP request using a four-layer defense system that combines a shared fleet secret, HMAC-SHA-256 signing, 30-second clock windows, and an in-memory nonce replay cache.

The denoland/celld distributed database engine secures communication between fleet nodes through a specialized peer-to-peer authentication protocol rather than application-layer TLS. Implemented primarily in crates/celld/peer_auth.rs, this scheme cryptographically binds each request to its contents, timestamp, and unique nonce, ensuring that only nodes possessing the shared bucket credentials can participate in the cluster.

Shared Fleet Secret

At the foundation of celld's security model lies a single shared fleet secret stored in the underlying S3-compatible bucket at fleet/peer-auth.json. This 32-byte random key is generated once and loaded by each node via the load_or_create function.

When a node initializes, it retrieves this secret from object storage, ensuring all participants in the fleet use identical key material for subsequent HMAC operations. This design delegates trust to the bucket's access control policies—only nodes with valid bucket credentials can obtain the signing key.

HMAC-SHA-256 Request Signing

Every outbound request carries a cryptographic signature in the x-cells-peer-signature header computed using HMAC-SHA-256. In crates/celld/peer_auth.rs, the sign method constructs a canonical request string from:

• HTTP method and path
• SHA-256 hash of the request body (sent as x-cells-peer-body-sha256)
• Source node ID and target node ID
• Millisecond timestamp (x-cells-peer-timestamp)
• 16-byte random nonce

The HMAC operates on this canonical representation using the shared fleet secret. The resulting 32-byte MAC is hex-encoded and attached to the request headers, creating a tamper-evident envelope that validates both origin and payload integrity.

Clock-Bounded Freshness

To prevent delayed attacks, celld enforces strict clock-bounded freshness checks using a configurable window defined by CLOCK_WINDOW_MS (default 30 seconds). Upon receiving a request, the verification logic in peer_auth.rs extracts the x-cells-peer-timestamp header and validates it against the local system clock.

If the sender's timestamp falls outside the 30-second window, the receiver immediately rejects the request with an Unauthorized status. This temporal constraint limits the window during which a captured request remains valid, complementing the cryptographic signature with a time-based validity check.

Nonce-Based Replay Protection

Even within the clock window, celld prevents duplicate requests through a replay cache mechanism. Each request includes a unique 16-byte nonce generated via rand::rngs::OsRng.fill_bytes and transmitted in the headers.

The ReplayCache structure maintains an in-memory map of observed nonces to their first-seen timestamps. Entries persist for REPLAY_RETENTION_MS (twice the clock window, or 60 seconds) and the cache enforces a hard limit of MAX_REPLAY_ENTRIES (1,000,000) entries, pruning old records as needed. If verification encounters a nonce already present in the cache, it returns Replay and drops the request.

Practical Implementation

The following examples demonstrate how nodes initialize the authentication layer and secure HTTP traffic.

Loading the shared secret and creating a PeerAuth instance:

use crates::celld::peer_auth::PeerAuth;
use crates::celld::bucket::Bucket;

// `bucket` is an already-configured S3-compatible client
let secret = PeerAuth::load_or_create(&bucket).await?;
let auth = PeerAuth::new(secret, "node-a.internal")?;

Signing an outbound request before dispatch:

let client = reqwest::Client::new();
let req = client.post("http://node-b.internal:8080/some/path")
    .body(b"{\"msg\":\"hello\"}");

let signed_req = auth.sign(req, "POST", "/some/path", b"{\"msg\":\"hello\"}", "node-b.internal")?;
let response = signed_req.send().await?;

Verifying an inbound request within an Axum handler:

async fn handler(
    axum::extract::Extension(auth): axum::extract::Extension<PeerAuth>,
    method: axum::http::Method,
    uri: axum::http::Uri,
    headers: axum::http::HeaderMap,
    body: bytes::Bytes,
) -> Result<impl axum::response::IntoResponse, axum::http::StatusCode> {
    // `expected_target` is this node's advertised address
    auth.verify(&method, uri.path_and_query().map(|p| p.as_str()).unwrap_or("/"),
                &headers, &body, "node-a.internal")
        .map_err(|e| e.status())?;
    // …handle the request…
    Ok(axum::Json(serde_json::json!({ "ok": true })))
}

Network Layer Considerations

According to the project's security documentation (docs/security.md), the peer protocol intentionally does not terminate TLS. Operators must deploy nodes on a trusted private network or encrypted overlay such as WireGuard or Tailscale. This separation of concerns places transport encryption at the network layer while celld handles application-layer authentication and integrity verification.

Summary

• Shared secret: A 32-byte key stored in fleet/peer-auth.json and loaded via load_or_create serves as the root of trust for all nodes.
• HMAC signatures: Every request carries an x-cells-peer-signature header computed via HMAC-SHA-256 over a canonical representation including body hash and node IDs.
• Clock bounds: Requests must arrive within CLOCK_WINDOW_MS (30 seconds) of their claimed timestamp or face immediate rejection.
• Replay cache: A memory-resident ReplayCache tracks nonces for 60 seconds, rejecting duplicate x-cells-peer-nonce values with Replay status.
• Network assumption: The protocol assumes a trusted network path; TLS is not implemented at the application layer.

Frequently Asked Questions

Does celld use TLS for peer-to-peer authentication?

No. As documented in docs/security.md, celld's peer-to-peer authentication operates over plain HTTP. The system assumes operators provide transport security through external means such as private networks, VPNs, or encrypted overlays like WireGuard. The HMAC-based scheme provides authentication and integrity, not confidentiality against network eavesdropping.

How does celld prevent replay attacks?

celld combines two mechanisms: a 30-second clock window (CLOCK_WINDOW_MS) and a nonce replay cache. The receiver rejects requests with timestamps outside the current window. Within that window, a 16-byte random nonce must be unique; the ReplayCache stores nonces for 60 seconds (REPLAY_RETENTION_MS) and rejects any duplicates with a Replay error.

What happens if node clocks drift out of sync?

If a sender's clock deviates by more than CLOCK_WINDOW_MS (30 seconds) from the receiver's clock, the verification logic in crates/celld/peer_auth.rs rejects the request as Unauthorized. This requires operators to maintain reasonable clock synchronization across the fleet, typically via NTP, to prevent legitimate requests from failing validation.

Where is the authentication secret stored?

The shared fleet secret resides in the S3-compatible bucket at fleet/peer-auth.json. Nodes retrieve this 32-byte key on startup using the load_or_create function. Because bucket access controls the ability to retrieve this secret, securing the underlying object storage credentials becomes critical to cluster security.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}