HMAC-Based Peer Authentication in Deno celld: How Cluster Nodes Securely Communicate

The celld daemon uses HMAC-SHA256 signatures with canonical request strings, timestamps, nonces, and replay caches to authenticate HTTP-style peer requests without centralized state.

The HMAC-based peer authentication mechanism in the Deno celld project secures intra-cluster communication between nodes. Unlike TLS mutual authentication or OAuth flows, this scheme relies on a pre-shared 32-byte secret and cryptographic signing of every request. The implementation spans crates/celld/peer_auth.rs and crates/logic/peer.rs, providing stateless verification with replay protection.

---

How Key Provisioning Works

Before any signing or verification occurs, nodes must share a common secret. The celld system stores this in the HexFleet bucket at fleet/peer-auth.json.

The PeerAuth::load_or_create and load_existing methods handle this in crates/celld/peer_auth.rs at lines 59-79:

use celld::peer_auth::PeerAuth;
use celld::bucket::Bucket;

// Load existing or generate new 32-byte secret
let key: [u8; 32] = PeerAuth::load_or_create(&bucket)
    .await
    .expect("failed to load peer auth key");

let peer_auth = PeerAuth::new(key, "node-01.cluster.celld")?;

If the bucket lacks a secret, load_or_create generates cryptographically random bytes, hex-encodes them for storage, and returns the raw bytes for runtime use.

---

The Signing Process (PeerAuth::sign)

When a node initiates a request, PeerAuth::sign constructs a canonical request string and computes its HMAC-SHA256 signature. The canonical format defined in crates/celld/peer_auth.rs (lines 38-52) is:

{DOMAIN}\n{PROTOCOL_VERSION}\n{method}\n{path_and_query}\n{body_hash}\n{source}\n{target}\n{timestamp}\n{nonce}

The signing procedure executes these steps:

• Phase 1: Generate a cryptographically random nonce
• Phase 2: Record the current Unix timestamp in milliseconds
• Phase 3: Compute SHA-256(body) as the body hash
• Phase 4: Assemble the canonical string with all metadata
• Phase 5: Calculate HMAC-SHA256(secret, canonical_string)
• Phase 6: Hex-encode the HMAC as x-cells-peer-signature

All values become headers on the outgoing request:

Header	Purpose
x-cells-peer-signature	The HMAC-SHA256 signature (hex)
x-cells-peer-source	Signer's identity
x-cells-peer-target	Intended recipient
x-cells-peer-timestamp	Unix epoch milliseconds
x-cells-peer-nonce	Random 16-byte nonce (hex)
x-cells-peer-body-sha256	SHA-256 of request body
x-cells-peer-protocol-version	Protocol version (currently 1)
x-cells-peer-domain	Domain identifier

Example signing code:

let signed_request = peer_auth.sign(
    http_request,           // reqwest::RequestBuilder
    "POST",                 // HTTP method
    "/rpc/append_entries",  // path and query string
    request_body_bytes,     // &[u8]
    "node-02.cluster.celld" // target identity
)?;

---

The Verification Process (PeerAuth::verify)

Incoming requests undergo seven pure predicates in PeerAuth::verify (crates/celld/peer_auth.rs, lines 73-115). No I/O occurs during verification—all checks operate on in-memory data structures.

1. Protocol Version Check

The x-cells-peer-protocol-version header must equal PROTOCOL_VERSION_TEXT ("1"). Mismatches cause immediate rejection. See lines 73-77.

2. Identity Validation

source and target identities pass through celld_logic::peer::valid_identity in crates/logic/peer.rs (lines 14-23). Valid characters: ASCII alphanumerics plus _, -, and ..

3. Clock Window Validation

The timestamp must be within ±CLOCK_WINDOW_MS (30 seconds) of the verifier's current time. This prevents indefinite replay of captured signatures. Implemented in crates/logic/peer.rs, lines 25-30.

4. Body Integrity Check

The SHA-256 hash of the received body must match x-cells-peer-body-sha256. Any mutation—accidental or malicious—invalidates the signature. See peer_auth.rs lines 93-97.

5. HMAC Validation

The verifier recomputes the canonical string using received headers and compares its HMAC against x-cells-peer-signature. Lines 102-112 implement this constant-time comparison.

6. Target Match

The target header must equal the verifier's own identity. This prevents signature forwarding attacks where node A's signed request is replayed to node B. Lines 113-115.

7. Replay Protection

Even valid signatures cannot be reused. The ReplayCache structure (peer_auth.rs lines 166-231) maintains an in-memory set of seen nonces with LRU eviction:

• Retention: REPLAY_RETENTION_MS (≥ 30 seconds, typically 5 minutes)
• Capacity: MAX_REPLAY_ENTRIES (1,000,000 entries)
• Pruning: Background task removes expired entries

Duplicate nonces return VerifyError::Replay.

Axum handler example showing full verification:

use axum::{
    extract::{Request, Body, Extension},
    http::StatusCode,
};
use celld::peer_auth::{PeerAuth, VerifyError};

async fn rpc_handler(
    Extension(auth): Extension<PeerAuth>,
    req: Request<Body>,
) -> Result<String, StatusCode> {
    let (parts, body) = req.into_parts();
    let body_bytes = axum::body::to_bytes(body, usize::MAX)
        .await
        .map_err(|_| StatusCode::BAD_REQUEST)?;

    auth.verify(
        &parts.method,
        parts.uri.path_and_query()
            .map(|p| p.as_str())
            .unwrap_or("/"),
        &parts.headers,
        &body_bytes,
        auth.source(), // we are the expected target
    ).map_err(|e| match e {
        VerifyError::Replay => StatusCode::CONFLICT,
        _ => StatusCode::UNAUTHORIZED,
    })?;

    Ok("verified".to_string())
}

---

Error Handling and Security Edge Cases

The VerifyError enum covers specific failure modes for observability:

match auth.verify(/* ... */) {
    Ok(_) => process_request(),
    Err(VerifyError::Replay) => {
        metrics::counter!("celld.auth.replay_detected").increment(1);
        return Err(StatusCode::CONFLICT);
    }
    Err(VerifyError::ClockSkew { received, now }) => {
        tracing::warn!(received, now, "peer clock skew");
        return Err(StatusCode::UNAUTHORIZED);
    }
    Err(VerifyError::InvalidSignature) => {
        return Err(StatusCode::UNAUTHORIZED);
    }
    // ... other variants
}

Critical security properties:

• No secret transmission: The 32-byte key never leaves the node's memory
• Request binding: Signatures are bound to method, path, body, source, target, timestamp, and nonce
• Temporal limits: 30-second clock window plus nonce tracking limits attack windows
• Memory-bounded replay cache: LRU eviction prevents unbounded growth

---

Why celld Uses HMAC Instead of TLS or JWT

The design tradeoffs reflect celld's operational constraints:

Approach	Why celld Avoids It	Why HMAC Wins
mTLS	Certificate management, rotation complexity, trust anchor distribution	Single shared secret, automatic provisioning
JWT + JWKS	Requires centralized coordination, token expiry complexity, larger payloads	Compact headers, no external dependencies
MAC tokens (AWS Signature v4)	Similar to celld's design, but celld simplifies for single-tenant clusters	Purpose-built for peer-to-peer meshes, not multi-tenant clouds

The HMAC-based peer authentication mechanism achieves stateless verification—any node can verify any peer's signature using only the shared secret, with replay protection handled locally.

---

Summary

• Key provisioning: 32-byte secrets stored hex-encoded in fleet/peer-auth.json, loaded by PeerAuth::load_or_create in crates/celld/peer_auth.rs
• Signing: Canonical request strings with 9 components, signed via HMAC-SHA256 into x-cells-peer-signature
• Verification: Seven sequential checks—version, identity, clock, body hash, HMAC, target match, and nonce replay—implemented as pure predicates
• Replay protection: In-memory LRU cache with 1M entry cap and configurable retention
• Source files: crates/celld/peer_auth.rs (main logic), crates/logic/peer.rs (validation predicates)

---

Frequently Asked Questions

What happens if cluster nodes have clock skew?

Requests fail verify with VerifyError::ClockSkew. The CLOCK_WINDOW_MS constant (30 seconds) tolerates minor drift. Operators should run NTP or similar time synchronization across the celld cluster.

Where is the shared secret stored and how is it secured?

The secret persists in the HexFleet bucket at fleet/peer-auth.json as hex-encoded bytes. At runtime, it exists only in the PeerAuth struct's memory. No logging or serialization exposes the raw bytes.

Can the HMAC-based peer authentication mechanism work across datacenters?

Yes, with caveats. The 30-second clock window and replay cache retention must accommodate network latency. Higher latency deployments should increase CLOCK_WINDOW_MS and REPLAY_RETENTION_MS at compile time or via configuration.

What prevents a compromised node from forging requests to others?

All nodes share the same secret, so compromise of any node grants signing capability for the entire cluster. This is intentional for celld's design: it provides authenticity and integrity for intra-cluster traffic, not authorization boundaries between peers. Additional application-layer checks should enforce least-privilege access.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}