# How celld Implements HMAC-Based Peer-to-Peer Authentication with Clock-Bounded Replay Protection

> Discover celld's robust peer-to-peer authentication. Learn how HMAC, clock-bounded windows, and replay protection secure inter-node HTTP requests with fleet secrets and nonce caches.

- Repository: [Deno/celld](https://github.com/denoland/celld)
- Tags: deep-dive
- Published: 2026-08-10

---

**celld authenticates every inter-node HTTP request using a four-layer defense system that combines a shared fleet secret, HMAC-SHA-256 signing, 30-second clock windows, and an in-memory nonce replay cache.**

The denoland/celld distributed database engine secures communication between fleet nodes through a specialized peer-to-peer authentication protocol rather than application-layer TLS. Implemented primarily in [`crates/celld/peer_auth.rs`](https://github.com/denoland/celld/blob/main/crates/celld/peer_auth.rs), this scheme cryptographically binds each request to its contents, timestamp, and unique nonce, ensuring that only nodes possessing the shared bucket credentials can participate in the cluster.

## Shared Fleet Secret

At the foundation of celld's security model lies a single **shared fleet secret** stored in the underlying S3-compatible bucket at [`fleet/peer-auth.json`](https://github.com/denoland/celld/blob/main/fleet/peer-auth.json). This 32-byte random key is generated once and loaded by each node via the `load_or_create` function.

When a node initializes, it retrieves this secret from object storage, ensuring all participants in the fleet use identical key material for subsequent HMAC operations. This design delegates trust to the bucket's access control policies—only nodes with valid bucket credentials can obtain the signing key.

## HMAC-SHA-256 Request Signing

Every outbound request carries a cryptographic signature in the `x-cells-peer-signature` header computed using **HMAC-SHA-256**. In [`crates/celld/peer_auth.rs`](https://github.com/denoland/celld/blob/main/crates/celld/peer_auth.rs), the `sign` method constructs a canonical request string from:

- HTTP method and path
- SHA-256 hash of the request body (sent as `x-cells-peer-body-sha256`)
- Source node ID and target node ID
- Millisecond timestamp (`x-cells-peer-timestamp`)
- 16-byte random nonce

The HMAC operates on this canonical representation using the shared fleet secret. The resulting 32-byte MAC is hex-encoded and attached to the request headers, creating a tamper-evident envelope that validates both origin and payload integrity.

## Clock-Bounded Freshness

To prevent delayed attacks, celld enforces strict **clock-bounded freshness** checks using a configurable window defined by `CLOCK_WINDOW_MS` (default 30 seconds). Upon receiving a request, the verification logic in [`peer_auth.rs`](https://github.com/denoland/celld/blob/main/peer_auth.rs) extracts the `x-cells-peer-timestamp` header and validates it against the local system clock.

If the sender's timestamp falls outside the 30-second window, the receiver immediately rejects the request with an **Unauthorized** status. This temporal constraint limits the window during which a captured request remains valid, complementing the cryptographic signature with a time-based validity check.

## Nonce-Based Replay Protection

Even within the clock window, celld prevents duplicate requests through a **replay cache** mechanism. Each request includes a unique 16-byte nonce generated via `rand::rngs::OsRng.fill_bytes` and transmitted in the headers.

The `ReplayCache` structure maintains an in-memory map of observed nonces to their first-seen timestamps. Entries persist for `REPLAY_RETENTION_MS` (twice the clock window, or 60 seconds) and the cache enforces a hard limit of `MAX_REPLAY_ENTRIES` (1,000,000) entries, pruning old records as needed. If verification encounters a nonce already present in the cache, it returns **Replay** and drops the request.

## Practical Implementation

The following examples demonstrate how nodes initialize the authentication layer and secure HTTP traffic.

Loading the shared secret and creating a `PeerAuth` instance:

```rust
use crates::celld::peer_auth::PeerAuth;
use crates::celld::bucket::Bucket;

// `bucket` is an already-configured S3-compatible client
let secret = PeerAuth::load_or_create(&bucket).await?;
let auth = PeerAuth::new(secret, "node-a.internal")?;

```

Signing an outbound request before dispatch:

```rust
let client = reqwest::Client::new();
let req = client.post("http://node-b.internal:8080/some/path")
    .body(b"{\"msg\":\"hello\"}");

let signed_req = auth.sign(req, "POST", "/some/path", b"{\"msg\":\"hello\"}", "node-b.internal")?;
let response = signed_req.send().await?;

```

Verifying an inbound request within an Axum handler:

```rust
async fn handler(
    axum::extract::Extension(auth): axum::extract::Extension<PeerAuth>,
    method: axum::http::Method,
    uri: axum::http::Uri,
    headers: axum::http::HeaderMap,
    body: bytes::Bytes,
) -> Result<impl axum::response::IntoResponse, axum::http::StatusCode> {
    // `expected_target` is this node's advertised address
    auth.verify(&method, uri.path_and_query().map(|p| p.as_str()).unwrap_or("/"),
                &headers, &body, "node-a.internal")
        .map_err(|e| e.status())?;
    // …handle the request…
    Ok(axum::Json(serde_json::json!({ "ok": true })))
}

```

## Network Layer Considerations

According to the project's security documentation ([`docs/security.md`](https://github.com/denoland/celld/blob/main/docs/security.md)), the peer protocol intentionally does **not** terminate TLS. Operators must deploy nodes on a trusted private network or encrypted overlay such as WireGuard or Tailscale. This separation of concerns places transport encryption at the network layer while celld handles application-layer authentication and integrity verification.

## Summary

- **Shared secret**: A 32-byte key stored in [`fleet/peer-auth.json`](https://github.com/denoland/celld/blob/main/fleet/peer-auth.json) and loaded via `load_or_create` serves as the root of trust for all nodes.
- **HMAC signatures**: Every request carries an `x-cells-peer-signature` header computed via HMAC-SHA-256 over a canonical representation including body hash and node IDs.
- **Clock bounds**: Requests must arrive within `CLOCK_WINDOW_MS` (30 seconds) of their claimed timestamp or face immediate rejection.
- **Replay cache**: A memory-resident `ReplayCache` tracks nonces for 60 seconds, rejecting duplicate `x-cells-peer-nonce` values with **Replay** status.
- **Network assumption**: The protocol assumes a trusted network path; TLS is not implemented at the application layer.

## Frequently Asked Questions

### Does celld use TLS for peer-to-peer authentication?

No. As documented in [`docs/security.md`](https://github.com/denoland/celld/blob/main/docs/security.md), celld's peer-to-peer authentication operates over plain HTTP. The system assumes operators provide transport security through external means such as private networks, VPNs, or encrypted overlays like WireGuard. The HMAC-based scheme provides authentication and integrity, not confidentiality against network eavesdropping.

### How does celld prevent replay attacks?

celld combines two mechanisms: a 30-second clock window (`CLOCK_WINDOW_MS`) and a nonce replay cache. The receiver rejects requests with timestamps outside the current window. Within that window, a 16-byte random nonce must be unique; the `ReplayCache` stores nonces for 60 seconds (`REPLAY_RETENTION_MS`) and rejects any duplicates with a **Replay** error.

### What happens if node clocks drift out of sync?

If a sender's clock deviates by more than `CLOCK_WINDOW_MS` (30 seconds) from the receiver's clock, the verification logic in [`crates/celld/peer_auth.rs`](https://github.com/denoland/celld/blob/main/crates/celld/peer_auth.rs) rejects the request as **Unauthorized**. This requires operators to maintain reasonable clock synchronization across the fleet, typically via NTP, to prevent legitimate requests from failing validation.

### Where is the authentication secret stored?

The shared fleet secret resides in the S3-compatible bucket at [`fleet/peer-auth.json`](https://github.com/denoland/celld/blob/main/fleet/peer-auth.json). Nodes retrieve this 32-byte key on startup using the `load_or_create` function. Because bucket access controls the ability to retrieve this secret, securing the underlying object storage credentials becomes critical to cluster security.