# How Fleet Secret Rotation and Peer Authentication Work in celld

> Learn how celld handles fleet secret rotation and peer authentication. Discover its HMAC-SHA256 request signing for secure node-to-node HTTP traffic.

- Repository: [Deno/celld](https://github.com/denoland/celld)
- Tags: deep-dive
- Published: 2026-08-10

---

**Celld stores a single fleet-wide secret in an S3-compatible bucket at [`fleet/peer-auth.json`](https://github.com/denoland/celld/blob/main/fleet/peer-auth.json) and uses HMAC-SHA256 request signing with timestamps and nonces to authenticate node-to-node HTTP traffic.**

The `denoland/celld` distributed storage system relies on a shared fleet secret to establish trust between nodes. According to the celld source code, this secret lives at a fixed path in the backing object storage and powers the `PeerAuth` mechanism that signs and verifies all inter-node requests. Understanding how this secret is created, rotated, and used for authentication is essential for securely operating a celld cluster.

## Loading and Creating the Fleet Secret

When a celld node starts, it invokes `peer_auth::load_or_create` from [[`crates/celld/peer_auth.rs`](https://github.com/denoland/celld/blob/main/crates/celld/peer_auth.rs)](https://github.com/denoland/celld/blob/main/crates/celld/peer_auth.rs) to obtain the shared key. This function implements an atomic load-or-initialize pattern using the underlying bucket's compare-and-swap semantics.

- **Load** – If [`fleet/peer-auth.json`](https://github.com/denoland/celld/blob/main/fleet/peer-auth.json) exists, the function reads the JSON object containing a `version` number and the secret `key` encoded as hexadecimal.
- **Create** – If the object is missing, the node generates a fresh 32-byte random key, constructs a `StoredSecret` (`version = 1`, `key = <hex>`), and writes it via `put_cas`. If the CAS operation succeeds, the new key becomes the fleet secret; if it fails due to concurrent creation, the node reloads the existing secret written by the peer.

Because the secret is stored as a versioned object in the bucket, **rotating it simply requires replacing [`fleet/peer-auth.json`](https://github.com/denoland/celld/blob/main/fleet/peer-auth.json)** with a new key. Nodes that restart will load the updated secret, while running nodes continue with their cached key until they reload.

## Peer Authentication Handshake

Each node constructs a `PeerAuth` instance using the loaded secret and its own identity (e.g., `"node-A.example.com"`). This instance provides `sign` and `verify` methods used for every node-to-node HTTP request.

### Signing Outbound Requests

Before transmitting a request, the source node calls `auth.sign`, which injects the following headers into the HTTP request:

- `x-cells-peer-version` – Protocol version
- `x-cells-peer-timestamp` – Current Unix timestamp (milliseconds)
- `x-cells-peer-nonce` – Fresh 16-byte random value
- `x-cells-peer-body-sha256` – SHA-256 hash of the request body
- `x-cells-peer-signature` – HMAC-SHA256 over a canonical string containing all above fields plus source and target identities

The signature ensures both **integrity** (body hash) and **authenticity** (HMAC with the shared secret).

### Verifying Inbound Requests

The receiving node processes the request through `auth.verify`, which performs the following validations in [[`peer_auth.rs`](https://github.com/denoland/celld/blob/main/peer_auth.rs)](https://github.com/denoland/celld/blob/main/crates/celld/peer_auth.rs):

1. **Protocol version check** – Ensures the `x-cells-peer-version` header is supported.
2. **Identity validation** – Validates source and target identities against allowed ASCII characters (letters, digits, `.`, `-`, `_`).
3. **Clock window enforcement** – Rejects requests outside a ±30 second window (`CLOCK_WINDOW_MS`) to mitigate replay attacks using old timestamps.
4. **Body integrity** – Recomputes the SHA-256 hash of the received body and compares it to the `x-cells-peer-body-sha256` header.
5. **Signature verification** – Reconstructs the canonical string and verifies the HMAC-SHA256 signature using the shared fleet secret.
6. **Target identity match** – Confirms the request's target identity matches the receiving node's identity.
7. **Replay protection** – Checks an in-memory `ReplayCache` (up to 1,000,000 entries) to ensure the `nonce` has not been seen within the retention period (2× clock window). Duplicates return `VerifyError::Replay`.

If any check fails, `verify` returns a `VerifyError` that maps to HTTP status codes such as 401 Unauthorized, 409 Conflict, or 426 Upgrade Required.

## Secret Rotation Workflow

Rotating the fleet secret in celld requires no specialized tooling—only standard bucket operations and rolling restarts:

1. **Generate a new secret** – Write a new [`fleet/peer-auth.json`](https://github.com/denoland/celld/blob/main/fleet/peer-auth.json) containing a fresh 32-byte key using the bucket's CAS API (e.g., `put_cas`). The JSON version field can be incremented to track format changes.
2. **Roll out gradually** – Restart each celld node (or trigger a configuration reload) so it calls `load_or_create` again and picks up the new secret.
3. **Graceful transition** – During the rollout, nodes running the old secret will reject requests signed with the new key. Once all nodes restart, the fleet converges on the new secret. The CAS semantics guarantee that only one secret exists in the bucket at any moment, preventing split-brain scenarios.

## Security Model and Network Requirements

The peer authentication protocol provides **authentication, integrity, and replay protection**, but it **does not encrypt the HTTP traffic itself**. Celld transmits requests as plain text. Consequently, operators must deploy celld nodes behind a trusted private network or an encrypted overlay such as WireGuard or Tailscale. Additionally, the S3-compatible bucket credentials used to access [`fleet/peer-auth.json`](https://github.com/denoland/celld/blob/main/fleet/peer-auth.json) must be kept secret, as possession of the fleet secret grants authentication capabilities within the cluster. For the complete security model, see [[`docs/security.md`](https://github.com/denoland/celld/blob/main/docs/security.md)](https://github.com/denoland/celld/blob/main/docs/security.md).

## Code Examples

```rust
// Loading or creating the secret during node startup
let bucket = Bucket::new(&s3_config)?;
let secret_key = peer_auth::load_or_create(&bucket).await?;

// Initializing PeerAuth with identity
let my_identity = "node-01.example.com";
let auth = PeerAuth::new(secret_key, my_identity)?;

// Signing an outbound replication request
let client = reqwest::Client::new();
let req = client.post("http://node-02.example.com/replicate")
    .body(b"{\"foo\":\"bar\"}");
let signed = auth.sign(
    req,
    "POST",
    "/replicate?rev=5",
    b"{\"foo\":\"bar\"}",
    "node-02.example.com"
)?;

```

```rust
// Verifying an inbound request in an Axum handler
async fn handler(
    Method(method): Method,
    Path(path): Path<String>,
    headers: HeaderMap,
    bytes: Bytes,
    Extension(auth): Extension<Arc<PeerAuth>>,
) -> Result<impl IntoResponse, StatusCode> {
    auth.verify(&method, &path, &headers, &bytes, "node-01.example.com")
        .map_err(|e| e.status())?;
    // Process authenticated request...
    Ok(StatusCode::OK)
}

```

## Summary

- Celld stores a single **fleet-wide secret** at the fixed S3 path [`fleet/peer-auth.json`](https://github.com/denoland/celld/blob/main/fleet/peer-auth.json), loaded via `peer_auth::load_or_create` during startup.
- Secret creation uses **compare-and-swap (CAS)** semantics to handle concurrent initialization safely.
- **Peer authentication** relies on HMAC-SHA256 signatures over canonical request strings including timestamps, nonces, and body hashes.
- **Replay protection** combines a ±30 second clock window with an in-memory cache of up to 1,000,000 recent nonces.
- **Rotation** involves writing a new secret to the bucket and restarting nodes; the design tolerates brief periods of mixed key usage.
- Celld does **not** provide transport encryption—operators must secure the network layer separately.

## Frequently Asked Questions

### Where is the fleet secret stored in celld?

The secret is stored as a JSON object in the S3-compatible bucket backing the fleet at the fixed path [`fleet/peer-auth.json`](https://github.com/denoland/celld/blob/main/fleet/peer-auth.json). This object contains a version number and a 32-byte key encoded in hexadecimal. All nodes read this file on startup via the `peer_auth::load_or_create` function.

### How does celld prevent replay attacks between peers?

Celld implements replay protection through a combination of timestamp validation and nonce tracking. Each request must fall within a ±30 second clock window (`CLOCK_WINDOW_MS`), and the receiving node checks an in-memory `ReplayCache` of up to 1,000,000 entries to ensure the 16-byte nonce has not been seen within the retention period (twice the clock window). Duplicate nonces trigger a `VerifyError::Replay`.

### Does celld encrypt traffic between nodes?

No. The peer authentication protocol signs requests for integrity and authenticity but does not terminate TLS or encrypt the HTTP body. The traffic remains plain text. Operators must place celld nodes behind a trusted private network or an encrypted overlay such as WireGuard or Tailscale, as documented in the project's [`security.md`](https://github.com/denoland/celld/blob/main/security.md).

### How do you rotate the fleet secret without downtime?

Rotation is performed by writing a new secret to [`fleet/peer-auth.json`](https://github.com/denoland/celld/blob/main/fleet/peer-auth.json) using the bucket's CAS API, then gradually restarting each node to reload the secret. During the transition, nodes running the old secret will reject requests signed with the new key until they restart, creating a brief mixed-key window that resolves once the rollout completes.