How Fleet Secret Rotation and Peer Authentication Work in celld

Celld stores a single fleet-wide secret in an S3-compatible bucket at fleet/peer-auth.json and uses HMAC-SHA256 request signing with timestamps and nonces to authenticate node-to-node HTTP traffic.

The denoland/celld distributed storage system relies on a shared fleet secret to establish trust between nodes. According to the celld source code, this secret lives at a fixed path in the backing object storage and powers the PeerAuth mechanism that signs and verifies all inter-node requests. Understanding how this secret is created, rotated, and used for authentication is essential for securely operating a celld cluster.

Loading and Creating the Fleet Secret

When a celld node starts, it invokes peer_auth::load_or_create from [crates/celld/peer_auth.rs](https://github.com/denoland/celld/blob/main/crates/celld/peer_auth.rs) to obtain the shared key. This function implements an atomic load-or-initialize pattern using the underlying bucket's compare-and-swap semantics.

• Load – If fleet/peer-auth.json exists, the function reads the JSON object containing a version number and the secret key encoded as hexadecimal.
• Create – If the object is missing, the node generates a fresh 32-byte random key, constructs a StoredSecret (version = 1, key = <hex>), and writes it via put_cas. If the CAS operation succeeds, the new key becomes the fleet secret; if it fails due to concurrent creation, the node reloads the existing secret written by the peer.

Because the secret is stored as a versioned object in the bucket, rotating it simply requires replacing fleet/peer-auth.json with a new key. Nodes that restart will load the updated secret, while running nodes continue with their cached key until they reload.

Peer Authentication Handshake

Each node constructs a PeerAuth instance using the loaded secret and its own identity (e.g., "node-A.example.com"). This instance provides sign and verify methods used for every node-to-node HTTP request.

Signing Outbound Requests

Before transmitting a request, the source node calls auth.sign, which injects the following headers into the HTTP request:

• x-cells-peer-version – Protocol version
• x-cells-peer-timestamp – Current Unix timestamp (milliseconds)
• x-cells-peer-nonce – Fresh 16-byte random value
• x-cells-peer-body-sha256 – SHA-256 hash of the request body
• x-cells-peer-signature – HMAC-SHA256 over a canonical string containing all above fields plus source and target identities

The signature ensures both integrity (body hash) and authenticity (HMAC with the shared secret).

Verifying Inbound Requests

The receiving node processes the request through auth.verify, which performs the following validations in [peer_auth.rs](https://github.com/denoland/celld/blob/main/crates/celld/peer_auth.rs):

1. Protocol version check – Ensures the x-cells-peer-version header is supported.
2. Identity validation – Validates source and target identities against allowed ASCII characters (letters, digits, ., -, _).
3. Clock window enforcement – Rejects requests outside a ±30 second window (CLOCK_WINDOW_MS) to mitigate replay attacks using old timestamps.
4. Body integrity – Recomputes the SHA-256 hash of the received body and compares it to the x-cells-peer-body-sha256 header.
5. Signature verification – Reconstructs the canonical string and verifies the HMAC-SHA256 signature using the shared fleet secret.
6. Target identity match – Confirms the request's target identity matches the receiving node's identity.
7. Replay protection – Checks an in-memory ReplayCache (up to 1,000,000 entries) to ensure the nonce has not been seen within the retention period (2× clock window). Duplicates return VerifyError::Replay.

If any check fails, verify returns a VerifyError that maps to HTTP status codes such as 401 Unauthorized, 409 Conflict, or 426 Upgrade Required.

Secret Rotation Workflow

Rotating the fleet secret in celld requires no specialized tooling—only standard bucket operations and rolling restarts:

1. Generate a new secret – Write a new fleet/peer-auth.json containing a fresh 32-byte key using the bucket's CAS API (e.g., put_cas). The JSON version field can be incremented to track format changes.
2. Roll out gradually – Restart each celld node (or trigger a configuration reload) so it calls load_or_create again and picks up the new secret.
3. Graceful transition – During the rollout, nodes running the old secret will reject requests signed with the new key. Once all nodes restart, the fleet converges on the new secret. The CAS semantics guarantee that only one secret exists in the bucket at any moment, preventing split-brain scenarios.

Security Model and Network Requirements

The peer authentication protocol provides authentication, integrity, and replay protection, but it does not encrypt the HTTP traffic itself. Celld transmits requests as plain text. Consequently, operators must deploy celld nodes behind a trusted private network or an encrypted overlay such as WireGuard or Tailscale. Additionally, the S3-compatible bucket credentials used to access fleet/peer-auth.json must be kept secret, as possession of the fleet secret grants authentication capabilities within the cluster. For the complete security model, see [docs/security.md](https://github.com/denoland/celld/blob/main/docs/security.md).

Code Examples

// Loading or creating the secret during node startup
let bucket = Bucket::new(&s3_config)?;
let secret_key = peer_auth::load_or_create(&bucket).await?;

// Initializing PeerAuth with identity
let my_identity = "node-01.example.com";
let auth = PeerAuth::new(secret_key, my_identity)?;

// Signing an outbound replication request
let client = reqwest::Client::new();
let req = client.post("http://node-02.example.com/replicate")
    .body(b"{\"foo\":\"bar\"}");
let signed = auth.sign(
    req,
    "POST",
    "/replicate?rev=5",
    b"{\"foo\":\"bar\"}",
    "node-02.example.com"
)?;

// Verifying an inbound request in an Axum handler
async fn handler(
    Method(method): Method,
    Path(path): Path<String>,
    headers: HeaderMap,
    bytes: Bytes,
    Extension(auth): Extension<Arc<PeerAuth>>,
) -> Result<impl IntoResponse, StatusCode> {
    auth.verify(&method, &path, &headers, &bytes, "node-01.example.com")
        .map_err(|e| e.status())?;
    // Process authenticated request...
    Ok(StatusCode::OK)
}

Summary

• Celld stores a single fleet-wide secret at the fixed S3 path fleet/peer-auth.json, loaded via peer_auth::load_or_create during startup.
• Secret creation uses compare-and-swap (CAS) semantics to handle concurrent initialization safely.
• Peer authentication relies on HMAC-SHA256 signatures over canonical request strings including timestamps, nonces, and body hashes.
• Replay protection combines a ±30 second clock window with an in-memory cache of up to 1,000,000 recent nonces.
• Rotation involves writing a new secret to the bucket and restarting nodes; the design tolerates brief periods of mixed key usage.
• Celld does not provide transport encryption—operators must secure the network layer separately.

Frequently Asked Questions

Where is the fleet secret stored in celld?

The secret is stored as a JSON object in the S3-compatible bucket backing the fleet at the fixed path fleet/peer-auth.json. This object contains a version number and a 32-byte key encoded in hexadecimal. All nodes read this file on startup via the peer_auth::load_or_create function.

How does celld prevent replay attacks between peers?

Celld implements replay protection through a combination of timestamp validation and nonce tracking. Each request must fall within a ±30 second clock window (CLOCK_WINDOW_MS), and the receiving node checks an in-memory ReplayCache of up to 1,000,000 entries to ensure the 16-byte nonce has not been seen within the retention period (twice the clock window). Duplicate nonces trigger a VerifyError::Replay.

Does celld encrypt traffic between nodes?

No. The peer authentication protocol signs requests for integrity and authenticity but does not terminate TLS or encrypt the HTTP body. The traffic remains plain text. Operators must place celld nodes behind a trusted private network or an encrypted overlay such as WireGuard or Tailscale, as documented in the project's security.md.

How do you rotate the fleet secret without downtime?

Rotation is performed by writing a new secret to fleet/peer-auth.json using the bucket's CAS API, then gradually restarting each node to reload the secret. During the transition, nodes running the old secret will reject requests signed with the new key until they restart, creating a brief mixed-key window that resolves once the rollout completes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}