How celld Implements the Peer Probe Mechanism for Secure Health Checking

Celld uses a signed, challenge-response peer probe mechanism where nodes verify each other's health by exchanging cryptographically signed challenges over HTTP, proving possession of Ed25519 private keys without relying on external services.

The denoland/celld repository implements a robust peer probe mechanism to validate that remote diagnostic nodes are both reachable and correctly configured. This cryptographic health-checking system ensures cluster nodes can verify each other's identities using Ed25519 signatures and canonical challenge payloads, eliminating reliance on mutable state or third-party services.

How the Peer Probe Mechanism Works in celld

Signer Initialization and Key Management

When a celld instance starts, it initializes its cryptographic identity through PeerProbeSigner::generate in crates/celld/peer_probe.rs (lines 30-36). The system either generates a fresh Ed25519 signing key or loads an existing one from the CELLD_REEXEC_PROBE_SIGNING_KEY environment variable (lines 68-76). The resulting public key is hex-encoded and stored in the global INSTALLED_SIGNER, then advertised to peers via the lease metadata field probe_public_key.

Generating and Sending Cryptographic Challenges

The probing node initiates health checks by generating a 32-byte random challenge using random_challenge() (lines 96-100). This value is transmitted in the HTTP header x-cells-probe-challenge to the target node's /__celld/probe endpoint. The request itself is authenticated using celld's standard peer-authentication scheme via PeerAuth::sign (lines 27-34), ensuring only authorized cluster members can issue probes.

Canonical Payload Signing and Response

Upon receiving a probe request, the target node invokes PeerProbeSigner::respond (lines 47-64) to validate the incoming challenge and generate a signed response. The signer constructs a canonical payload with the following strict format:

cells‑peer‑probe‑v1
<node>
<advertise>
<challenge>

The resulting PeerProbeResponse contains the version, node identifier, advertised address, challenge, and a hex-encoded signature (lines 21-28).

Response Verification and Health Determination

The probing node verifies responses through the verify method (lines 66-90), which validates that the version, node name, advertise address, and challenge match expectations, and cryptographically verifies the signature against the public key stored in the node lease (probe_public_key). Any signature mismatch, field discrepancy, or oversized body triggers an error, marking the node as unhealthy. Successful verification confirms the target possesses the correct private key and matches its lease advertisement.

Security Integration with Peer Authentication

The peer probe mechanism leverages the existing peer authentication layer implemented in crates/celld/peer_auth.rs. According to the source code, probe requests utilize PeerAuth for signing, while response validation checks the x-cells-peer-version header through validate_response (lines 108-119). This integration provides replay protection, timestamp validation, and nonce guarantees that protect against man-in-the-middle and replay attacks.

Implementing Peer Probes in Practice

Installing the Probe Signer at Startup

During node initialization, install the signer and expose the public key in the lease metadata:

// Executed during startup sequence
let public_key = celld::peer_probe::install_signer()?;
// Store `public_key` in node lease as `probe_public_key` for peer verification

Initiating Health Checks from Client Nodes

To check a remote node's health, use the high-level probe function which handles the full request-response cycle:

use reqwest::Client;
use celld::{peer_probe, peer_auth::PeerAuth};

async fn verify_node_health(
    client: &Client, 
    node: &DiagnosticNode, 
    auth: &PeerAuth
) -> anyhow::Result<()> {
    // Performs challenge generation, request signing, and response verification
    peer_probe::probe(client, node, auth).await?;
    println!("Node {} is healthy and authentic", node.node);
    Ok(())
}

Handling Incoming Probe Requests

Target nodes expose the /__celld/probe endpoint to respond to health checks:

use axum::extract::State;
use celld::peer_probe::PeerProbeResponse;

async fn probe_handler(
    headers: axum::http::HeaderMap,
) -> Result<axum::Json<PeerProbeResponse>, axum::http::StatusCode> {
    let challenge = headers
        .get("x-cells-probe-challenge")
        .ok_or(axum::http::StatusCode::BAD_REQUEST)?
        .to_str()
        .map_err(|_| axum::http::StatusCode::BAD_REQUEST)?;
    
    let response = celld::peer_probe::respond(
        &node_name,
        &advertise_addr,
        challenge,
    ).map_err(|_| axum::http::StatusCode::INTERNAL_SERVER_ERROR)?;
    
    Ok(axum::Json(response))
}

Summary

• Ed25519 Cryptography: The peer probe mechanism uses Ed25519 signatures to bind node identity to health check responses, with keys generated via PeerProbeSigner::generate in crates/celld/peer_probe.rs.
• Challenge-Response Protocol: Each probe sends a 32-byte random challenge via the x-cells-probe-challenge header and verifies the signed canonical payload containing node metadata.
• Lease-Bound Verification: Responses are validated against the probe_public_key stored in the node lease, ensuring the responding node matches its advertised identity.
• Layered Security: Probe requests are signed via PeerAuth::sign and responses validated through validate_response in crates/celld/peer_auth.rs, providing replay protection and timestamp verification.

Frequently Asked Questions

What cryptographic algorithm does the peer probe mechanism use?

The celld peer probe mechanism uses Ed25519 for digital signatures. When a node starts, PeerProbeSigner::generate creates an Ed25519 signing key (lines 30-36), or loads one from the CELLD_REEXEC_PROBE_SIGNING_KEY environment variable. The corresponding public key is distributed through the lease metadata, allowing peers to verify probe responses cryptographically.

How does celld prevent replay attacks during peer health checks?

Replay protection comes from the integration with celld's peer authentication layer. According to crates/celld/peer_auth.rs (lines 108-119), the validate_response function checks the x-cells-peer-version header and enforces timestamp and nonce validation. Additionally, the 32-byte random challenge in each probe request ensures responses cannot be reused across sessions.

What happens if a node fails the peer probe verification?

If verification fails—whether due to an invalid signature, mismatched node identity, incorrect challenge response, or oversized body—the verify method in crates/celld/peer_probe.rs (lines 66-90) returns an error. The probing node interprets this as a health check failure, marking the target as unhealthy and preventing it from participating in cluster operations until it passes subsequent checks.

Does the peer probe mechanism require external services or databases?

No. The peer probe mechanism is self-contained and does not rely on external services or mutable state. It operates purely through cryptographic proof—specifically, possession of the Ed25519 private key corresponding to the public key advertised in the node lease. This design allows health checks to function even during network partitions or third-party service outages.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}