How the Celld Diagnose Command Enumerates and Probes Node Leases

The celld diagnose command scans the configured S3 bucket for JSON lease files under the nodes/ prefix, filters out expired entries by checking the expires_ms timestamp, and executes cryptographically signed HTTP challenge-response probes against each live node's advertised address to verify identity and fleet health.

The celld diagnose command serves as the primary health inspection tool for operators managing a Celld fleet. According to the denoland/celld source code, the command implements a two-phase workflow that first enumerates distributed node leases from object storage, then probes each valid peer using a signed authentication protocol. Understanding how the command enumerates and probes node leases is essential for troubleshooting node connectivity and verifying cluster state.

Enumerating Node Leases from the Fleet Bucket

The enumeration phase begins in crates/celld/main.rs, where the CLI parser extracts the --bucket flag and optional --peer list to construct an Action::Diagnose instance. If no specific peers are provided, the command proceeds to discover all nodes automatically.

Before scanning, the validate_bucket function in crates/celld/fleet.rs verifies the S3 endpoint is reachable. The diagnostic_node_ids function then lists every object under the nodes/ prefix and extracts the node ID from each filename, which follows the pattern <node-id>.json.

For each discovered ID, the diagnostic_node function retrieves the lease file and deserializes it into the DiagnosticNode struct. This struct contains the node's advertised address, expiration timestamp (expires_ms), protocol version, probe public key, and current load metrics. The function immediately returns None for expired leases, causing the diagnostic loop to skip stale entries and count them as "skipped" in the final report.

The Signed Challenge-Response Probe Protocol

For each valid lease, the command initiates a cryptographic handshake to verify the node's identity. The implementation in crates/celld/fleet.rs first loads the operator's signing credentials via PeerAuth::new, which reads the stored peer key from the bucket for request authentication.

The actual network probe occurs in crates/celld/peer_probe.rs within the probe function. This function generates a random challenge and constructs a signed payload with the format:

cells‑peer‑probe‑v1\n{node}\n{advertise}\n{challenge}

A short-lived reqwest::Client with aggressive connection timeouts sends this payload via HTTP POST to http://{addr}/__celld/probe. The target node must sign the response with its private CELLD_REEXEC_PROBE_SIGNING_KEY, proving possession of the keypair associated with the public key stored in the lease.

Safety Controls and Network Restrictions

Before transmitting any probe, the implementation validates the node's advertised address in crates/celld/fleet.rs. If the address resolves to a public IP and the operator has not provided the --unsafe-public-advertise flag, the command aborts the probe for that specific peer to prevent accidental exposure of internal endpoints.

The HTTP client configuration uses short connect and overall timeouts to ensure the command completes efficiently even when nodes are offline or unreachable, preventing the diagnose operation from hanging on network failures.

Usage Examples

To enumerate and probe all nodes in a fleet:

celld diagnose \
  --bucket s3://my-fleet-bucket \
  --endpoint https://s3.amazonaws.com \
  --region us-east-1

To diagnose specific nodes without scanning the entire bucket:

celld diagnose \
  --bucket s3://my-fleet-bucket \
  --peer node-abc123 \
  --peer node-def456

The following Rust snippet demonstrates the core lease validation and probing logic:

// From crates/celld/fleet.rs
let node = diagnostic_node(&bucket, "node-abc123").await?;
if let Some(node) = node {
    // Only probe if lease is not expired
    peer_probe::probe(&http_client, &node, &auth).await?;
}

Summary

• The celld diagnose command retrieves all lease files from the nodes/ prefix in the configured S3 bucket via the diagnostic_node_ids function.
• It filters expired leases by comparing the current time against the expires_ms field in the DiagnosticNode struct before initiating network connections.
• Each live node receives a signed HTTP probe containing a random challenge to the /__celld/probe endpoint, with the payload formatted as cells‑peer‑probe‑v1\n{node}\n{advertise}\n{challenge}.
• The command verifies the node's identity by validating the signature on the PeerProbeResponse against the stored public key using the PeerAuth credentials.
• Safety mechanisms in crates/celld/fleet.rs prevent probing public IP addresses unless explicitly enabled with the --unsafe-public-advertise flag.

Frequently Asked Questions

Where does celld store node lease information?

The Celld system stores node lease data as JSON objects in the configured S3 bucket under the nodes/ prefix. Each file follows the naming convention <node-id>.json and contains the DiagnosticNode struct with expiration timestamps, advertised addresses, and cryptographic keys for probe verification.

How does celld diagnose handle expired node leases?

During the enumeration phase, the diagnostic_node function in crates/celld/fleet.rs checks the expires_ms field against the current system time. If the lease has expired, the function returns None, causing the diagnostic loop to skip that node and report it as "skipped" rather than attempting a failed connection.

What cryptographic mechanism verifies node identity during probing?

The probe uses a challenge-response protocol where the operator's client signs a payload containing a random nonce using the PeerAuth signing key loaded from the bucket. The target node must respond with a signature created using its private CELLD_REEXEC_PROBE_SIGNING_KEY, which the client verifies against the public key stored in the lease file.

Why does celld diagnose refuse to probe certain IP addresses?

By default, the command refuses to probe nodes advertising public IP addresses to prevent security risks associated with exposing the probe endpoint to the internet. The implementation checks the address type before connecting and aborts unless the operator explicitly passes the --unsafe-public-advertise flag, which bypasses this restriction for testing or specific network topologies.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}