# How the Celld Diagnose Command Enumerates and Probes Node Leases

> Learn how the celld diagnose command probes node leases. It scans S3, filters expired leases, and verifies node identity and fleet health with signed HTTP challenges.

- Repository: [Deno/celld](https://github.com/denoland/celld)
- Tags: how-to-guide
- Published: 2026-08-10

---

**The `celld diagnose` command scans the configured S3 bucket for JSON lease files under the `nodes/` prefix, filters out expired entries by checking the `expires_ms` timestamp, and executes cryptographically signed HTTP challenge-response probes against each live node's advertised address to verify identity and fleet health.**

The `celld diagnose` command serves as the primary health inspection tool for operators managing a Celld fleet. According to the `denoland/celld` source code, the command implements a two-phase workflow that first enumerates distributed node leases from object storage, then probes each valid peer using a signed authentication protocol. Understanding how the command enumerates and probes node leases is essential for troubleshooting node connectivity and verifying cluster state.

## Enumerating Node Leases from the Fleet Bucket

The enumeration phase begins in [`crates/celld/main.rs`](https://github.com/denoland/celld/blob/main/crates/celld/main.rs), where the CLI parser extracts the `--bucket` flag and optional `--peer` list to construct an `Action::Diagnose` instance. If no specific peers are provided, the command proceeds to discover all nodes automatically.

Before scanning, the `validate_bucket` function in [`crates/celld/fleet.rs`](https://github.com/denoland/celld/blob/main/crates/celld/fleet.rs) verifies the S3 endpoint is reachable. The `diagnostic_node_ids` function then lists every object under the `nodes/` prefix and extracts the node ID from each filename, which follows the pattern `<node-id>.json`.

For each discovered ID, the `diagnostic_node` function retrieves the lease file and deserializes it into the `DiagnosticNode` struct. This struct contains the node's **advertised address**, **expiration timestamp** (`expires_ms`), **protocol version**, **probe public key**, and current load metrics. The function immediately returns `None` for expired leases, causing the diagnostic loop to skip stale entries and count them as "skipped" in the final report.

## The Signed Challenge-Response Probe Protocol

For each valid lease, the command initiates a cryptographic handshake to verify the node's identity. The implementation in [`crates/celld/fleet.rs`](https://github.com/denoland/celld/blob/main/crates/celld/fleet.rs) first loads the operator's signing credentials via `PeerAuth::new`, which reads the stored peer key from the bucket for request authentication.

The actual network probe occurs in [`crates/celld/peer_probe.rs`](https://github.com/denoland/celld/blob/main/crates/celld/peer_probe.rs) within the `probe` function. This function generates a random challenge and constructs a signed payload with the format:

```

cells‑peer‑probe‑v1\n{node}\n{advertise}\n{challenge}

```

A short-lived `reqwest::Client` with aggressive connection timeouts sends this payload via HTTP POST to `http://{addr}/__celld/probe`. The target node must sign the response with its private `CELLD_REEXEC_PROBE_SIGNING_KEY`, proving possession of the keypair associated with the public key stored in the lease.

## Safety Controls and Network Restrictions

Before transmitting any probe, the implementation validates the node's advertised address in [`crates/celld/fleet.rs`](https://github.com/denoland/celld/blob/main/crates/celld/fleet.rs). If the address resolves to a public IP and the operator has not provided the `--unsafe-public-advertise` flag, the command aborts the probe for that specific peer to prevent accidental exposure of internal endpoints.

The HTTP client configuration uses short connect and overall timeouts to ensure the command completes efficiently even when nodes are offline or unreachable, preventing the diagnose operation from hanging on network failures.

## Usage Examples

To enumerate and probe all nodes in a fleet:

```bash
celld diagnose \
  --bucket s3://my-fleet-bucket \
  --endpoint https://s3.amazonaws.com \
  --region us-east-1

```

To diagnose specific nodes without scanning the entire bucket:

```bash
celld diagnose \
  --bucket s3://my-fleet-bucket \
  --peer node-abc123 \
  --peer node-def456

```

The following Rust snippet demonstrates the core lease validation and probing logic:

```rust
// From crates/celld/fleet.rs
let node = diagnostic_node(&bucket, "node-abc123").await?;
if let Some(node) = node {
    // Only probe if lease is not expired
    peer_probe::probe(&http_client, &node, &auth).await?;
}

```

## Summary

- The `celld diagnose` command retrieves all lease files from the `nodes/` prefix in the configured S3 bucket via the `diagnostic_node_ids` function.
- It filters expired leases by comparing the current time against the `expires_ms` field in the `DiagnosticNode` struct before initiating network connections.
- Each live node receives a signed HTTP probe containing a random challenge to the `/__celld/probe` endpoint, with the payload formatted as `cells‑peer‑probe‑v1\n{node}\n{advertise}\n{challenge}`.
- The command verifies the node's identity by validating the signature on the `PeerProbeResponse` against the stored public key using the `PeerAuth` credentials.
- Safety mechanisms in [`crates/celld/fleet.rs`](https://github.com/denoland/celld/blob/main/crates/celld/fleet.rs) prevent probing public IP addresses unless explicitly enabled with the `--unsafe-public-advertise` flag.

## Frequently Asked Questions

### Where does celld store node lease information?

The Celld system stores node lease data as JSON objects in the configured S3 bucket under the `nodes/` prefix. Each file follows the naming convention `<node-id>.json` and contains the `DiagnosticNode` struct with expiration timestamps, advertised addresses, and cryptographic keys for probe verification.

### How does celld diagnose handle expired node leases?

During the enumeration phase, the `diagnostic_node` function in [`crates/celld/fleet.rs`](https://github.com/denoland/celld/blob/main/crates/celld/fleet.rs) checks the `expires_ms` field against the current system time. If the lease has expired, the function returns `None`, causing the diagnostic loop to skip that node and report it as "skipped" rather than attempting a failed connection.

### What cryptographic mechanism verifies node identity during probing?

The probe uses a challenge-response protocol where the operator's client signs a payload containing a random nonce using the `PeerAuth` signing key loaded from the bucket. The target node must respond with a signature created using its private `CELLD_REEXEC_PROBE_SIGNING_KEY`, which the client verifies against the public key stored in the lease file.

### Why does celld diagnose refuse to probe certain IP addresses?

By default, the command refuses to probe nodes advertising public IP addresses to prevent security risks associated with exposing the probe endpoint to the internet. The implementation checks the address type before connecting and aborts unless the operator explicitly passes the `--unsafe-public-advertise` flag, which bypasses this restriction for testing or specific network topologies.