# How to Monitor Celld Fleet Health with the `celld diagnose` Command

> Learn how to monitor Celld fleet health using the diagnose command. This guide covers bucket validation, lease enumeration, and HTTP probes for efficient fleet management.

- Repository: [Deno/celld](https://github.com/denoland/celld)
- Tags: how-to-guide
- Published: 2026-08-15

---

**The `celld diagnose` command checks Celld fleet health through three stages: bucket validation, lease enumeration from the object store, and direct signed HTTP probes to each live peer node.**

Monitoring Celld fleet health is essential for operators running distributed deployments across multiple nodes. The `celld diagnose` command, implemented in the **denoland/celld** repository, provides a comprehensive diagnostic tool that validates storage connectivity, enumerates node leases, and probes individual peers for runtime metrics and protocol compatibility.

## How the `celld diagnose` Command Works

The diagnostic process follows a strict three-stage pipeline defined in [`crates/celld/fleet.rs`](https://github.com/denoland/celld/blob/main/crates/celld/fleet.rs). Each stage must succeed for the command to report a healthy fleet.

### Stage 1: Bucket Validation

Before examining any nodes, `celld diagnose` verifies that the configured object-store bucket (S3, GCS, or compatible) is reachable and satisfies the conditional-write contract required for safe cell ownership.

- **Implementation**: `validate_bucket` → `probe_storage` in [`fleet.rs`](https://github.com/denoland/celld/blob/main/fleet.rs) lines 5-33
- **Failure mode**: If write access is unavailable, use `--read-only` to skip the write probe

### Stage 2: Lease Enumeration

The command reads all node lease objects from `nodes/<node>.json` in the bucket. By default, every live lease is enumerated. When `--peer` flags are supplied, only the specified nodes are examined—useful during rolling upgrades or targeted troubleshooting.

- **Implementation**: `diagnostic_node_ids` and `diagnostic_node` in [`fleet.rs`](https://github.com/denoland/celld/blob/main/fleet.rs) lines 35-48

### Stage 3: Direct Peer Probing

For each valid lease, the command sends a **signed HTTP request** to the node's advertised address. The probe validates:

| Check | Purpose |
|-------|---------|
| Lease validity | Distinguishes expired from live leases |
| Address correctness | Detects unsafe public IP advertisements vs. private addresses |
| Authentication success | Verifies `PeerAuth` token acceptance |
| Protocol version agreement | Ensures cluster-wide compatibility |
| Runtime metrics | Captures resident cells, websockets, memory, CPU, file descriptors, pressure state, shed cells, restoring count, and load age |

Successful probes emit `ok peer …` lines. Failures emit `fail peer …` messages, and the command aborts with a non-zero exit status if any peer probe fails.

## Interpreting `celld diagnose` Output

Understanding the output columns helps operators assess fleet health quickly:

- **`peer`** — Node identifier from the lease's `node` field
- **`addr`** — Advertised address used for the probe
- **`protocol`** — Celld protocol version reported by the node
- **`resident_cells`** — Cells currently held in memory
- **`websockets`** — Active WebSocket connections
- **`rss_bytes`** — Resident-set-size memory (or "unknown")
- **`in_use_bytes`** — Memory actually used by cells (or "unknown")
- **`cpu_percent`** — CPU utilization percentage
- **`fds / fd_limit`** — Open file descriptors vs. process limit
- **`pressured`** — Memory pressure flag (0 or 1)
- **`shed_cells`** — Cells shed due to memory pressure
- **`restoring`** — Cold routes awaiting activation permits
- **`load_age_ms`** — Milliseconds since the last load sample

A healthy fleet shows **`ok peer …`** for every live node, **`ok fleet … node lease(s) enumerated`**, and **zero `fail` messages**.

## Practical Code Examples

Run a comprehensive fleet health check:

```bash
celld diagnose \
  --bucket "s3://my-celld-bucket" \
  --endpoint "https://s3.us-west-2.amazonaws.com" \
  --region "us-west-2"

```

Target specific nodes during maintenance:

```bash
celld diagnose \
  --bucket "$CELLD_BUCKET" \
  --peer node-01 \
  --peer node-03

```

Execute with read-only credentials (CI jobs, restricted environments):

```bash
celld diagnose \
  --bucket "$CELLD_BUCKET" \
  --read-only

```

## Sample Output Patterns

**Successful fleet health check:**

```

ok bucket s3://my-celld-bucket
ok fleet 5 node lease(s) enumerated
ok peer node-01 at 10.0.0.5:8080 (signed direct probe) protocol=1 resident_cells=42 websockets=3 rss_bytes=12345678 in_use_bytes=987654 cpu_percent=12.34 fds=56/1024 pressured=0 shed_cells=0 restoring=0 load_age_ms=150
ok peer node-02 at 10.0.0.6:8080 (signed direct probe) protocol=1 resident_cells=38 websockets=2 rss_bytes=11300000 in_use_bytes=880000 cpu_percent=9.87 fds=45/1024 pressured=0 shed_cells=0 restoring=0 load_age_ms=120
ok fleet skipped 0 expired node lease(s)

```

**Failed probe (non-zero exit status):**

```

fail peer node-03: TLS handshake failed

```

The `bail!("fleet diagnostics failed …")` invocation in the source ensures the process exits with an error code, enabling automated alerting and CI/CD pipeline integration.

## When to Use `--read-only`

The `--read-only` flag bypasses the write probe in `probe_storage`. Use this when:

- Operator credentials lack bucket write permissions
- Running diagnostics from CI/CD pipelines with minimal privileges
- Validating fleet health without risking storage mutations

Without this flag, credential limitations cause immediate command failure before any peer probing occurs.

## Key Source Files

Reference these locations in **denoland/celld** for implementation details:

- **[`crates/celld/fleet.rs`](https://github.com/denoland/celld/blob/main/crates/celld/fleet.rs)** — `diagnose` entry point, bucket validation, lease enumeration, and peer probing logic
- **[`crates/celld/peer_probe.rs`](https://github.com/denoland/celld/blob/main/crates/celld/peer_probe.rs)** — Signed HTTP probe implementation against each node's `/state` endpoint
- **[`crates/celld/peer_auth.rs`](https://github.com/denoland/celld/blob/main/crates/celld/peer_auth.rs)** — Authentication token generation for diagnostic requests
- **[`crates/celld/main/cli.rs`](https://github.com/denoland/celld/blob/main/crates/celld/main/cli.rs)** — CLI argument parsing for the `diagnose` subcommand
- **[`docs/README.md`](https://github.com/denoland/celld/blob/main/docs/README.md)** — User-facing documentation for command options

## Summary

- **`celld diagnose`** validates bucket connectivity, enumerates node leases, and probes peers via signed HTTP requests
- The command exits non-zero if any peer fails authentication, protocol version checks, or network reachability
- Use **`--peer`** flags to scope diagnostics to specific nodes during rolling operations
- Apply **`--read-only`** when credentials lack bucket write access
- Output metrics expose memory pressure, cell distribution, and resource utilization across the fleet

## Frequently Asked Questions

### What does `pressured=1` mean in `celld diagnose` output?

A `pressured=1` flag indicates the node is under **memory pressure** and has begun shedding cells to protect stability. Check the `shed_cells` counter to see how many cells were evicted. Persistent pressure suggests the node needs more memory, fewer resident cells, or load balancing adjustments.

### Why does `celld diagnose` fail with a bucket error when my credentials work for other tools?

The diagnostic command specifically tests **conditional-write capability** required for safe cell ownership. Standard read/list permissions are insufficient. Either grant write access to the bucket or add `--read-only` to skip this validation and proceed with peer probing only.

### How can I automate fleet health monitoring with `celld diagnose`?

Run `celld diagnose` in a cron job or monitoring loop and check the exit status. Zero indicates success; non-zero triggers alerts. Parse the `ok peer` or `fail peer` lines for metric ingestion. Combine with `--read-only` for secure, periodic health checks from minimal-privilege service accounts.

### What causes `fail peer` messages with protocol version mismatches?

The `protocol` field in probe responses must match the diagnosing node's expected version. Mismatches occur during **rolling upgrades** when nodes temporarily run different Celld versions. Target specific peers with `--peer` to isolate version boundaries, or complete the upgrade to restore uniform protocol versions across the fleet.