How to Monitor Celld Fleet Health with the `celld diagnose` Command

The celld diagnose command checks Celld fleet health through three stages: bucket validation, lease enumeration from the object store, and direct signed HTTP probes to each live peer node.

Monitoring Celld fleet health is essential for operators running distributed deployments across multiple nodes. The celld diagnose command, implemented in the denoland/celld repository, provides a comprehensive diagnostic tool that validates storage connectivity, enumerates node leases, and probes individual peers for runtime metrics and protocol compatibility.

How the celld diagnose Command Works

The diagnostic process follows a strict three-stage pipeline defined in crates/celld/fleet.rs. Each stage must succeed for the command to report a healthy fleet.

Stage 1: Bucket Validation

Before examining any nodes, celld diagnose verifies that the configured object-store bucket (S3, GCS, or compatible) is reachable and satisfies the conditional-write contract required for safe cell ownership.

• Implementation: validate_bucket → probe_storage in fleet.rs lines 5-33
• Failure mode: If write access is unavailable, use --read-only to skip the write probe

Stage 2: Lease Enumeration

The command reads all node lease objects from nodes/<node>.json in the bucket. By default, every live lease is enumerated. When --peer flags are supplied, only the specified nodes are examined—useful during rolling upgrades or targeted troubleshooting.

• Implementation: diagnostic_node_ids and diagnostic_node in fleet.rs lines 35-48

Stage 3: Direct Peer Probing

For each valid lease, the command sends a signed HTTP request to the node's advertised address. The probe validates:

Check	Purpose
Lease validity	Distinguishes expired from live leases
Address correctness	Detects unsafe public IP advertisements vs. private addresses
Authentication success	Verifies PeerAuth token acceptance
Protocol version agreement	Ensures cluster-wide compatibility
Runtime metrics	Captures resident cells, websockets, memory, CPU, file descriptors, pressure state, shed cells, restoring count, and load age

Successful probes emit ok peer … lines. Failures emit fail peer … messages, and the command aborts with a non-zero exit status if any peer probe fails.

Interpreting celld diagnose Output

Understanding the output columns helps operators assess fleet health quickly:

• peer — Node identifier from the lease's node field
• addr — Advertised address used for the probe
• protocol — Celld protocol version reported by the node
• resident_cells — Cells currently held in memory
• websockets — Active WebSocket connections
• rss_bytes — Resident-set-size memory (or "unknown")
• in_use_bytes — Memory actually used by cells (or "unknown")
• cpu_percent — CPU utilization percentage
• fds / fd_limit — Open file descriptors vs. process limit
• pressured — Memory pressure flag (0 or 1)
• shed_cells — Cells shed due to memory pressure
• restoring — Cold routes awaiting activation permits
• load_age_ms — Milliseconds since the last load sample

A healthy fleet shows ok peer … for every live node, ok fleet … node lease(s) enumerated, and zero fail messages.

Practical Code Examples

Run a comprehensive fleet health check:

celld diagnose \
  --bucket "s3://my-celld-bucket" \
  --endpoint "https://s3.us-west-2.amazonaws.com" \
  --region "us-west-2"

Target specific nodes during maintenance:

celld diagnose \
  --bucket "$CELLD_BUCKET" \
  --peer node-01 \
  --peer node-03

Execute with read-only credentials (CI jobs, restricted environments):

celld diagnose \
  --bucket "$CELLD_BUCKET" \
  --read-only

Sample Output Patterns

Successful fleet health check:

ok bucket s3://my-celld-bucket
ok fleet 5 node lease(s) enumerated
ok peer node-01 at 10.0.0.5:8080 (signed direct probe) protocol=1 resident_cells=42 websockets=3 rss_bytes=12345678 in_use_bytes=987654 cpu_percent=12.34 fds=56/1024 pressured=0 shed_cells=0 restoring=0 load_age_ms=150
ok peer node-02 at 10.0.0.6:8080 (signed direct probe) protocol=1 resident_cells=38 websockets=2 rss_bytes=11300000 in_use_bytes=880000 cpu_percent=9.87 fds=45/1024 pressured=0 shed_cells=0 restoring=0 load_age_ms=120
ok fleet skipped 0 expired node lease(s)

Failed probe (non-zero exit status):

fail peer node-03: TLS handshake failed

The bail!("fleet diagnostics failed …") invocation in the source ensures the process exits with an error code, enabling automated alerting and CI/CD pipeline integration.

When to Use --read-only

The --read-only flag bypasses the write probe in probe_storage. Use this when:

• Operator credentials lack bucket write permissions
• Running diagnostics from CI/CD pipelines with minimal privileges
• Validating fleet health without risking storage mutations

Without this flag, credential limitations cause immediate command failure before any peer probing occurs.

Key Source Files

Reference these locations in denoland/celld for implementation details:

• crates/celld/fleet.rs — diagnose entry point, bucket validation, lease enumeration, and peer probing logic
• crates/celld/peer_probe.rs — Signed HTTP probe implementation against each node's /state endpoint
• crates/celld/peer_auth.rs — Authentication token generation for diagnostic requests
• crates/celld/main/cli.rs — CLI argument parsing for the diagnose subcommand
• docs/README.md — User-facing documentation for command options

Summary

• celld diagnose validates bucket connectivity, enumerates node leases, and probes peers via signed HTTP requests
• The command exits non-zero if any peer fails authentication, protocol version checks, or network reachability
• Use --peer flags to scope diagnostics to specific nodes during rolling operations
• Apply --read-only when credentials lack bucket write access
• Output metrics expose memory pressure, cell distribution, and resource utilization across the fleet

Frequently Asked Questions

What does pressured=1 mean in celld diagnose output?

A pressured=1 flag indicates the node is under memory pressure and has begun shedding cells to protect stability. Check the shed_cells counter to see how many cells were evicted. Persistent pressure suggests the node needs more memory, fewer resident cells, or load balancing adjustments.

Why does celld diagnose fail with a bucket error when my credentials work for other tools?

The diagnostic command specifically tests conditional-write capability required for safe cell ownership. Standard read/list permissions are insufficient. Either grant write access to the bucket or add --read-only to skip this validation and proceed with peer probing only.

How can I automate fleet health monitoring with celld diagnose?

Run celld diagnose in a cron job or monitoring loop and check the exit status. Zero indicates success; non-zero triggers alerts. Parse the ok peer or fail peer lines for metric ingestion. Combine with --read-only for secure, periodic health checks from minimal-privilege service accounts.

What causes fail peer messages with protocol version mismatches?

The protocol field in probe responses must match the diagnosing node's expected version. Mismatches occur during rolling upgrades when nodes temporarily run different Celld versions. Target specific peers with --peer to isolate version boundaries, or complete the upgrade to restore uniform protocol versions across the fleet.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}