Why Celld Requires a Storage Probe at Startup: Durability Guarantees in Distributed Systems

Celld requires a storage probe at startup to verify that its object-store bucket supports conditional-write (CAS) operations and is writable before accepting any traffic, preventing silent data loss and cascade failures.

The storage probe is a critical safety mechanism in denoland/celld that validates storage backend conformance during node initialization. Without this check, a celld node could appear healthy while actually being unable to persist data, violating the platform's durability guarantees and potentially causing cluster-wide failures.

What the Storage Probe Verifies

The probe confirms two essential properties of the configured object store:

• Writability – The bucket accepts PUT operations
• CAS semantics – Conditional-write operations succeed and behave correctly

These checks ensure the bucket acts as a true compare-and-swap store, which celld relies on for distributed coordination and lease management.

According to the source code in crates/celld/fleet.rs, the probe_storage function performs this validation by writing a temporary test object and immediately deleting it:

// fleet.rs — probe_storage implementation
pub async fn probe_storage(bucket: &Bucket) -> Result<()> {
    bucket.probe_cas().await?;  // Performs conditional write test
    // Cleanup: delete the temporary probe object
    ...
}

When and How the Probe Runs

The storage probe executes during the startup sequence in crates/celld/main.rs at approximately line 4330:

// main.rs — startup sequence
if settings.storage_probe {
    // Abort startup if bucket fails conformance check
    fleet::probe_storage_before_serving(&client, settings.control_plane).await?;
}

The probe_storage_before_serving wrapper handles any errors by failing fast with a clear diagnostic:

// fleet.rs
pub async fn probe_storage_before_serving(...) -> Result<()> {
    probe_storage(bucket).await.map_err(|e| {
        bail!("bucket failed the storage conformance probe: {e}")
    })?;
    Ok(())
}

Configuration: Enabling and Disabling

Two mechanisms control the storage probe behavior:

Method	Effect
CELLD_STORAGE_PROBE=false environment variable	Disables the probe entirely
--read-only CLI flag	Skips write probe (diagnostic mode)

These are defined in crates/celld/cli.rs and processed at startup.

Running without the probe for troubleshooting:

# Disable via environment variable

CELLD_STORAGE_PROBE=false ./celld run --bucket s3://my-bucket

# Or use the read-only diagnostic mode

./celld run --read-only --bucket s3://my-bucket

Note: The --read-only mode is intended for diagnostics only. A production node serving write traffic must pass the storage probe.

Storage Probe vs. Peer Probe

Celld performs two distinct probes at startup:

• Storage probe (fleet.rs) – Validates the local durability path to object storage
• Signed peer probe (peer_probe.rs) – Validates the network path by exposing GET /__celld/probe

The storage probe protects against misconfigured buckets, IAM permission errors, or read-only storage mounts. The peer probe ensures other cluster nodes can reach this node. Both must succeed for a healthy cluster node.

Consequences of Skipping the Storage Probe

Without mandatory storage probing, a celld node could:

1. Appear healthy while losing data – Kubernetes health checks pass, but writes fail silently
2. Violate durability contracts – Accepted writes never persist, breaking application guarantees
3. Trigger cascade failures – Other nodes block on leases that can never be fulfilled, spreading outages

The probe's fail-fast behavior converts a potentially silent, catastrophic failure into an immediate, observable startup error.

Summary

• The storage probe verifies CAS-capable writability before celld serves traffic
• Implementation locations: crates/celld/fleet.rs (probe logic), crates/celld/main.rs (startup call)
• Configuration: CELLD_STORAGE_PROBE environment variable or --read-only CLI flag
• Complementary check: The peer probe (peer_probe.rs) validates network reachability
• Critical purpose: Prevents silent data loss and cascade failures in distributed deployments

Frequently Asked Questions

What happens if the storage probe fails?

Celld aborts startup with a clear error message: bucket failed the storage conformance probe. The process exits non-zero, preventing the node from joining the cluster with broken storage. This fail-fast behavior allows orchestrators like Kubernetes to surface the problem immediately rather than discovering data loss later.

Can I run celld without a storage probe in production?

No. Disabling CELLD_STORAGE_PROBE or using --read-only bypasses a critical safety check and should only be used for diagnostic troubleshooting. A production node serving any write traffic must verify storage conformance to maintain durability guarantees.

How does the storage probe relate to Kubernetes health checks?

The storage probe runs once at startup before the HTTP server starts, while Kubernetes health checks run continuously. The probe prevents the pod from becoming ready in the first place if storage is misconfigured. Health checks (documented in docs/telemetry.md) rely on the assumption that storage was validated by the probe.

What storage backends does the probe work with?

The probe uses Bucket::probe_cas(), which abstracts over any S3-compatible object store. It verifies that the backend supports conditional PUT operations (If-Match/If-None-Match semantics), which celld requires for distributed coordination.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}