# Why Celld Requires a Storage Probe at Startup: Durability Guarantees in Distributed Systems

> Celld needs a storage probe at startup to ensure your object-store bucket supports CAS operations and is writable, preventing data loss and cascade failures.

- Repository: [Deno/celld](https://github.com/denoland/celld)
- Tags: internals
- Published: 2026-08-15

---

**Celld requires a storage probe at startup to verify that its object-store bucket supports conditional-write (CAS) operations and is writable before accepting any traffic, preventing silent data loss and cascade failures.**

The **storage probe** is a critical safety mechanism in `denoland/celld` that validates storage backend conformance during node initialization. Without this check, a celld node could appear healthy while actually being unable to persist data, violating the platform's durability guarantees and potentially causing cluster-wide failures.

## What the Storage Probe Verifies

The probe confirms two essential properties of the configured object store:

- **Writability** – The bucket accepts PUT operations
- **CAS semantics** – Conditional-write operations succeed and behave correctly

These checks ensure the bucket acts as a true compare-and-swap store, which celld relies on for distributed coordination and lease management.

According to the source code in [`crates/celld/fleet.rs`](https://github.com/denoland/celld/blob/main/crates/celld/fleet.rs), the `probe_storage` function performs this validation by writing a temporary test object and immediately deleting it:

```rust
// fleet.rs — probe_storage implementation
pub async fn probe_storage(bucket: &Bucket) -> Result<()> {
    bucket.probe_cas().await?;  // Performs conditional write test
    // Cleanup: delete the temporary probe object
    ...
}

```

## When and How the Probe Runs

The storage probe executes during the startup sequence in [`crates/celld/main.rs`](https://github.com/denoland/celld/blob/main/crates/celld/main.rs) at approximately line 4330:

```rust
// main.rs — startup sequence
if settings.storage_probe {
    // Abort startup if bucket fails conformance check
    fleet::probe_storage_before_serving(&client, settings.control_plane).await?;
}

```

The `probe_storage_before_serving` wrapper handles any errors by failing fast with a clear diagnostic:

```rust
// fleet.rs
pub async fn probe_storage_before_serving(...) -> Result<()> {
    probe_storage(bucket).await.map_err(|e| {
        bail!("bucket failed the storage conformance probe: {e}")
    })?;
    Ok(())
}

```

## Configuration: Enabling and Disabling

Two mechanisms control the storage probe behavior:

| Method | Effect |
|--------|--------|
| `CELLD_STORAGE_PROBE=false` environment variable | Disables the probe entirely |
| `--read-only` CLI flag | Skips write probe (diagnostic mode) |

These are defined in [`crates/celld/cli.rs`](https://github.com/denoland/celld/blob/main/crates/celld/cli.rs) and processed at startup.

Running without the probe for troubleshooting:

```bash

# Disable via environment variable

CELLD_STORAGE_PROBE=false ./celld run --bucket s3://my-bucket

# Or use the read-only diagnostic mode

./celld run --read-only --bucket s3://my-bucket

```

**Note:** The `--read-only` mode is intended for diagnostics only. A production node serving write traffic must pass the storage probe.

## Storage Probe vs. Peer Probe

Celld performs **two distinct probes** at startup:

- **Storage probe** ([`fleet.rs`](https://github.com/denoland/celld/blob/main/fleet.rs)) – Validates the *local* durability path to object storage
- **Signed peer probe** ([`peer_probe.rs`](https://github.com/denoland/celld/blob/main/peer_probe.rs)) – Validates the *network* path by exposing `GET /__celld/probe`

The storage probe protects against misconfigured buckets, IAM permission errors, or read-only storage mounts. The peer probe ensures other cluster nodes can reach this node. Both must succeed for a healthy cluster node.

## Consequences of Skipping the Storage Probe

Without mandatory storage probing, a celld node could:

1. **Appear healthy while losing data** – Kubernetes health checks pass, but writes fail silently
2. **Violate durability contracts** – Accepted writes never persist, breaking application guarantees
3. **Trigger cascade failures** – Other nodes block on leases that can never be fulfilled, spreading outages

The probe's fail-fast behavior converts a potentially silent, catastrophic failure into an immediate, observable startup error.

## Summary

- **The storage probe** verifies CAS-capable writability before celld serves traffic
- **Implementation locations:** [`crates/celld/fleet.rs`](https://github.com/denoland/celld/blob/main/crates/celld/fleet.rs) (probe logic), [`crates/celld/main.rs`](https://github.com/denoland/celld/blob/main/crates/celld/main.rs) (startup call)
- **Configuration:** `CELLD_STORAGE_PROBE` environment variable or `--read-only` CLI flag
- **Complementary check:** The peer probe ([`peer_probe.rs`](https://github.com/denoland/celld/blob/main/peer_probe.rs)) validates network reachability
- **Critical purpose:** Prevents silent data loss and cascade failures in distributed deployments

## Frequently Asked Questions

### What happens if the storage probe fails?

Celld aborts startup with a clear error message: `bucket failed the storage conformance probe`. The process exits non-zero, preventing the node from joining the cluster with broken storage. This fail-fast behavior allows orchestrators like Kubernetes to surface the problem immediately rather than discovering data loss later.

### Can I run celld without a storage probe in production?

No. Disabling `CELLD_STORAGE_PROBE` or using `--read-only` bypasses a critical safety check and should only be used for diagnostic troubleshooting. A production node serving any write traffic must verify storage conformance to maintain durability guarantees.

### How does the storage probe relate to Kubernetes health checks?

The storage probe runs once at startup before the HTTP server starts, while Kubernetes health checks run continuously. The probe prevents the pod from becoming *ready* in the first place if storage is misconfigured. Health checks (documented in [`docs/telemetry.md`](https://github.com/denoland/celld/blob/main/docs/telemetry.md)) rely on the assumption that storage was validated by the probe.

### What storage backends does the probe work with?

The probe uses `Bucket::probe_cas()`, which abstracts over any S3-compatible object store. It verifies that the backend supports conditional PUT operations (If-Match/If-None-Match semantics), which celld requires for distributed coordination.