Output Gate and Durability Guarantees in Celld: How Writes Achieve Zero RPO

Celld guarantees durable writes by holding client responses in an output gate until the replication layer proves each write is persisted, achieving RPO = 0 when enabled.

Celld's output gate is a core mechanism that bridges write replication and client acknowledgment. When enabled—which is the default—every write operation blocks its response until the system can prove the data survives node crashes. This article explains how the output gate works, how to configure it via CELLD_OUTPUT_GATE, and what durability guarantees you receive under each setting.

How the Output Gate Works

The output gate sits between write execution and client response in celld's request pipeline. When a write request arrives, the core initiates replication and immediately opens an output gate for that operation.

In crates/celld/main.rs, the write handling logic explicitly opens the output gate after recording the write position. The gate holds the HTTP response in a pending state while the replication subsystem works to persist the data.

The gate closes only when one of two conditions occurs:

• Success: The replication layer reports the write position as durable via await_durable or ensure_durable
• Failure: Replication fails or times out, causing the write to error

This design prevents the classic distributed systems hazard of acknowledging a write that could vanish if the originating node crashes moments later.

Durability Proof From the Replication Layer

The replication subsystem provides the proof that closes the output gate. Two functions in crates/celld/runtime.rs implement this:

// runtime.rs § line 360-368
runtime.await_durable(cell, epoch, position).await?;
runtime.ensure_durable(cell, epoch, position).await?;

Both functions block until the write's position is persisted in the underlying bucket. The proved durable position represents the highest log index known to survive any single-node failure.

If replication becomes unreachable or stalls, these calls eventually fail. This failure propagates back through the output gate, ensuring no false-positive acknowledgments—the client receives an error rather than a success for a potentially lost write.

CELLD_OUTPUT_GATE Environment Variable

Celld exposes the output gate behavior through the CELLD_OUTPUT_GATE environment variable. This toggle lets operators trade durability for latency when appropriate.

Value	Behavior	Use Case
1 (default)	Wait for durability proof before responding	Production workloads requiring data safety
0	Acknowledge writes immediately	Development, testing, or latency-sensitive ephemeral data

The default configuration appears in docs/README.md at lines 110-112:

"The default is 1, so celld proves each write durable before it acknowledges the write. Set 0 to remove the replication wait and accept possible loss of an acknowledged write."

Configuration Examples

Enable the output gate (default behavior):

# No environment variable needed—defaults to enabled

celld --bucket "$CELLD_BUCKET" \
      --endpoint "$S3_ENDPOINT" \
      --region "$AWS_REGION"

Disable the output gate for lower latency:

export CELLD_OUTPUT_GATE=0
celld --bucket "$CELLD_BUCKET" --endpoint "$S3_ENDPOINT"

Code-Level Implementation Details

The output gate mechanism spans three critical files in the celld codebase.

Gate State Management in crates/logic/lib.rs

The core gate implementation lives in the logic crate, with functions for opening, tracking, and closing gates:

• open_output_gate — creates a pending gate for a specific write
• gate_pinned — checks if a gate still holds a response
• gated_writes — collection tracking all active gates

Request Handling in crates/celld/main.rs

The main request handler integrates the output gate into the write path. After a successful write position is obtained, the code explicitly opens the output gate and attaches the pending response. The response releases only upon durability confirmation or failure.

Runtime Durability Functions in crates/celld/runtime.rs

The runtime provides the await_durable and ensure_durable functions that the output gate depends on. These interface with the underlying bucket storage to verify persistence.

API-Level Durability Semantics

When using celld through its API, the output gate's behavior is transparent but consequential. A Durable Object handler, for example, experiences this as natural request latency:

// The response delays until durability is proven
export default {
  async fetch(request, env) {
    const durable = await env.COUNTER.fetch("increment");
    // Output gate holds this response until replicated
    return new Response("OK");
  }
}

For Rust-native code using the runtime directly, you can manually await durability:

let position = runtime.write_position(cell, epoch, data).await?;
runtime.await_durable(cell, epoch, position).await?; // explicit durability wait

Durability Guarantees Summary

With CELLD_OUTPUT_GATE=1 (default):

• RPO = 0: No acknowledged write can be lost to node crash
• Strong consistency: Once acknowledged, writes are globally visible via the bucket's read-after-write semantics
• Conditional writes: The underlying bucket prevents silent data loss from concurrent modifications

With CELLD_OUTPUT_GATE=0:

• Best-effort durability: Writes acknowledged immediately, with replication occurring asynchronously
• Potential data loss: Node crash may lose recently acknowledged writes
• Minimal write latency: No blocking on replication round-trip

The bucket's conditional writes and read-after-write consistency properties ensure that once the output gate releases, the durable state is immediately observable by subsequent operations across all clients.

Summary

• Output gate holds write responses until durability is proven, preventing acknowledged-but-lost writes
• CELLD_OUTPUT_GATE=1 (default) enables strong durability with RPO = 0; set to 0 for lower latency at durability risk
• Replication proof comes from await_durable/ensure_durable in runtime.rs, verified against bucket persistence
• Key implementation files: crates/logic/lib.rs (gate state), crates/celld/main.rs (request integration), crates/celld/runtime.rs (durability functions)

Frequently Asked Questions

What happens if replication is slow or unavailable?

The output gate prevents the response from releasing until await_durable succeeds or fails. If replication stalls, the client request eventually times out or receives an error. This avoids the false-positive acknowledgment of a write that cannot be guaranteed durable.

Can I mix output gate settings across a celld cluster?

Yes, but carefully. Individual nodes respect their own CELLD_OUTPUT_GATE setting. A cluster with mixed configurations will have some nodes waiting for durability and others acknowledging immediately. Clients interacting with different nodes may observe inconsistent durability behaviors.

How does the output gate affect read-after-write consistency?

The output gate guarantees that once a write is acknowledged, subsequent reads anywhere will observe it. The gate closes only after the bucket confirms persistence, and the bucket provides native read-after-write consistency. This eliminates the need for client-side retry logic to verify write visibility.

Is there a performance cost to enabling the output gate?

Yes—latency increases by roughly one replication round-trip per write. For geographically distributed deployments, this may add tens to hundreds of milliseconds. However, throughput typically remains high since gates are tracked asynchronously and the system processes many concurrent writes. For latency-critical, non-critical data, CELLD_OUTPUT_GATE=0 removes this cost.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}