How Celld's Pressure Shedding Mechanism Works: Watermarks, Load Factors, and Hysteresis

Celld's pressure shedding mechanism uses configurable high watermarks for RSS and CPU, combined with an 80% hysteresis latch, to deterministically classify resource samples and shed approximately 10% of resident cells when pressure exceeds safe thresholds.

Celld, the distributed WebSocket runtime from Deno, protects nodes from OOM and CPU overload through a deterministic pressure shedding mechanism that evaluates resource usage against configured watermarks. This system relies on the PressureConfig struct defined in crates/logic/pressure.rs to classify load samples and trigger cell evictions when memory or processor usage crosses critical thresholds. By implementing a hysteresis-based state machine, the mechanism prevents rapid oscillation between shedding and normal operation while maintaining predictable resource boundaries.

Core Concepts: Watermarks and Load Factors

The pressure shedding system hinges on two fundamental abstractions: watermarks that define resource limits, and load factors that represent real-time resource consumption.

PressureConfig and High Watermarks

The PressureConfig struct stores optional high watermarks for RSS (in bytes) and CPU (as percent × 100). These values are read once from environment variables when the node starts via pressure_config_from_environment in crates/celld/main.rs.

According to the source code in crates/logic/pressure.rs, the configuration uses implicit low watermarks set at 80% of each high watermark value. These low thresholds serve as hysteresis bounds that prevent the shedding latch from releasing immediately when usage dips slightly below high watermarks.

The Load Struct and Sampling

Every sampling interval generates a Load struct containing three critical metrics: resident cell count, current RSS in bytes, and current CPU percentage (× 100). This sample feeds directly into the classifier's state machine. The Load definition in crates/logic/pressure.rs represents the instantaneous resource factor that the system evaluates against configured watermarks.

The Classification Workflow

Celld implements a pure-function classifier that consumes a load sample and the previous shedding state to output a deterministic pressure state. This workflow operates through five distinct stages:

High-Watermark Detection

The PressureConfig::trigger(&self, s: Load) method compares the sample against high watermarks. If rss_bytes exceeds rss_high_bytes, the trigger returns "rss"; if cpu_percent_x100 exceeds cpu_high_x100, it returns "cpu"; otherwise it returns None. This initial check identifies which resource, if any, has crossed into dangerous territory.

Hysteresis Latch and Low Watermarks

Shedding does not clear immediately when samples fall below high watermarks. The PressureConfig::relieved method requires that both resources drop below their respective low watermarks (80% of the high values) before releasing the latch. This hysteresis prevents rapid on/off flipping that could destabilize the cluster.

Shedding Decisions and State Transitions

The PressureConfig::shedding_trigger(s, was_shedding) method implements the core decision logic. It prefers instantaneous high-watermark triggers when available; otherwise, while the latch remains active, it checks whether the sample remains above each resource's low watermark.

The PressureConfig::state(s, was_shedding) method combines these evaluations into a PressureState struct:

PressureState {
    shedding: self.shedding_trigger(s, was_shedding).is_some(),
    trigger: self.trigger(s),
}

The shedding boolean drives the scheduler's eviction decisions, while trigger records the specific resource causing the transition for metrics and logging purposes.

Calculating Eviction Targets

When shedding activates, PressureConfig::release_target(resident_cells) computes the target cell count by reducing the resident population by roughly 10% (rounded up to at least one cell). The eviction walk-down then uses this target to select specific cells for removal, excluding those protected by outbound WebSocket pins via may_pin_outbound.

Implementation Details

The pressure shedding mechanism operates as a pure-function classifier: it receives a load sample and the previous shedding flag, then outputs a deterministic pressure state without side effects or clock reads. This design makes the decision fast, testable, and fully reproducible across nodes.

While evaluating shedding candidates, the system respects cells pinned by outbound WebSockets. Pinned cells are excluded from the eviction pool, ensuring that active outbound connections remain intact during pressure events.

Practical Example

The following example demonstrates how to configure pressure thresholds and evaluate samples using the Celld logic crate:

use celld_logic::pressure::{PressureConfig, Load};

// 1️⃣ Build the pressure configuration (normally read from env vars)
let pressure = PressureConfig {
    // e.g. 1 GiB RSS high watermark, 85 % CPU high watermark
    rss_high_bytes: Some(1 << 30),      // 1 GiB
    cpu_high_x100: Some(8500),          // 85.00 %
};

// 2️⃣ Create a sample of current resource usage
let sample = Load {
    resident_cells: 12_000,
    rss_bytes: 950_000_000,   // 950 MiB
    cpu_percent_x100: 8100,   // 81.00 %
};

// 3️⃣ Determine the pressure state (assume we were not shedding before)
let prev_shedding = false;
let state = pressure.state(sample, prev_shedding);

println!("Shedding? {}", state.shedding); // false – still below low watermarks
println!("Trigger?  {:?}", state.trigger); // Some("rss") – crossed high RSS

// 4️⃣ When a later sample pushes us over the high watermark:
let heavy = Load {
    resident_cells: 12_000,
    rss_bytes: 1_200_000_000, // > 1 GiB
    cpu_percent_x100: 8600,   // > 85 %
};
let state2 = pressure.state(heavy, state.shedding);
assert!(state2.shedding); // latch is now active

// 5️⃣ Compute how many cells to evict
let target = pressure.release_target(heavy.resident_cells);
println!("Target resident cells after shedding: {}", target);

Summary

• Watermark Configuration: Celld reads high watermarks for RSS and CPU from environment variables at startup via pressure_config_from_environment, with implicit low watermarks set at 80% for hysteresis.
• Load Sampling: The Load struct captures resident cell count, RSS bytes, and CPU percentage (× 100) at each interval for evaluation.
• Hysteresis Protection: The shedding latch remains active until both resources drop below their 80% low watermarks, preventing rapid state oscillation.
• Deterministic Classification: PressureConfig::state implements a pure function that returns a PressureState containing the shedding boolean and trigger resource.
• Eviction Strategy: When shedding activates, release_target reduces the resident cell count by approximately 10%, while respecting outbound WebSocket pins to preserve active connections.

Frequently Asked Questions

What triggers pressure shedding in Celld?

Pressure shedding triggers when a Load sample exceeds either the rss_high_bytes or cpu_high_x100 threshold defined in PressureConfig. The PressureConfig::trigger method checks these high watermarks and returns the specific resource type ("rss" or "cpu") that crossed its boundary, causing the state machine to activate the shedding latch.

How does the hysteresis latch prevent flapping?

The hysteresis latch prevents rapid on/off cycles by requiring that both RSS and CPU fall below their respective low watermarks (80% of the high values) before clearing the shedding state. The PressureConfig::relieved method enforces this constraint, ensuring that transient dips below high watermarks do not immediately resume normal operation while the system remains near capacity.

How many cells does Celld shed when under pressure?

When shedding activates, PressureConfig::release_target calculates a target resident count by reducing the current population by approximately 10%, rounding up to ensure at least one cell is evicted. For example, a node with 12,000 resident cells would shed roughly 1,200 cells during a pressure event, with the actual eviction walk-down selecting specific candidates while excluding pinned cells.

Can active WebSocket connections prevent cell eviction?

Yes. The shedding logic respects cells pinned by outbound WebSockets via the may_pin_outbound mechanism. These pinned cells are excluded from the eviction pool, ensuring that active outbound connections are not torn down while the node is shedding load. This preservation of active connections maintains user experience during resource pressure events.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}