# How to Debug Slow Client Disconnections and the Grace Limit Mechanism in Buzz

> Debug slow client disconnections in Buzz by understanding the grace limit mechanism. Learn how Buzz detects and handles back-pressure and ensures graceful shutdowns.

- Repository: [Block Open Source/buzz](https://github.com/block/buzz)
- Tags: how-to-guide
- Published: 2026-08-29

---

**Buzz uses a configurable grace-limit counter to detect slow clients when send buffers fill up, logging warnings before terminating connections that repeatedly trigger back-pressure, while the same mechanism ensures graceful shutdowns complete within the 30-second hard deadline.**

The `buzz-relay` server must protect itself from clients that consume data too slowly, whether due to poor network conditions or malicious stalling. Understanding how to debug these disconnections requires familiarity with the **grace-limit mechanism**, a back-pressure counter defined in the Buzz source code that tracks consecutive buffer-full events per connection.

## Understanding the Grace-Limit Mechanism

The grace limit is a defense mechanism implemented in [`crates/buzz-relay/src/connection.rs`](https://github.com/block/buzz/blob/main/crates/buzz-relay/src/connection.rs) that monitors outbound buffer pressure on a per-connection basis.

### Configuration and Environment Variables

The limit is defined as a `u8` in `Config::slow_client_grace_limit` at [`crates/buzz-relay/src/config.rs#L178`](https://github.com/block/buzz/blob/main/crates/buzz-relay/src/config.rs#L178). You can override the default value at runtime using the environment variable `BUZZ_SLOW_CLIENT_GRACE_LIMIT`, which is parsed during initialization in the same file (lines 657–1219).

The configuration enforces a strict validation: the value must be greater than zero. An assertion at startup (`assert!(config.slow_client_grace_limit > 0)`) will panic if you attempt to set the limit to zero.

### Per-Connection Back-Pressure Detection

Each `Connection` struct maintains a `grace_limit: u8` field (declared at [`crates/buzz-relay/src/connection.rs#L85`](https://github.com/block/buzz/blob/main/crates/buzz-relay/src/connection.rs#L85)) alongside a transient counter. When a send operation encounters a full buffer, the server increments this counter and emits a structured warning log:

```rust
// Logic around crates/buzz-relay/src/connection.rs#L104-L109
if count >= self.grace_limit {
    warn!(conn_id = %self.conn_id, count, grace = self.grace_limit,
         "send buffer full — grace {count}/{}", self.grace_limit);
}

```

This log line is the primary diagnostic signal that a client is approaching termination.

### Reset Behavior and Termination Logic

A successful send resets the grace counter to zero (implemented at [`crates/buzz-relay/src/connection.rs#L93`](https://github.com/block/buzz/blob/main/crates/buzz-relay/src/connection.rs#L93)), ensuring that transient spikes do not accumulate against the limit. Only consecutive buffer-full events trigger termination.

When the counter reaches the configured limit, the connection is flagged for closure. The actual cancellation occurs during the next graceful-shutdown cycle or background cleanup, specifically around lines 1478–1481 in [`connection.rs`](https://github.com/block/buzz/blob/main/connection.rs).

## Debugging Slow Client Disconnections Step by Step

Follow this structured approach to diagnose whether a client disconnection is caused by the grace-limit mechanism:

1. **Verify the configured limit**  
   Check the runtime configuration or environment variable to confirm the threshold:  
   ```bash
   env | grep BUZZ_SLOW_CLIENT_GRACE_LIMIT
   ```

   Alternatively, inspect the generated `Config` object in your relay startup logs.

2. **Observe warning logs**  
   Search relay logs for the diagnostic pattern:  
   ```bash
   grep "grace" /var/log/buzz-relay.log
   ```

   Look for entries matching:  
   ```

   WARN  conn_id=abc-123 count=3 grace=3 "send buffer full — grace 3/3"
   ```

3. **Correlate with client behavior**  
   Extract the `conn_id` from warning logs and trace the client’s activity. High grace counts typically correlate with mobile clients on poor networks or clients that fail to consume the TCP send buffer.

4. **Confirm graceful shutdown timing**  
   During shutdown, the relay logs a sequence starting with `Starting graceful drain (30s timeout)` followed by grace period details. Verify that slow clients identified by high grace counters are closed before the 30-second hard deadline expires.

5. **Adjust the limit dynamically**  
   To tolerate more transient back-pressure, increase the limit before starting the relay:  
   ```rust
   // Example: Overriding the grace limit via env var
   std::env::set_var("BUZZ_SLOW_CLIENT_GRACE_LIMIT", "5");
   let state = State::new(Config::from_env()).await?;
   ```

6. **Re-run the scenario**  
   Reproduce the slow-client condition using network throttling tools (e.g., `tc` on Linux or Network Link Conditioner on macOS) and verify the new limit behaves as expected.

## Graceful Shutdown Interaction

The grace-limit mechanism intersects with the relay’s shutdown sequence defined in [`crates/buzz-relay/src/main.rs#L1244-L1278`](https://github.com/block/buzz/blob/main/crates/buzz-relay/src/main.rs#L1244). During a normal shutdown, Buzz initiates a **5-second grace period** before enforcing a **30-second hard-drain timeout**.

While the grace window is open, the server stops accepting new traffic but continues servicing outstanding events. Connections flagged as slow (those that have hit their grace limit) are prioritized for early termination. This ensures a single misbehaving client cannot block the entire shutdown process until the 30-second deadline.

## Configuration Tuning and Common Pitfalls

When tuning `BUZZ_SLOW_CLIENT_GRACE_LIMIT`, avoid these common mistakes:

- **Misinterpreting transient spikes**: A single buffer-full warning is normal under load; only consecutive occurrences indicate a truly slow client. Do not raise the limit in response to one-off warnings.
- **Setting the limit to zero**: The startup assertion will panic. The minimum valid value is `1`.
- **Confusing graceful periods**: The connection-level grace counter (`grace_limit`) is distinct from the shutdown grace period (5 seconds). Both share similar logging prefixes but serve different purposes—one tracks buffer health, the other tracks server lifecycle.

You can programmatically check if a connection is flagged as slow:

```rust
// Example: Manually checking a connection’s grace counter
fn is_slow(conn: &Connection) -> bool {
    conn.grace_counter >= conn.grace_limit
}

```

For integration testing, simulate a slow client by setting a low grace limit and stalling the consumer:

```rust
// Example: Simulating a slow client in tests
#[tokio::test]
async fn test_slow_client_disconnect() {
    // Set a low grace limit to trigger disconnect quickly.
    std::env::set_var("BUZZ_SLOW_CLIENT_GRACE_LIMIT", "2");
    // … spin up a relay and a client that deliberately stalls …
}

```

## Summary

- The **grace-limit mechanism** in Buzz tracks consecutive send-buffer-full events using a `u8` counter per connection.
- The limit is configured via `Config::slow_client_grace_limit` and can be overridden with the `BUZZ_SLOW_CLIENT_GRACE_LIMIT` environment variable.
- Warning logs containing `send buffer full — grace` indicate a client is approaching the threshold.
- The counter resets on successful sends, preventing transient spikes from causing disconnections.
- During graceful shutdown, flagged connections are closed early to ensure the 30-second hard deadline is respected.
- The value must be greater than zero; a zero configuration will trigger a panic at startup.

## Frequently Asked Questions

### What triggers the grace limit counter in Buzz?

The counter increments each time the relay attempts to send data to a client but finds the outbound buffer full. This typically occurs when the client’s network interface cannot consume data as fast as the relay produces it, creating back-pressure.

### How do I adjust the grace limit for slow clients?

Set the `BUZZ_SLOW_CLIENT_GRACE_LIMIT` environment variable before starting the relay. The value must be a positive integer (`u8`). For example, `export BUZZ_SLOW_CLIENT_GRACE_LIMIT=5` allows five consecutive buffer-full events before termination, whereas the default may be lower depending on your configuration.

### What is the difference between the grace limit and the shutdown grace period?

The **grace limit** is a per-connection counter tracking send buffer health, configured via `slow_client_grace_limit`. The **shutdown grace period** is a global 5-second window during server shutdown before the 30-second hard timeout begins. They share the term "grace" in logs but operate independently—one protects the server from slow clients, the other ensures timely process termination.

### Why did my connection close during a buffer spike?

If the spike was brief, the connection should not have closed because the grace counter resets to zero after each successful send. However, if the spike consisted of multiple consecutive full-buffer events exceeding your configured `grace_limit`, the relay treats this as sustained slowness and terminates the connection to protect overall system health.