How to Debug Slow Client Disconnections and the Grace Limit Mechanism in Buzz

Buzz uses a configurable grace-limit counter to detect slow clients when send buffers fill up, logging warnings before terminating connections that repeatedly trigger back-pressure, while the same mechanism ensures graceful shutdowns complete within the 30-second hard deadline.

The buzz-relay server must protect itself from clients that consume data too slowly, whether due to poor network conditions or malicious stalling. Understanding how to debug these disconnections requires familiarity with the grace-limit mechanism, a back-pressure counter defined in the Buzz source code that tracks consecutive buffer-full events per connection.

Understanding the Grace-Limit Mechanism

The grace limit is a defense mechanism implemented in crates/buzz-relay/src/connection.rs that monitors outbound buffer pressure on a per-connection basis.

Configuration and Environment Variables

The limit is defined as a u8 in Config::slow_client_grace_limit at crates/buzz-relay/src/config.rs#L178. You can override the default value at runtime using the environment variable BUZZ_SLOW_CLIENT_GRACE_LIMIT, which is parsed during initialization in the same file (lines 657–1219).

The configuration enforces a strict validation: the value must be greater than zero. An assertion at startup (assert!(config.slow_client_grace_limit > 0)) will panic if you attempt to set the limit to zero.

Per-Connection Back-Pressure Detection

Each Connection struct maintains a grace_limit: u8 field (declared at crates/buzz-relay/src/connection.rs#L85) alongside a transient counter. When a send operation encounters a full buffer, the server increments this counter and emits a structured warning log:

// Logic around crates/buzz-relay/src/connection.rs#L104-L109
if count >= self.grace_limit {
    warn!(conn_id = %self.conn_id, count, grace = self.grace_limit,
         "send buffer full — grace {count}/{}", self.grace_limit);
}

This log line is the primary diagnostic signal that a client is approaching termination.

Reset Behavior and Termination Logic

A successful send resets the grace counter to zero (implemented at crates/buzz-relay/src/connection.rs#L93), ensuring that transient spikes do not accumulate against the limit. Only consecutive buffer-full events trigger termination.

When the counter reaches the configured limit, the connection is flagged for closure. The actual cancellation occurs during the next graceful-shutdown cycle or background cleanup, specifically around lines 1478–1481 in connection.rs.

Debugging Slow Client Disconnections Step by Step

Follow this structured approach to diagnose whether a client disconnection is caused by the grace-limit mechanism:

1. Verify the configured limit
   Check the runtime configuration or environment variable to confirm the threshold:

   env | grep BUZZ_SLOW_CLIENT_GRACE_LIMIT

   Alternatively, inspect the generated Config object in your relay startup logs.

2. Observe warning logs
   Search relay logs for the diagnostic pattern:

   grep "grace" /var/log/buzz-relay.log

   Look for entries matching:

   
   WARN  conn_id=abc-123 count=3 grace=3 "send buffer full — grace 3/3"
   

3. Correlate with client behavior
   Extract the conn_id from warning logs and trace the client’s activity. High grace counts typically correlate with mobile clients on poor networks or clients that fail to consume the TCP send buffer.

4. Confirm graceful shutdown timing
   During shutdown, the relay logs a sequence starting with Starting graceful drain (30s timeout) followed by grace period details. Verify that slow clients identified by high grace counters are closed before the 30-second hard deadline expires.

5. Adjust the limit dynamically
   To tolerate more transient back-pressure, increase the limit before starting the relay:

   // Example: Overriding the grace limit via env var
   std::env::set_var("BUZZ_SLOW_CLIENT_GRACE_LIMIT", "5");
   let state = State::new(Config::from_env()).await?;

6. Re-run the scenario
   Reproduce the slow-client condition using network throttling tools (e.g., tc on Linux or Network Link Conditioner on macOS) and verify the new limit behaves as expected.

Graceful Shutdown Interaction

The grace-limit mechanism intersects with the relay’s shutdown sequence defined in crates/buzz-relay/src/main.rs#L1244-L1278. During a normal shutdown, Buzz initiates a 5-second grace period before enforcing a 30-second hard-drain timeout.

While the grace window is open, the server stops accepting new traffic but continues servicing outstanding events. Connections flagged as slow (those that have hit their grace limit) are prioritized for early termination. This ensures a single misbehaving client cannot block the entire shutdown process until the 30-second deadline.

Configuration Tuning and Common Pitfalls

When tuning BUZZ_SLOW_CLIENT_GRACE_LIMIT, avoid these common mistakes:

• Misinterpreting transient spikes: A single buffer-full warning is normal under load; only consecutive occurrences indicate a truly slow client. Do not raise the limit in response to one-off warnings.
• Setting the limit to zero: The startup assertion will panic. The minimum valid value is 1.
• Confusing graceful periods: The connection-level grace counter (grace_limit) is distinct from the shutdown grace period (5 seconds). Both share similar logging prefixes but serve different purposes—one tracks buffer health, the other tracks server lifecycle.

You can programmatically check if a connection is flagged as slow:

// Example: Manually checking a connection’s grace counter
fn is_slow(conn: &Connection) -> bool {
    conn.grace_counter >= conn.grace_limit
}

For integration testing, simulate a slow client by setting a low grace limit and stalling the consumer:

// Example: Simulating a slow client in tests
#[tokio::test]
async fn test_slow_client_disconnect() {
    // Set a low grace limit to trigger disconnect quickly.
    std::env::set_var("BUZZ_SLOW_CLIENT_GRACE_LIMIT", "2");
    // … spin up a relay and a client that deliberately stalls …
}

Summary

• The grace-limit mechanism in Buzz tracks consecutive send-buffer-full events using a u8 counter per connection.
• The limit is configured via Config::slow_client_grace_limit and can be overridden with the BUZZ_SLOW_CLIENT_GRACE_LIMIT environment variable.
• Warning logs containing send buffer full — grace indicate a client is approaching the threshold.
• The counter resets on successful sends, preventing transient spikes from causing disconnections.
• During graceful shutdown, flagged connections are closed early to ensure the 30-second hard deadline is respected.
• The value must be greater than zero; a zero configuration will trigger a panic at startup.

Frequently Asked Questions

What triggers the grace limit counter in Buzz?

The counter increments each time the relay attempts to send data to a client but finds the outbound buffer full. This typically occurs when the client’s network interface cannot consume data as fast as the relay produces it, creating back-pressure.

How do I adjust the grace limit for slow clients?

Set the BUZZ_SLOW_CLIENT_GRACE_LIMIT environment variable before starting the relay. The value must be a positive integer (u8). For example, export BUZZ_SLOW_CLIENT_GRACE_LIMIT=5 allows five consecutive buffer-full events before termination, whereas the default may be lower depending on your configuration.

What is the difference between the grace limit and the shutdown grace period?

The grace limit is a per-connection counter tracking send buffer health, configured via slow_client_grace_limit. The shutdown grace period is a global 5-second window during server shutdown before the 30-second hard timeout begins. They share the term "grace" in logs but operate independently—one protects the server from slow clients, the other ensures timely process termination.

Why did my connection close during a buffer spike?

If the spike was brief, the connection should not have closed because the grace counter resets to zero after each successful send. However, if the spike consisted of multiple consecutive full-buffer events exceeding your configured grace_limit, the relay treats this as sustained slowness and terminates the connection to protect overall system health.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}