How to Debug gRPC Connection Issues in r-nacos Using RNACOS_GRPC_DETECTION_TIMEOUT_SECOND

Set the RNACOS_GRPC_DETECTION_TIMEOUT_SECOND environment variable to increase the idle-detection interval (default 15 seconds) and enable RNACOS_ENABLE_GRPC_DETECTION_LOG=true to trace why r-nacos closes gRPC streams.

r-nacos is a lightweight, Rust-based implementation of the Nacos service discovery and configuration management platform. When troubleshooting dropped gRPC connections between clients and the r-nacos server, understanding the heartbeat detection mechanism controlled by the RNACOS_GRPC_DETECTION_TIMEOUT_SECOND environment variable is essential for identifying whether the server is prematurely closing idle streams.

How the gRPC Detection Timeout Works

Environment Variable Loading

The timeout configuration originates in src/common/mod.rs, where the server reads the environment variable at startup (lines 52-56):

let grpc_detection_timeout = std::env::var("RNACOS_GRPC_DETECTION_TIMEOUT_SECOND")
    .unwrap_or("15".to_owned())
    .parse()
    .unwrap_or(15)
    * 1000;          // convert to ms

This value is stored in AppSysConfig.grpc_detection_timeout. If the variable is missing or unparsable, the system falls back to 15 seconds (15000 ms).

Stream Manager Initialization

During the Actix dependency injection phase, BiStreamManage receives the parsed timeout in src/grpc/bistream_manage.rs (lines 93-99):

if let Some(sys_config) = factory_data.get_bean::<crate::common::AppSysConfig>() {
    self.detection_time_out = sys_config.grpc_detection_timeout;
    log::info!(
        "BiStreamManage inject complete, detection_time_out:{}",
        self.detection_time_out
    );
}

The detection_time_out field now drives the idle detection logic for all bidirectional gRPC streams.

Heartbeat Detection Cycle

The manager runs a periodic task (time_out_heartbeat) that checks two timeout sets:

// src/grpc/bistream_manage.rs (lines 63-71)
ctx.run_later(Duration::new(2, 0), |act, ctx| {
    let now = now_millis();
    act.check_active_time_set(now);
    act.check_response_time_set(now);
    act.time_out_heartbeat(ctx);
});

• check_active_time_set identifies idle streams (last_active_time + detection_time_out ≤ now). For each idle client, the server sends a Detection request (BiStreamSenderCmd::Detection).
• check_response_time_set ensures the client replies within the response timeout (self.response_time_out). If the client fails to answer, the server closes the stream (BiStreamSenderCmd::Close).

Configuring RNACOS_GRPC_DETECTION_TIMEOUT_SECOND

Increasing the Idle Timeout

If your gRPC clients experience frequent disconnections during periods of low activity, increase the detection window. Set the environment variable before starting the r-nacos server:

export RNACOS_GRPC_DETECTION_TIMEOUT_SECOND=60

This extends the idle tolerance to 60 seconds (60000 ms), allowing connections to remain open during longer quiet periods before the server probes them.

Enabling Debug Logging with RNACOS_ENABLE_GRPC_DETECTION_LOG

To observe the detection flow in server logs, enable the verbose logging flag:

export RNACOS_ENABLE_GRPC_DETECTION_LOG=true

This variable is evaluated in src/grpc/handler/mod.rs (lines 95-99) to control whether ServerCheck (detection) requests are logged:

if SERVER_CHECK_REQUEST.eq(url) {
    if self.app.sys_config.enable_grpc_detection_log {
        return HandleLogArgs::None;   // keep normal log output
    }
    return HandleLogArgs::Ignore;     // suppress noise
}

With this enabled, you will see entries detailing when detection requests are sent and whether clients respond.

Practical Debugging Steps

Verify Current Configuration

Check the active timeout value by inspecting the environment or the server startup logs:

echo $RNACOS_GRPC_DETECTION_TIMEOUT_SECOND

Alternatively, examine the log line emitted by BiStreamManage during initialization:

[INFO] BiStreamManage inject complete, detection_time_out:15000

Monitor Server Logs

Run the server with detection logging enabled and filter for the relevant components:

RUST_LOG=info cargo run --release | grep -E "(BiStreamManage|check timeout)"

Look for these key patterns:

• check timeout detection client, size:N – Indicates N idle streams are being probed.
• check timeout close client, size:N – Indicates N streams were closed due to missing responses.

If close client counts increase immediately after startup, the detection timeout is likely too aggressive for your network latency.

Correlate with Client Behavior

On the client side (for example, using the Rust SDK in sdk-examples/rust/nacos_rust_client/), enable debug logging to confirm receipt of detection requests:

RUST_LOG=debug ./client_binary

Verify that the client logs indicate it is receiving Detection messages and sending responses. If the client never logs receipt, investigate network intermediaries (load balancers, proxies) that might drop or buffer the detection packets.

Key Source Files to Review

File	Purpose	Direct Link
src/common/mod.rs	Loads RNACOS_GRPC_DETECTION_TIMEOUT_SECOND and RNACOS_ENABLE_GRPC_DETECTION_LOG into AppSysConfig.	src/common/mod.rs
src/grpc/bistream_manage.rs	Implements BiStreamManage with detection_time_out, check_active_time_set, and check_response_time_set logic.	src/grpc/bistream_manage.rs
src/grpc/handler/mod.rs	Controls detection request logging via enable_grpc_detection_log flag.	src/grpc/handler/mod.rs
sdk-examples/rust/nacos_rust_client/config/src/main.rs	Reference client implementation for testing detection behavior.	sdk-examples/rust/nacos_rust_client/config/src/main.rs

Summary

• Set RNACOS_GRPC_DETECTION_TIMEOUT_SECOND to configure how long r-nacos waits before probing idle gRPC streams (default 15 seconds, converted to milliseconds in src/common/mod.rs).
• Enable RNACOS_ENABLE_GRPC_DETECTION_LOG=true to expose detection events in server logs, helping identify when BiStreamManage sends probes or closes unresponsive clients.
• Monitor BiStreamManage logs for check timeout detection client and check timeout close client to determine if timeouts are too aggressive.
• Verify client-side behavior to ensure clients respond to Detection requests within the response window defined in src/grpc/bistream_manage.rs.

Frequently Asked Questions

What is the default value of RNACOS_GRPC_DETECTION_TIMEOUT_SECOND?

The default value is 15 seconds. If the environment variable is not set or contains an invalid value, r-nacos falls back to 15 seconds in src/common/mod.rs and multiplies it by 1000 to store 15000 milliseconds in AppSysConfig.grpc_detection_timeout.

Why does r-nacos close my gRPC connection immediately after startup?

Immediate closures usually indicate that the detection timeout is too short for your network latency or that the client is not responding to Detection requests. Check the server logs for check timeout close client messages. If these appear shortly after connection establishment, increase RNACOS_GRPC_DETECTION_TIMEOUT_SECOND to 60 seconds or higher to allow more time for initial handshakes.

How do I enable verbose logging for gRPC detection events?

Set the environment variable RNACOS_ENABLE_GRPC_DETECTION_LOG=true before starting the server. This flag is evaluated in src/grpc/handler/mod.rs and controls whether ServerCheck (detection) requests are logged. When enabled, you will see detailed entries in the server logs showing when detection requests are sent and whether clients are responding.

Can I adjust the response timeout as well as the detection timeout?

The response timeout is controlled by a separate internal constant (self.response_time_out) within BiStreamManage in src/grpc/bistream_manage.rs, distinct from the configurable detection interval. While RNACOS_GRPC_DETECTION_TIMEOUT_SECOND controls how long the server waits before sending a detection probe, the response timeout determines how long the server waits for the client to answer that probe. To modify the response timeout, you must change the constant in the source code and recompile, unlike the detection timeout which is configurable via environment variable.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}