# How to Configure the Server Shutdown Timeout for Graceful Draining in NVIDIA Switchyard

> Configure the NVIDIA Switchyard server shutdown timeout for graceful draining. Override the default 30-second wait for in-flight requests using CLI flags, Rust structs, or Python parameters.

- Repository: [NVIDIA-NeMo/Switchyard](https://github.com/NVIDIA-NeMo/Switchyard)
- Tags: how-to-guide
- Published: 2026-08-17

---

**Switchyard performs a graceful drain on shutdown by waiting for in-flight requests to complete, with a default timeout of 30 seconds that you can override via the `--shutdown_timeout` CLI flag, the `ServerRunOptions` struct in Rust, or the `timeout_secs` parameter in Python.**

NVIDIA Switchyard is a high-performance HTTP inference server that supports graceful shutdown to prevent abrupt termination of active requests. Configuring the server shutdown timeout ensures your deployment waits long enough for in-flight inference to complete before exiting. This guide covers three methods to set this timeout using the official `NVIDIA-NeMo/Switchyard` source code.

## Understanding the Graceful Shutdown Timeout

The graceful shutdown mechanism stops accepting new connections while allowing existing requests to finish processing. According to the source code in [`crates/switchyard-server/src/lib.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/lib.rs), the constant `DEFAULT_GRACEFUL_SHUTDOWN_TIMEOUT` defines a 30-second default duration. When the shutdown signal triggers, the server invokes `handle.graceful_shutdown(Some(timeout))` within the `serve_until_shutdown` function to begin the draining process.

## Configuring the Shutdown Timeout

### Command-Line Interface (Rust Binary)

When running the `switchyard-server` binary, pass the `--shutdown_timeout` flag with a human-readable duration format. The CLI parser in [`crates/switchyard-server/src/cli.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/cli.rs) (lines 43-46) populates the `ServerRunOptions.shutdown_timeout` field with your specified value.

```bash

# Run the server with a 45-second graceful shutdown window

switchyard-server \
  --config path/to/deployment.toml \
  --shutdown_timeout 45s

```

### Programmatic Rust API

For embedded deployments, construct a `ServerRunOptions` struct and set the `shutdown_timeout` field to a `std::time::Duration` before calling `run_server`. The function signature in [`crates/switchyard-server/src/lib.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/lib.rs) (lines 90-101) accepts these options to configure the server's behavior.

```rust
use switchyard_server::{run_server, ServerRunOptions, ServerState};

let (state, _) = load_server_state("deployment.toml")?;
let options = ServerRunOptions {
    addr: "0.0.0.0:4000".parse().unwrap(),
    backlog: switchyard_server::DEFAULT_LISTEN_BACKLOG,
    dry_run: false,
    shutdown_timeout: std::time::Duration::from_secs(60), // 1-minute timeout
    tls: None,
};

run_server(state, options).await?;

```

### Python Bindings

When using `switchyard-py`, call `Server.close(timeout_secs=...)` where the parameter accepts a float representing seconds. Note that the Python default differs from the Rust server: the constant `DEFAULT_SHUTDOWN_TIMEOUT_SECS` in [`crates/switchyard-py/src/server_bindings.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-py/src/server_bindings.rs) (lines 22-30) sets a 2-second default for the Python API.

```python
from switchyard_rust.server import Server

# Start a loopback server

srv = Server("deployment.toml")
print(f"Listening at {srv.base_url}")

# Later, request a graceful shutdown with a 5-second timeout

srv.close(timeout_secs=5.0)

```

## Implementation Details and Timeout Behavior

Internally, Switchyard converts your timeout parameter into a `std::time::Duration`. If active requests complete before the timeout expires, the server shuts down immediately. However, if the timeout elapses first, the server aborts remaining tasks. This logic resides in `close_inner` within [`crates/switchyard-py/src/server_bindings.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-py/src/server_bindings.rs) (lines 33-51), which handles the forced termination of dangling connections.

## Summary

- The default graceful shutdown timeout is **30 seconds** in the Rust server but **2 seconds** in Python bindings.
- Override via CLI with `--shutdown_timeout`, Rust with `ServerRunOptions.shutdown_timeout`, or Python with `timeout_secs`.
- The server calls `graceful_shutdown` in `serve_until_shutdown` to initiate the draining process.
- Unfinished requests are aborted if they exceed the configured timeout during shutdown.

## Frequently Asked Questions

### What happens if requests exceed the shutdown timeout?

If in-flight requests do not complete before the timeout expires, Switchyard aborts the remaining tasks and proceeds with shutdown. This prevents the server from hanging indefinitely during deployment rollouts or pod terminations in Kubernetes.

### Can I disable the shutdown timeout entirely?

While the internal `graceful_shutdown` method accepts `None` to wait indefinitely, the current Switchyard implementations require a finite duration. You can effectively disable the timeout by setting an extremely large value, such as `24h` in the CLI or `Duration::from_secs(86400)` in Rust.

### Why does the Python API use a different default timeout?

The Python bindings default to 2 seconds (`DEFAULT_SHUTDOWN_TIMEOUT_SECS`) because Python-based inference scripts often run in interactive or testing environments where rapid shutdown is preferred. The Rust server assumes production workloads requiring the standard 30-second window.

### How do I format the duration for the CLI flag?

The `--shutdown_timeout` flag accepts human-readable formats such as `30s` for 30 seconds, `2m` for 2 minutes, or `1h` for 1 hour. The parser converts these strings into a `Duration` before passing them to the server configuration.