How Switchyard Handles LLM Request Metadata: Session ID and Correlation ID

Switchyard normalizes heterogeneous HTTP headers into a canonical Metadata struct that extracts session IDs and correlation IDs, attaches them to internal request objects for distributed tracing, and propagates them downstream to LLM backends to maintain end-to-end observability.

Switchyard is NVIDIA's open-source inference router for large language models. Understanding how Switchyard handles LLM request metadata like session ID and correlation ID is critical for debugging multi-turn conversations and maintaining trace context across distributed services. The system implements a vendor-agnostic normalization layer that reconciles conflicting header conventions from OpenAI, Claude, Codex, and internal NVIDIA services into a unified metadata envelope.

From HTTP Headers to Structured Metadata

The normalization process begins in [crates/protocol/src/metadata.rs](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/protocol/src/metadata.rs), where the Metadata::from_headers function transforms incoming HTTP headers into a structured representation. This function consults a precedence table defined in HEADER_CONFIG to resolve conflicts, ensuring that explicit x-switchyard-* headers take priority over vendor-specific aliases. The implementation also handles nested JSON values in headers (such as x-codex-turn-metadata.session_id) and falls back to flat header values when JSON parsing yields empty results.

Session ID and Correlation ID Extraction

Switchyard extracts two critical identifiers from the normalized headers to track requests through the system.

Session ID for Conversation State

The session_id field provides a stable identifier for multi-turn conversations, enabling Switchyard to maintain context across related requests. The extractor maps multiple vendor headers to this canonical field, including x-switchyard-session-id, x-claude-code-session-id, x-nemo-relay-session-id, and the generic session-id. This aliasing allows clients using different LLM providers to integrate without modifying their header conventions.

Correlation ID for Distributed Tracing

The correlation_id (also referenced as request_id in tracing contexts) enables end-to-end request tracing across service boundaries. Switchyard extracts this value from x-switchyard-request-id or its aliases x-request-id and x-client-request-id, then injects it into OpenTelemetry spans and Prometheus metrics.

Server-Side Metadata Processing

When requests arrive at the Rust HTTP server, the metadata integration happens at the entry point and continues through the observability stack.

Request Lifecycle Initialization

In [crates/switchyard-server/src/lib.rs](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/lib.rs), the inbound HeaderMap is immediately converted into a Metadata instance using Metadata::from_headers. This metadata is then attached to the internal switchyard_protocol::Request object, making the identifiers available to all downstream processing stages including routing algorithms and sub-agent dispatchers.

Observability Integration

The [crates/switchyard-server/src/observability.rs](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/observability.rs) module creates a tracing span for each incoming request that explicitly records session_id and correlation_id. This integration ensures that logs and metrics emitted during request processing carry these identifiers, allowing operators to correlate events across Switchyard instances and external services.

Routing and Downstream Propagation

Switchyard does not merely extract metadata for internal use; it actively propagates these identifiers to maintain context across service boundaries.

Sub-Agent State Management

The Metadata struct drives sub-agent routing decisions through fields like is_subagent and is_subagent_work. The session ID serves as a key for per-session state in Switchyard's routing algorithms, enabling sticky routing where subsequent requests in a conversation reach the same backend or context pool.

Backend Header Forwarding

When forwarding requests to LLM backends via the libsy-llm-client crate, specifically in [crates/libsy-llm-client/src/client.rs](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/libsy-llm-client/src/client.rs), Switchyard includes the correlation ID in outbound HTTP headers. This propagation ensures that backend services can participate in the same distributed trace, maintaining observability continuity from the initial client request through the final LLM provider response.

Implementing Metadata Handling in Python

Switchyard exposes its metadata handling through Python bindings defined in [crates/switchyard-py/src/lib.rs](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-py/src/lib.rs). The following example demonstrates how to construct a request with explicit session and correlation identifiers:

import switchyard
from switchyard import Request, Metadata

# Build a request with explicit metadata headers

metadata_headers = [
    ("x-switchyard-session-id", "session-1234"),
    ("x-switchyard-request-id", "trace-5678"),
]
metadata = Metadata.from_headers(metadata_headers)

req = Request(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello"}],
    metadata=metadata,
)

# Send through Switchyard (the Python façade forwards to the Rust server)

response = switchyard.run(req)
print(response.metadata.session_id)      # → "session-1234"

print(response.metadata.correlation_id) # → "trace-5678"

Summary

• Switchyard normalizes vendor-specific HTTP headers into a canonical Metadata struct using Metadata::from_headers in crates/protocol/src/metadata.rs.
• Session IDs enable multi-turn conversation tracking across requests, extracted from headers like x-switchyard-session-id and x-claude-code-session-id.
• Correlation IDs facilitate distributed tracing, sourced from x-switchyard-request-id or its aliases and recorded in OpenTelemetry spans via the observability module.
• The server attaches metadata to internal Request objects at the entry point in crates/switchyard-server/src/lib.rs.
• Downstream propagation to LLM backends occurs through crates/libsy-llm-client/src/client.rs, ensuring end-to-end trace continuity.

Frequently Asked Questions

What HTTP headers does Switchyard accept for session identification?

Switchyard accepts multiple header aliases for session IDs, including x-switchyard-session-id, x-claude-code-session-id, x-nemo-relay-session-id, and the generic session-id. The system uses a precedence table where explicit Switchyard headers take priority over vendor-specific alternatives.

How does Switchyard resolve conflicting metadata headers from different LLM providers?

The Metadata::from_headers function implements a precedence hierarchy defined in HEADER_CONFIG, ensuring that x-switchyard-* headers override vendor aliases like OpenAI or Claude headers. For nested JSON values (e.g., x-codex-turn-metadata.session_id), the parser extracts internal fields while falling back to flat header values when JSON is absent or empty.

Can correlation IDs be traced through downstream LLM providers?

Yes. When Switchyard forwards requests via crates/libsy-llm-client/src/client.rs, it includes the correlation ID in outbound HTTP headers, allowing backend LLM services to participate in the same distributed trace. This propagation ensures that logs and metrics from both Switchyard and the upstream provider share the same request identifier.

How is session state maintained across multiple requests in Switchyard?

Switchyard uses the normalized session_id field as a key for per-session state in its routing algorithms, particularly when handling sub-agent requests. This enables sticky routing and context preservation across multiple turns of a conversation, even when requests traverse different server instances.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}