How Switchyard Handles Context Window Exceeded Errors and Maps Them to SwitchyardError

Switchyard converts upstream LLM "context window exceeded" errors into a unified LlmClientError::ContextWindowExceeded variant, then maps these through its routing layer, HTTP server, and Python bindings to deliver consistent SwitchyardError responses with the canonical code context_length_exceeded.

The NVIDIA NeMo Switchyard repository provides a unified gateway for routing requests to multiple LLM providers. When a prompt exceeds a model's token limit, Switchyard detects this condition early and propagates it through several architectural layers—from the Rust LLM client through to HTTP responses and Python exceptions—ensuring callers receive predictable, actionable error information regardless of their integration method.

Detection in the Rust LLM Client

The error handling pipeline begins in libsy-llm-client, Switchyard's low-level provider interface.

When an upstream LLM returns a context window overflow response—for example, OpenAI's error.code == "context_length_exceeded"—the client creates a specific error variant. This definition appears in crates/libsy-llm-client/src/client.rs:

LlmClientError::ContextWindowExceeded { model, message, .. }

This early detection allows the system to distinguish token limit violations from transient failures or authentication errors, enabling appropriate fallback strategies and clear user messaging.

Propagation Through the Routing Layer

Rather than immediately surfacing the error, Switchyard's routing algorithm in crates/libsy/src/algorithms/llm_class.rs records the condition as a routing fallback reason:

RoutingFallbackReason::ContextWindow

This distinction matters for multi-model deployments. When one model rejects a prompt for length reasons, Switchyard can attempt the request against alternative models with larger context windows. The error bubbles up to the server layer unchanged, preserving the full diagnostic information for the final response.

Mapping to HTTP Responses

The central error translation occurs in crates/switchyard-server/src/lib.rs within the client_error function. When processing LlmClientError::ContextWindowExceeded, the server constructs an ApiError with these exact fields:

Field	Value
status	400 Bad Request
error_type	"invalid_request_error"
code	"context_length_exceeded"
message	Upstream error message preserved

The implementation at lines 909-914:

LlmClientError::ContextWindowExceeded { message, .. } => error_response(
    StatusCode::BAD_REQUEST,
    message,
    "invalid_request_error",
    "context_length_exceeded",
),

The ApiError is then serialized into the provider-specific JSON envelope—matching OpenAI or Anthropic response formats—and returned to the caller.

Python Exception Mapping

For Python integrations, switchyard-py bridges Rust errors to Python exceptions. The conversion happens in two locations:

1. Exception class definition in crates/switchyard-py/src/errors.rs (lines 11-14) creates ContextWindowExceededError
2. Binding conversion in crates/switchyard-py/src/libsy_bindings.rs (lines 278-283) inspects the Rust error and wraps it appropriately

Practical Code Examples

Rust: Direct LLM Client Usage

use switchyard_libsy::client::LlmClient;
use switchyard_libsy::errors::LlmClientError;

async fn request_model() -> Result<String, LlmClientError> {
    let client = LlmClient::new(...);
    match client.chat(...).await {
        Ok(resp) => Ok(resp),
        Err(LlmClientError::ContextWindowExceeded { message, .. }) => {
            // Custom handling: truncate prompt or inform user
            log::warn!("Context window exceeded: {}", message);
            Err(LlmClientError::ContextWindowExceeded { message, .. })
        }
        Err(e) => Err(e),
    }
}

Python: Catching Mapped Exceptions

from switchyard import Switchyard
from switchyard.errors import ContextWindowExceededError

sw = Switchyard()
try:
    sw.chat(model="gpt-4", messages=[...])
except ContextWindowExceededError as e:
    print("Prompt too long:", e)
    # Application logic: trim prompt or request smaller model

HTTP: Error Response Format

curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"<very long prompt>"}]}'

Response:

{
  "error": {
    "message": "This model's maximum context length is 8192 tokens...",
    "type": "invalid_request_error",
    "code": "context_length_exceeded"
  }
}

Key Implementation Files

Component	File Path	Purpose
Server error mapping	crates/switchyard-server/src/lib.rs#L909-L914	Translates ContextWindowExceeded to HTTP 400 with standardized error code
LLM client error definition	crates/libsy-llm-client/src/client.rs#L356-L363	Defines the ContextWindowExceeded variant
Routing fallback handling	crates/libsy/src/algorithms/llm_class.rs#L530-L537	Marks error as RoutingFallbackReason::ContextWindow
Python exception class	crates/switchyard-py/src/errors.rs#L11-L14	Creates ContextWindowExceededError
Python binding conversion	crates/switchyard-py/src/libsy_bindings.rs#L278-L283	Maps Rust error to Python exception

Summary

• Early detection: libsy-llm-client captures context window violations as LlmClientError::ContextWindowExceeded
• Routing intelligence: The error triggers RoutingFallbackReason::ContextWindow for potential model failover
• Consistent HTTP responses: Server returns 400 Bad Request with code: "context_length_exceeded"
• Python parity: Rust errors convert to ContextWindowExceededError for SDK users
• Provider compatibility: Error format matches OpenAI/Anthropic conventions for drop-in replacement

Frequently Asked Questions

What HTTP status code does Switchyard return for context window exceeded errors?

Switchyard returns 400 Bad Request with error_type: "invalid_request_error" and code: "context_length_exceeded". This matches OpenAI's API behavior, making Switchyard compatible with existing client implementations expecting that specific error format.

Can Switchyard automatically retry with a different model when the context window is exceeded?

Yes. The routing layer in crates/libsy/src/algorithms/llm_class.rs classifies this error as RoutingFallbackReason::ContextWindow, allowing the scheduler to attempt the request against alternative models with larger context windows if configured in your routing policy.

How do I catch context window errors in Python code using Switchyard?

Import ContextWindowExceededError from switchyard.errors and wrap your chat() calls in a try-except block. The exception preserves the upstream message for debugging while providing a typed catch condition for programatic response—such as truncating your prompt or selecting a different model.

Where is the error code "context_length_exceeded" defined in the Switchyard source?

The canonical string "context_length_exceeded" is hardcoded in crates/switchyard-server/src/lib.rs at line 912 within the client_error function's match arm for LlmClientError::ContextWindowExceeded. This ensures consistency across all response paths regardless of which upstream provider originally signaled the error.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}