How Switchyard Handles Streaming Responses Between OpenAI and Anthropic Formats

Switchyard abstracts streaming API differences by converting every provider chunk into a neutral LlmResponseChunk representation, then re-encoding it for the target format through a decode-state-encode pipeline.

Unlike other translation layers that require manual mapping, NVIDIA's Switchyard solves the OpenAI-Anthropic streaming incompatibility with a unified intermediate representation. The system lets you stream from Claude and consume as OpenAI-format chunks (or vice versa) without client-side changes. This article examines the three-stage pipeline, key data mappings, and the Rust implementation powering this translation.

The Three-Stage Translation Pipeline

Switchyard's streaming architecture in crates/switchyard-translation/src/engine.rs follows a consistent pattern regardless of source or destination provider:

1. Decode — Parse provider-specific JSON events into neutral chunks
2. Maintain State — Track stream lifecycle via StreamTranslationState
3. Encode — Emit target-format chunks from neutral representation

Stage 1: Decode with Provider-Specific Codecs

Each supported provider implements a stream codec with two core methods: decode_event for inbound translation and encode_* for outbound generation.

The OpenAiChatStreamCodec processes OpenAI's chat.completion.chunk events. Its decode_event implementation in crates/switchyard-translation/src/codecs/openai_chat/stream.rs (lines 24-48) parses JSON deltas and extracts:

• Text content from choices[0].delta.content
• Tool call fragments from choices[0].delta.tool_calls
• Finish reasons from choices[0].finish_reason
• Usage statistics from top-level usage objects

Similarly, AnthropicMessagesStreamCodec handles Anthropic's Server-Sent Events format with message_start, content_block_start, content_block_delta, and message_stop event types.

Both codecs output the same neutral type: LlmResponseChunk.

Stage 2: State Management Per Stream

The StreamTranslationState struct in engine.rs (lines 1-30) persists across all chunks in a single request-response cycle. It tracks:

• Whether MessageStart has been emitted
• Accumulated token usage (prompt and completion)
• Cached model identifier
• Stop reason and error flags
• Pending tool call state

This state enables correct encoding decisions. For example, Anthropic does not emit usage in its streaming format—Switchyard preserves usage data in state so it can be attached to final OpenAI-format chunks when that is the target.

Stage 3: Encode for Target Format

The target codec transforms neutral chunks into provider-compliant JSON. Two encoding functions demonstrate the symmetry:

OpenAI encoding (encode_openai_chat_stream, lines 60-89 in openai_chat/stream.rs) builds objects with:

• id, object, created, model metadata
• choices[0].delta containing content/tool_calls
• usage and finish_reason on terminal chunks

Anthropic encoding (AnthropicMessagesStreamCodec, lines 30-78 in anthropic/stream.rs) produces:

• message_start with model and role
• content_block_delta with thinking or text types
• message_delta with stop reason
• message_stop to signal completion

Key Data Mappings: Neutral to Provider Formats

The LlmResponseChunk enum provides six variants that map differently to OpenAI and Anthropic outputs:

Neutral Chunk	OpenAI Output	Anthropic Output
TextDelta { text }	"content": "<text>"	"content": "<text>"
ReasoningDelta { text }	"reasoning_content": "<text>"	"reasoning_content": "<text>"
ToolCallDelta { index, id, name, arguments_delta }	"tool_calls[].function.{name,arguments}" with index, id	"tool_calls[].{name,input,id,type,index}"
Usage	Full usage object	State-only (not emitted in stream)
MessageStop { reason }	"finish_reason": "stop|length|tool_calls"	"stop_reason": "end_turn|max_tokens|tool_use"
StreamError / DecodeError	{"error":{"message":"..."}} (terminal)	Same error frame

Finish-Reason Normalization

Bidirectional mapping requires vocabulary translation. The helper openai_finish_reason in openai_chat/stream.rs (lines 94-101) converts:

• Anthropic end_turn → OpenAI stop
• Anthropic max_tokens → OpenAI length
• Anthropic tool_use → OpenAI tool_calls

The inverse mapping in the Anthropic codec ensures round-trip fidelity.

Error Handling: Terminal Stream Failure

Both codecs treat decode errors as stream-terminating events. From openai_chat/stream.rs (lines 30-36), when DecodeError occurs:

1. Emit an error chunk with {"error":{"message":"..."}} structure
2. Set state.errored = true
3. Drop all subsequent chunks from the source stream

This prevents partial or corrupted translations from propagating downstream.

Streaming Flow in Practice

When using Switchyard's Python interface in switchyard_rust/libsy.py, the translation engine operates transparently:

import asyncio
from switchyard import launch

async def stream_comparison():
    # Same consumption code works for both providers

    for provider in ["openai", "claude"]:
        client = launch(provider, model="switchyard", stream=True)
        
        async for chunk in client.run_stream({"prompt": "Explain quantum computing"}):
            # chunk is always LlmResponseChunk regardless of source

            match chunk.type:
                case "TextDelta":
                    print(chunk.text, end="", flush=True)
                case "ToolCallDelta":
                    print(f"\n[Tool: {chunk.name}]")
                case "MessageStop":
                    print(f"\n[Finished: {chunk.reason}]")

asyncio.run(stream_comparison())

The underlying Rust translation guarantees format-agnostic consumption. The run_stream method in libsy.py drives the engine from crates/switchyard-translation/src/stream.rs, which re-exports the public streaming API.

Implementation Reference

File	Purpose	Key Components
crates/switchyard-translation/src/engine.rs	Orchestration layer	StreamTranslationState, translation engine
crates/switchyard-translation/src/stream.rs	Public API surface	pub use crate::codecs::stream::*
crates/switchyard-translation/src/codecs/openai_chat/stream.rs	OpenAI codec	OpenAiChatStreamCodec, encode_openai_chat_stream, openai_finish_reason
crates/switchyard-translation/src/codecs/anthropic/stream.rs	Anthropic codec	AnthropicMessagesStreamCodec, reasoning_content handling
switchyard_rust/libsy.py	Python bindings	Async run_stream method
examples/libsy.py	Usage demonstration	End-to-end streaming examples

Summary

Switchyard handles streaming response translation between OpenAI and Anthropic formats through three architectural decisions:

• Neutral intermediate representation — LlmResponseChunk decouples source and target formats
• Stateful stream tracking — StreamTranslationState preserves cross-chunk information like usage and stop reasons
• Bidirectional codec symmetry — Each provider implements decode and encode, enabling any-to-any translation

The system preserves reasoning content, tool calls, usage statistics, and finish reasons across format boundaries, with terminal error handling that prevents partial translation failures.

Frequently Asked Questions

What is the LlmResponseChunk type in Switchyard?

LlmResponseChunk is a Rust enum defined in the translation engine that represents a streaming response in provider-neutral form. It has variants for text deltas, reasoning content, tool call fragments, usage data, stream completion, and errors. All provider codecs convert to and from this type, making it the central abstraction for cross-format streaming.

Does Switchyard support reasoning content from Claude in OpenAI format?

Yes. The ReasoningDelta variant carries reasoning text from Claude's thinking content blocks. When encoding to OpenAI format, Switchyard emits this as "reasoning_content": "<text>" in the delta object—a non-standard but widely-supported extension. The neutral representation ensures reasoning content is preserved regardless of translation direction.

How does Switchyard handle usage statistics when streaming from Anthropic to OpenAI?

Anthropic's streaming API does not include usage data in its SSE events. Switchyard accumulates token counts in StreamTranslationState during the decode phase. When encoding to OpenAI format, the final chunk includes a synthesized usage object with prompt_tokens and completion_tokens populated from state. This allows OpenAI-format consumers to receive complete usage information even from Anthropic sources.

What happens when a stream decode error occurs mid-response?

The codec sets state.errored = true and emits a terminal error chunk with JSON structure {"error":{"message":"..."}}. All subsequent chunks from the upstream provider are dropped. This design prevents partial or malformed translations from reaching downstream consumers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}