# How Switchyard Handles Streaming Responses Between OpenAI and Anthropic Formats

> Learn how Switchyard effortlessly bridges OpenAI and Anthropic streaming formats by converting API responses into a neutral format for seamless interoperability.

- Repository: [NVIDIA-NeMo/Switchyard](https://github.com/NVIDIA-NeMo/Switchyard)
- Tags: how-to-guide
- Published: 2026-08-15

---

**Switchyard abstracts streaming API differences by converting every provider chunk into a neutral `LlmResponseChunk` representation, then re-encoding it for the target format through a decode-state-encode pipeline.**

Unlike other translation layers that require manual mapping, NVIDIA's Switchyard solves the OpenAI-Anthropic streaming incompatibility with a unified intermediate representation. The system lets you stream from Claude and consume as OpenAI-format chunks (or vice versa) without client-side changes. This article examines the three-stage pipeline, key data mappings, and the Rust implementation powering this translation.

## The Three-Stage Translation Pipeline

Switchyard's streaming architecture in [`crates/switchyard-translation/src/engine.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-translation/src/engine.rs) follows a consistent pattern regardless of source or destination provider:

1. **Decode** — Parse provider-specific JSON events into neutral chunks
2. **Maintain State** — Track stream lifecycle via `StreamTranslationState`
3. **Encode** — Emit target-format chunks from neutral representation

### Stage 1: Decode with Provider-Specific Codecs

Each supported provider implements a **stream codec** with two core methods: `decode_event` for inbound translation and `encode_*` for outbound generation.

The `OpenAiChatStreamCodec` processes OpenAI's `chat.completion.chunk` events. Its `decode_event` implementation in [`crates/switchyard-translation/src/codecs/openai_chat/stream.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-translation/src/codecs/openai_chat/stream.rs) (lines 24-48) parses JSON deltas and extracts:

- Text content from `choices[0].delta.content`
- Tool call fragments from `choices[0].delta.tool_calls`
- Finish reasons from `choices[0].finish_reason`
- Usage statistics from top-level `usage` objects

Similarly, `AnthropicMessagesStreamCodec` handles Anthropic's Server-Sent Events format with `message_start`, `content_block_start`, `content_block_delta`, and `message_stop` event types.

Both codecs output the same neutral type: `LlmResponseChunk`.

### Stage 2: State Management Per Stream

The `StreamTranslationState` struct in [`engine.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/engine.rs) (lines 1-30) persists across all chunks in a single request-response cycle. It tracks:

- Whether `MessageStart` has been emitted
- Accumulated token usage (prompt and completion)
- Cached model identifier
- Stop reason and error flags
- Pending tool call state

This state enables correct encoding decisions. For example, Anthropic does not emit usage in its streaming format—Switchyard preserves usage data in state so it can be attached to final OpenAI-format chunks when that is the target.

### Stage 3: Encode for Target Format

The target codec transforms neutral chunks into provider-compliant JSON. Two encoding functions demonstrate the symmetry:

**OpenAI encoding** (`encode_openai_chat_stream`, lines 60-89 in [`openai_chat/stream.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/openai_chat/stream.rs)) builds objects with:
- `id`, `object`, `created`, `model` metadata
- `choices[0].delta` containing content/tool_calls
- `usage` and `finish_reason` on terminal chunks

**Anthropic encoding** (`AnthropicMessagesStreamCodec`, lines 30-78 in [`anthropic/stream.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/anthropic/stream.rs)) produces:
- `message_start` with model and role
- `content_block_delta` with `thinking` or `text` types
- `message_delta` with stop reason
- `message_stop` to signal completion

## Key Data Mappings: Neutral to Provider Formats

The `LlmResponseChunk` enum provides six variants that map differently to OpenAI and Anthropic outputs:

| Neutral Chunk | OpenAI Output | Anthropic Output |
|-------------|-------------|----------------|
| `TextDelta { text }` | `"content": "<text>"` | `"content": "<text>"` |
| `ReasoningDelta { text }` | `"reasoning_content": "<text>"` | `"reasoning_content": "<text>"` |
| `ToolCallDelta { index, id, name, arguments_delta }` | `"tool_calls[].function.{name,arguments}"` with `index`, `id` | `"tool_calls[].{name,input,id,type,index}"` |
| `Usage` | Full `usage` object | State-only (not emitted in stream) |
| `MessageStop { reason }` | `"finish_reason": "stop\|length\|tool_calls"` | `"stop_reason": "end_turn\|max_tokens\|tool_use"` |
| `StreamError` / `DecodeError` | `{"error":{"message":"..."}}` (terminal) | Same error frame |

### Finish-Reason Normalization

Bidirectional mapping requires vocabulary translation. The helper `openai_finish_reason` in [`openai_chat/stream.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/openai_chat/stream.rs) (lines 94-101) converts:

- Anthropic `end_turn` → OpenAI `stop`
- Anthropic `max_tokens` → OpenAI `length`
- Anthropic `tool_use` → OpenAI `tool_calls`

The inverse mapping in the Anthropic codec ensures round-trip fidelity.

## Error Handling: Terminal Stream Failure

Both codecs treat decode errors as stream-terminating events. From [`openai_chat/stream.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/openai_chat/stream.rs) (lines 30-36), when `DecodeError` occurs:

1. Emit an error chunk with `{"error":{"message":"..."}}` structure
2. Set `state.errored = true`
3. Drop all subsequent chunks from the source stream

This prevents partial or corrupted translations from propagating downstream.

## Streaming Flow in Practice

When using Switchyard's Python interface in [`switchyard_rust/libsy.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard_rust/libsy.py), the translation engine operates transparently:

```python
import asyncio
from switchyard import launch

async def stream_comparison():
    # Same consumption code works for both providers

    for provider in ["openai", "claude"]:
        client = launch(provider, model="switchyard", stream=True)
        
        async for chunk in client.run_stream({"prompt": "Explain quantum computing"}):
            # chunk is always LlmResponseChunk regardless of source

            match chunk.type:
                case "TextDelta":
                    print(chunk.text, end="", flush=True)
                case "ToolCallDelta":
                    print(f"\n[Tool: {chunk.name}]")
                case "MessageStop":
                    print(f"\n[Finished: {chunk.reason}]")

asyncio.run(stream_comparison())

```

The underlying Rust translation guarantees format-agnostic consumption. The `run_stream` method in [`libsy.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/libsy.py) drives the engine from [`crates/switchyard-translation/src/stream.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-translation/src/stream.rs), which re-exports the public streaming API.

## Implementation Reference

| File | Purpose | Key Components |
|------|---------|--------------|
| [`crates/switchyard-translation/src/engine.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-translation/src/engine.rs) | Orchestration layer | `StreamTranslationState`, translation engine |
| [`crates/switchyard-translation/src/stream.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-translation/src/stream.rs) | Public API surface | `pub use crate::codecs::stream::*` |
| [`crates/switchyard-translation/src/codecs/openai_chat/stream.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-translation/src/codecs/openai_chat/stream.rs) | OpenAI codec | `OpenAiChatStreamCodec`, `encode_openai_chat_stream`, `openai_finish_reason` |
| [`crates/switchyard-translation/src/codecs/anthropic/stream.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-translation/src/codecs/anthropic/stream.rs) | Anthropic codec | `AnthropicMessagesStreamCodec`, `reasoning_content` handling |
| [`switchyard_rust/libsy.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard_rust/libsy.py) | Python bindings | Async `run_stream` method |
| [`examples/libsy.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/examples/libsy.py) | Usage demonstration | End-to-end streaming examples |

## Summary

Switchyard handles streaming response translation between OpenAI and Anthropic formats through three architectural decisions:

- **Neutral intermediate representation** — `LlmResponseChunk` decouples source and target formats
- **Stateful stream tracking** — `StreamTranslationState` preserves cross-chunk information like usage and stop reasons
- **Bidirectional codec symmetry** — Each provider implements decode and encode, enabling any-to-any translation

The system preserves reasoning content, tool calls, usage statistics, and finish reasons across format boundaries, with terminal error handling that prevents partial translation failures.

## Frequently Asked Questions

### What is the `LlmResponseChunk` type in Switchyard?

`LlmResponseChunk` is a Rust enum defined in the translation engine that represents a streaming response in provider-neutral form. It has variants for text deltas, reasoning content, tool call fragments, usage data, stream completion, and errors. All provider codecs convert to and from this type, making it the central abstraction for cross-format streaming.

### Does Switchyard support reasoning content from Claude in OpenAI format?

Yes. The `ReasoningDelta` variant carries reasoning text from Claude's `thinking` content blocks. When encoding to OpenAI format, Switchyard emits this as `"reasoning_content": "<text>"` in the delta object—a non-standard but widely-supported extension. The neutral representation ensures reasoning content is preserved regardless of translation direction.

### How does Switchyard handle usage statistics when streaming from Anthropic to OpenAI?

Anthropic's streaming API does not include usage data in its SSE events. Switchyard accumulates token counts in `StreamTranslationState` during the decode phase. When encoding to OpenAI format, the final chunk includes a synthesized `usage` object with `prompt_tokens` and `completion_tokens` populated from state. This allows OpenAI-format consumers to receive complete usage information even from Anthropic sources.

### What happens when a stream decode error occurs mid-response?

The codec sets `state.errored = true` and emits a terminal error chunk with JSON structure `{"error":{"message":"..."}}`. All subsequent chunks from the upstream provider are dropped. This design prevents partial or malformed translations from reaching downstream consumers.