# How WebSocket and WebRTC Transports Handle Audio Streaming in the Speech-to-Speech Library

> Discover how WebSocket and WebRTC handle audio streaming in the Speech-to-Speech library. Learn about their distinct transport mechanisms for real-time voice data.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: internals
- Published: 2026-08-06

---

**The Speech-to-Speech library uses a pluggable transport architecture where WebSocket sends base64-encoded PCM16 audio inside JSON events, while WebRTC streams raw PCM over an RTP media track and routes control events through a separate data channel.**

Both transports implement the same `SessionTransport` interface in [`src/speech_to_speech/api/openai_realtime/transports.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/transports.py), enabling seamless switching between connection types without changing the core pipeline logic. This design supports real-time speech-to-speech applications where low latency and adaptive buffering are critical requirements.

## SessionTransport: The Common Interface for Audio Streaming

The `SessionTransport` abstract base class defines three required methods that all transports must implement. This abstraction allows the [`websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/websocket_router.py) send loop to remain transport-agnostic while handling audio streaming and event delivery.

The interface requires:

- `send_events(events: list[ServerEvent])` — forwards session events to the client
- `send_audio_chunk(service, session_id, pcm)` — transmits PCM audio data
- `discard_pending_audio()` — handles buffer cleanup for barge-in scenarios

When a session initializes, the router assigns either a `WebSocketTransport` or `WebRTCSession` instance to `session_state.transport` based on the client connection type.

## WebSocketTransport: JSON-Encapsulated Audio Streaming

The `WebSocketTransport` class handles audio streaming by encoding raw PCM data into base64 deltas wrapped in JSON events. This approach works with standard WebSocket clients without requiring additional media handling.

### Audio Encoding and Transmission

In [`service.py`](https://github.com/huggingface/speech-to-speech/blob/main/service.py), the `RealtimeService.encode_audio_chunk` method converts PCM16 buffers into base64-encoded deltas. The transport wraps these in standard event objects and transmits them via `await ws.send_json(event.model_dump())`.

```python

# Simplified send loop showing WebSocket transport usage

transport = session_state.transport  # WebSocketTransport instance

await transport.send_audio_chunk(service, session_id, pcm_chunk)
await transport.send_events([event1, event2])

```

The `send_ws_event` helper manages graceful shutdown handling, ensuring events complete transmission even during connection teardown.

### Buffer Management Limitations

WebSocket transport has **no server-side buffer control**. The `discard_pending_audio()` method is a no-op because audio buffering occurs entirely on the client side. For barge-in scenarios where the assistant must stop speaking immediately, the client application must handle audio truncation locally.

## WebRTCTransport: Direct RTP Media Streaming

The `WebRTCSession` class in [`webrtc_session.py`](https://github.com/huggingface/speech-to-speech/blob/main/webrtc_session.py) provides native WebRTC support with separate channels for media and control data. This architecture reduces latency and enables server-side buffer management critical for responsive conversational AI.

### RTP Audio Track and Data Channel Architecture

Audio streaming uses a dedicated **RTP media track** on the peer connection. The server pushes raw PCM16 frames directly to this track, which the browser plays back immediately. This bypasses JSON encoding overhead and reduces latency compared to WebSocket transport.

Control events travel on a separate **data channel named `oai-events`**. This separation prevents head-of-line blocking and keeps session management responsive even during high-volume audio streaming.

```python
from speech_to_speech.api.openai_realtime.webrtc_session import WebRTCSession

# Establish WebRTC peer connection on client side first

webrtc = WebRTCSession(pc)  # pc = RTCPeerConnection

session_state.transport = webrtc

# Stream raw PCM directly over RTP track

await webrtc.send_audio_chunk(service, session_id, pcm_chunk)

# Send control event via data channel

await webrtc.send_events([event])

```

### Server-Side Buffer Flushing for Barge-In

Unlike WebSocket, WebRTC transport can **flush pending audio server-side** using `discard_pending_audio()`. This method clears the RTP track buffer when the user interrupts the assistant, enabling immediate response without waiting for buffered speech to complete.

```python

# Force flush RTP track buffer during barge-in

session_state.transport.discard_pending_audio()

```

This capability is essential for natural conversational flow where users expect immediate response to interruptions.

## Transport Selection in the Router Pipeline

The [`websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/websocket_router.py) file contains the transport-agnostic send loop that mediates between the speech processing pipeline and the active transport. The router examines `SessionState` to determine which transport class to instantiate and invoke.

Key design benefits of this architecture:

- **Unified pipeline logic** — the same code path handles both WebSocket and WebRTC connections
- **Runtime transport switching** — session state determines transport without pipeline modifications
- **Extensible design** — new transports implement three methods and integrate automatically

## Summary

- The `SessionTransport` abstract base class in [`transports.py`](https://github.com/huggingface/speech-to-speech/blob/main/transports.py) defines the contract for all audio streaming implementations
- **WebSocketTransport** encodes audio as base64 PCM16 deltas inside JSON events, with client-side buffering only
- **WebRTCSession** streams raw PCM over an RTP media track and routes events through the `oai-events` data channel
- WebRTC enables **server-side buffer flushing** via `discard_pending_audio()`, critical for barge-in handling
- The transport-agnostic [`websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/websocket_router.py) send loop selects implementations based on `SessionState` configuration

## Frequently Asked Questions

### Does WebRTC provide lower latency than WebSocket for audio streaming?

Yes. WebRTC eliminates base64 encoding overhead and sends raw PCM directly over RTP. The separate `oai-events` data channel prevents control messages from queuing behind audio data. For real-time speech-to-speech applications, this architecture typically achieves sub-100ms end-to-end latency versus 150-300ms for WebSocket transport.

### What happens if a user interrupts the assistant while audio is streaming?

The behavior depends on the active transport. With WebRTC, `discard_pending_audio()` flushes the RTP track buffer immediately, stopping playback within milliseconds. With WebSocket, the server has no mechanism to recall in-flight audio; the client must implement its own truncation logic or wait for queued audio events to complete.

### Can I switch between WebSocket and WebRTC without changing application code?

Yes, if your application uses the transport abstraction correctly. The [`websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/websocket_router.py) pipeline calls transport methods generically. However, your client implementation must support both protocols, and any transport-specific features like `discard_pending_audio()` should include fallback handling for WebSocket sessions.

### Where is the base64 audio encoding implemented?

The `RealtimeService.encode_audio_chunk` method in [`service.py`](https://github.com/huggingface/speech-to-speech/blob/main/service.py) handles PCM16-to-base64 conversion for WebSocket transport. WebRTC transport bypasses this entirely and writes raw PCM frames directly to the RTP track in [`webrtc_session.py`](https://github.com/huggingface/speech-to-speech/blob/main/webrtc_session.py).