How WebSocket and WebRTC Transports Handle Audio Streaming in the Speech-to-Speech Library

The Speech-to-Speech library uses a pluggable transport architecture where WebSocket sends base64-encoded PCM16 audio inside JSON events, while WebRTC streams raw PCM over an RTP media track and routes control events through a separate data channel.

Both transports implement the same SessionTransport interface in src/speech_to_speech/api/openai_realtime/transports.py, enabling seamless switching between connection types without changing the core pipeline logic. This design supports real-time speech-to-speech applications where low latency and adaptive buffering are critical requirements.

SessionTransport: The Common Interface for Audio Streaming

The SessionTransport abstract base class defines three required methods that all transports must implement. This abstraction allows the websocket_router.py send loop to remain transport-agnostic while handling audio streaming and event delivery.

The interface requires:

• send_events(events: list[ServerEvent]) — forwards session events to the client
• send_audio_chunk(service, session_id, pcm) — transmits PCM audio data
• discard_pending_audio() — handles buffer cleanup for barge-in scenarios

When a session initializes, the router assigns either a WebSocketTransport or WebRTCSession instance to session_state.transport based on the client connection type.

WebSocketTransport: JSON-Encapsulated Audio Streaming

The WebSocketTransport class handles audio streaming by encoding raw PCM data into base64 deltas wrapped in JSON events. This approach works with standard WebSocket clients without requiring additional media handling.

Audio Encoding and Transmission

In service.py, the RealtimeService.encode_audio_chunk method converts PCM16 buffers into base64-encoded deltas. The transport wraps these in standard event objects and transmits them via await ws.send_json(event.model_dump()).

# Simplified send loop showing WebSocket transport usage

transport = session_state.transport  # WebSocketTransport instance

await transport.send_audio_chunk(service, session_id, pcm_chunk)
await transport.send_events([event1, event2])

The send_ws_event helper manages graceful shutdown handling, ensuring events complete transmission even during connection teardown.

Buffer Management Limitations

WebSocket transport has no server-side buffer control. The discard_pending_audio() method is a no-op because audio buffering occurs entirely on the client side. For barge-in scenarios where the assistant must stop speaking immediately, the client application must handle audio truncation locally.

WebRTCTransport: Direct RTP Media Streaming

The WebRTCSession class in webrtc_session.py provides native WebRTC support with separate channels for media and control data. This architecture reduces latency and enables server-side buffer management critical for responsive conversational AI.

RTP Audio Track and Data Channel Architecture

Audio streaming uses a dedicated RTP media track on the peer connection. The server pushes raw PCM16 frames directly to this track, which the browser plays back immediately. This bypasses JSON encoding overhead and reduces latency compared to WebSocket transport.

Control events travel on a separate data channel named oai-events. This separation prevents head-of-line blocking and keeps session management responsive even during high-volume audio streaming.

from speech_to_speech.api.openai_realtime.webrtc_session import WebRTCSession

# Establish WebRTC peer connection on client side first

webrtc = WebRTCSession(pc)  # pc = RTCPeerConnection

session_state.transport = webrtc

# Stream raw PCM directly over RTP track

await webrtc.send_audio_chunk(service, session_id, pcm_chunk)

# Send control event via data channel

await webrtc.send_events([event])

Server-Side Buffer Flushing for Barge-In

Unlike WebSocket, WebRTC transport can flush pending audio server-side using discard_pending_audio(). This method clears the RTP track buffer when the user interrupts the assistant, enabling immediate response without waiting for buffered speech to complete.

# Force flush RTP track buffer during barge-in

session_state.transport.discard_pending_audio()

This capability is essential for natural conversational flow where users expect immediate response to interruptions.

Transport Selection in the Router Pipeline

The websocket_router.py file contains the transport-agnostic send loop that mediates between the speech processing pipeline and the active transport. The router examines SessionState to determine which transport class to instantiate and invoke.

Key design benefits of this architecture:

• Unified pipeline logic — the same code path handles both WebSocket and WebRTC connections
• Runtime transport switching — session state determines transport without pipeline modifications
• Extensible design — new transports implement three methods and integrate automatically

Summary

• The SessionTransport abstract base class in transports.py defines the contract for all audio streaming implementations
• WebSocketTransport encodes audio as base64 PCM16 deltas inside JSON events, with client-side buffering only
• WebRTCSession streams raw PCM over an RTP media track and routes events through the oai-events data channel
• WebRTC enables server-side buffer flushing via discard_pending_audio(), critical for barge-in handling
• The transport-agnostic websocket_router.py send loop selects implementations based on SessionState configuration

Frequently Asked Questions

Does WebRTC provide lower latency than WebSocket for audio streaming?

Yes. WebRTC eliminates base64 encoding overhead and sends raw PCM directly over RTP. The separate oai-events data channel prevents control messages from queuing behind audio data. For real-time speech-to-speech applications, this architecture typically achieves sub-100ms end-to-end latency versus 150-300ms for WebSocket transport.

What happens if a user interrupts the assistant while audio is streaming?

The behavior depends on the active transport. With WebRTC, discard_pending_audio() flushes the RTP track buffer immediately, stopping playback within milliseconds. With WebSocket, the server has no mechanism to recall in-flight audio; the client must implement its own truncation logic or wait for queued audio events to complete.

Can I switch between WebSocket and WebRTC without changing application code?

Yes, if your application uses the transport abstraction correctly. The websocket_router.py pipeline calls transport methods generically. However, your client implementation must support both protocols, and any transport-specific features like discard_pending_audio() should include fallback handling for WebSocket sessions.

Where is the base64 audio encoding implemented?

The RealtimeService.encode_audio_chunk method in service.py handles PCM16-to-base64 conversion for WebSocket transport. WebRTC transport bypasses this entirely and writes raw PCM frames directly to the RTP track in webrtc_session.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}