Setting Up WebRTC Sessions for Browser-Based Voice Clients

The Hugging Face speech-to-speech library provides an optional WebRTC transport built on aiortc that enables peer-to-peer voice streaming between browsers and the server at 48 kHz, automatically resampling to the pipeline's internal 16 kHz rate while using a dedicated data channel for JSON events.

The speech-to-speech repository implements a production-ready WebRTC transport that emulates the OpenAI Realtime API over peer-to-peer connections. Setting up WebRTC sessions for browser-based voice clients allows you to stream audio with lower latency than WebSocket alternatives while maintaining full compatibility with the existing pipeline infrastructure.

Architecture Overview

The WebRTC implementation consists of four coordinated layers that handle media, signaling, and lifecycle management.

Transport Layer

The WebRTCSession class in src/speech_to_speech/api/openai_realtime/webrtc_session.py subclasses SessionTransport and implements the same interface as the default WebSocket transport. This design allows the pipeline to switch transports without modifying downstream components.

Media Handling

Audio flows through two custom media tracks that manage codec conversion and resampling:

• Outbound audio is delivered by PipelineAudioTrack, a custom MediaStreamTrack that buffers PCM data, resamples it to 48 kHz, and emits paced RTP frames every 20 ms.
• Inbound audio arrives on an audio track from the browser, is resampled from 48 kHz to the pipeline rate (16 kHz) by PcmResampler, and fed back into the pipeline via the on_audio callback.

All audio is transmitted as Opus-encoded RTP at 48 kHz and automatically resampled to/from the internal 16 kHz pipeline rate using av.AudioResampler (source lines 79‑96).

Data Channel

A dedicated RTC data channel named oai-events carries all JSON-encoded client events, mirroring the WebSocket event flow. Messages are queued in _dc_messages to preserve order before being dispatched to the on_client_event handler.

Lifecycle Management

The session implements robust connection state tracking:

• setup() wires aiortc callbacks for track, datachannel, and connection-state changes.
• negotiate() processes the client’s SDP offer, creates an answer, and waits for ICE gathering with a configurable timeout.
• _connect_watchdog ensures sessions that never reach the "connected" state are cleaned up after a timeout.
• close() cancels background tasks, stops the audio track, and closes the peer connection (source lines 69‑86).

Prerequisites and Installation

The WebRTC transport is optional and guarded by the extra webrtc, which pulls in aiortc. Install the package with:

pip install "speech-to-speech[webrtc]"

If the extra is not present, the module gracefully degrades; attempts to use WebRTC endpoints return a clear error message at [websocket_router.py line 590](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/websocket_router.py#L590).

Implementing WebRTC Sessions

Initializing the Server-Side Session

Create a WebRTCSession instance by passing an RTCPeerConnection and callback handlers:

from speech_to_speech.api.openai_realtime.webrtc_session import WebRTCSession
from aiortc import RTCPeerConnection

async def start_webrtc(offer_sdp: str):
    pc = RTCPeerConnection()
    sess = WebRTCSession(
        pc,
        on_client_event=handle_event,
        on_audio=handle_audio,
        on_open=on_open,
        on_closed=on_closed,
    )
    sess.setup()
    answer_sdp = await sess.negotiate(offer_sdp)
    return answer_sdp, sess

handle_event, handle_audio, on_open, and on_closed are user-provided callbacks that integrate with your pipeline logic.

Streaming Audio to the Browser

When your pipeline generates PCM chunks at 16 kHz, send them to the client:

# Inside the pipeline when you have a PCM chunk (16 kHz, mono)

await webrtc_session.send_audio_chunk(service, session_id, pcm_chunk)

The send_audio_chunk method resamples the PCM to 48 kHz, writes it to the outbound track, and optionally sends pending events over the data channel (source lines 300‑307).

Processing Incoming Browser Audio

Handle incoming audio through the callback specified during initialization:

def handle_audio(pcm: bytes):
    # `pcm` is 48 kHz Opus-decoded data already resampled to 16 kHz

    pipeline.feed_audio(pcm)

The inbound audio path is managed by _consume_inbound_audio, which continuously reads frames from the incoming track, resamples them, and forwards the PCM to the on_audio callback (source lines 42‑52).

Terminating the Connection

Cleanly close the session to release resources:

await webrtc_session.close()

This cancels all background tasks, stops the audio track, and safely tears down the peer connection.

Key Implementation Files

• src/speech_to_speech/api/openai_realtime/webrtc_session.py – Core WebRTC transport implementation including peer connection management, media tracks, and data channel handling.
• src/speech_to_speech/api/openai_realtime/websocket_router.py – Router that selects between WebSocket and WebRTC transports; contains fallback logic and HTTP endpoints.
• tests/openai_realtime/test_webrtc.py – Test suite validating session negotiation, audio flow, and error handling for the WebRTC transport.

Summary

• The WebRTCSession class in webrtc_session.py provides a drop-in replacement for WebSocket transports, implementing the SessionTransport interface.
• Audio streams at 48 kHz over Opus-encoded RTP and is automatically resampled to 16 kHz for pipeline compatibility using av.AudioResampler.
• The oai-events data channel handles JSON event messaging separately from the audio stream.
• Install the transport via pip install "speech-to-speech[webrtc]" to enable the aiortc dependency.
• Lifecycle methods setup(), negotiate(), and close() manage the peer connection state, ICE gathering, and resource cleanup.

Frequently Asked Questions

What audio codecs and sample rates does the WebRTC transport support?

The transport transmits audio as Opus-encoded RTP at 48 kHz, which is the WebRTC standard. The internal pipeline operates at 16 kHz, so the implementation uses av.AudioResampler to convert between these rates automatically. Outbound audio is resampled from 16 kHz to 48 kHz before transmission, while inbound audio is downsampled from 48 kHz to 16 kHz after decoding.

How does the WebRTC implementation differ from the WebSocket transport?

While both implement the SessionTransport interface and support the same OpenAI Realtime API events, WebRTC establishes a peer-to-peer connection using aiortc rather than a persistent HTTP socket. This reduces latency by eliminating server hops for media relay and separates audio streams (via RTP) from control events (via the oai-events data channel), whereas WebSocket multiplexes everything through a single connection.

What happens if the aiortc dependency is missing?

If you attempt to use WebRTC endpoints without installing the [webrtc] extra, the system gracefully degrades. The router in websocket_router.py detects the missing dependency and returns a clear error message to the client (source line 590), preventing runtime exceptions while indicating that the transport is unavailable.

How does the session handle connection failures or timeouts?

The _connect_watchdog background task monitors the connection state during initialization. If the peer connection fails to reach the "connected" state within the configured timeout, the watchdog triggers cleanup. Additionally, the negotiate() method includes ICE gathering timeouts, and the close() method ensures all tasks are cancelled and the peer connection is destroyed even if the session terminates unexpectedly.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}