# Setting Up WebRTC Sessions for Browser-Based Voice Clients

> Learn how to set up WebRTC sessions for browser-based voice clients using the Hugging Face speech-to-speech library. Enable seamless peer-to-peer voice streaming with aiortc.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-09

---

**The Hugging Face `speech-to-speech` library provides an optional WebRTC transport built on aiortc that enables peer-to-peer voice streaming between browsers and the server at 48 kHz, automatically resampling to the pipeline's internal 16 kHz rate while using a dedicated data channel for JSON events.**

The `speech-to-speech` repository implements a production-ready WebRTC transport that emulates the OpenAI Realtime API over peer-to-peer connections. Setting up WebRTC sessions for browser-based voice clients allows you to stream audio with lower latency than WebSocket alternatives while maintaining full compatibility with the existing pipeline infrastructure.

## Architecture Overview

The WebRTC implementation consists of four coordinated layers that handle media, signaling, and lifecycle management.

### Transport Layer

The **`WebRTCSession`** class in [`src/speech_to_speech/api/openai_realtime/webrtc_session.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/webrtc_session.py) subclasses `SessionTransport` and implements the same interface as the default WebSocket transport. This design allows the pipeline to switch transports without modifying downstream components.

### Media Handling

Audio flows through two custom media tracks that manage codec conversion and resampling:

- **Outbound audio** is delivered by **`PipelineAudioTrack`**, a custom `MediaStreamTrack` that buffers PCM data, resamples it to **48 kHz**, and emits paced RTP frames every **20 ms**.
- **Inbound audio** arrives on an `audio` track from the browser, is resampled from 48 kHz to the pipeline rate (16 kHz) by **`PcmResampler`**, and fed back into the pipeline via the `on_audio` callback.

All audio is transmitted as **Opus-encoded RTP** at 48 kHz and automatically resampled to/from the internal 16 kHz pipeline rate using `av.AudioResampler` ([source lines 79‑96](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/webrtc_session.py#L79-L96)).

### Data Channel

A dedicated RTC data channel named **`oai-events`** carries all JSON-encoded client events, mirroring the WebSocket event flow. Messages are queued in **`_dc_messages`** to preserve order before being dispatched to the `on_client_event` handler.

### Lifecycle Management

The session implements robust connection state tracking:

- **`setup()`** wires aiortc callbacks for track, datachannel, and connection-state changes.
- **`negotiate()`** processes the client’s SDP offer, creates an answer, and waits for ICE gathering with a configurable timeout.
- **`_connect_watchdog`** ensures sessions that never reach the "connected" state are cleaned up after a timeout.
- **`close()`** cancels background tasks, stops the audio track, and closes the peer connection ([source lines 69‑86](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/webrtc_session.py#L69-L86)).

## Prerequisites and Installation

The WebRTC transport is optional and guarded by the extra `webrtc`, which pulls in **aiortc**. Install the package with:

```bash
pip install "speech-to-speech[webrtc]"

```

If the extra is not present, the module gracefully degrades; attempts to use WebRTC endpoints return a clear error message at [[`websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/websocket_router.py) line 590](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/websocket_router.py#L590).

## Implementing WebRTC Sessions

### Initializing the Server-Side Session

Create a `WebRTCSession` instance by passing an `RTCPeerConnection` and callback handlers:

```python
from speech_to_speech.api.openai_realtime.webrtc_session import WebRTCSession
from aiortc import RTCPeerConnection

async def start_webrtc(offer_sdp: str):
    pc = RTCPeerConnection()
    sess = WebRTCSession(
        pc,
        on_client_event=handle_event,
        on_audio=handle_audio,
        on_open=on_open,
        on_closed=on_closed,
    )
    sess.setup()
    answer_sdp = await sess.negotiate(offer_sdp)
    return answer_sdp, sess

```

*`handle_event`*, *`handle_audio`*, *`on_open`*, and *`on_closed`* are user-provided callbacks that integrate with your pipeline logic.

### Streaming Audio to the Browser

When your pipeline generates PCM chunks at 16 kHz, send them to the client:

```python

# Inside the pipeline when you have a PCM chunk (16 kHz, mono)

await webrtc_session.send_audio_chunk(service, session_id, pcm_chunk)

```

The **`send_audio_chunk`** method resamples the PCM to 48 kHz, writes it to the outbound track, and optionally sends pending events over the data channel ([source lines 300‑307](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/webrtc_session.py#L300-L307)).

### Processing Incoming Browser Audio

Handle incoming audio through the callback specified during initialization:

```python
def handle_audio(pcm: bytes):
    # `pcm` is 48 kHz Opus-decoded data already resampled to 16 kHz

    pipeline.feed_audio(pcm)

```

The inbound audio path is managed by **`_consume_inbound_audio`**, which continuously reads frames from the incoming track, resamples them, and forwards the PCM to the `on_audio` callback ([source lines 42‑52](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/webrtc_session.py#L42-L52)).

### Terminating the Connection

Cleanly close the session to release resources:

```python
await webrtc_session.close()

```

This cancels all background tasks, stops the audio track, and safely tears down the peer connection.

## Key Implementation Files

- **[`src/speech_to_speech/api/openai_realtime/webrtc_session.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/webrtc_session.py)** – Core WebRTC transport implementation including peer connection management, media tracks, and data channel handling.
- **[`src/speech_to_speech/api/openai_realtime/websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/websocket_router.py)** – Router that selects between WebSocket and WebRTC transports; contains fallback logic and HTTP endpoints.
- **[`tests/openai_realtime/test_webrtc.py`](https://github.com/huggingface/speech-to-speech/blob/main/tests/openai_realtime/test_webrtc.py)** – Test suite validating session negotiation, audio flow, and error handling for the WebRTC transport.

## Summary

- The **`WebRTCSession`** class in [`webrtc_session.py`](https://github.com/huggingface/speech-to-speech/blob/main/webrtc_session.py) provides a drop-in replacement for WebSocket transports, implementing the `SessionTransport` interface.
- Audio streams at **48 kHz** over Opus-encoded RTP and is automatically resampled to **16 kHz** for pipeline compatibility using `av.AudioResampler`.
- The **`oai-events`** data channel handles JSON event messaging separately from the audio stream.
- Install the transport via `pip install "speech-to-speech[webrtc]"` to enable the aiortc dependency.
- Lifecycle methods **`setup()`**, **`negotiate()`**, and **`close()`** manage the peer connection state, ICE gathering, and resource cleanup.

## Frequently Asked Questions

### What audio codecs and sample rates does the WebRTC transport support?

The transport transmits audio as **Opus-encoded RTP at 48 kHz**, which is the WebRTC standard. The internal pipeline operates at 16 kHz, so the implementation uses `av.AudioResampler` to convert between these rates automatically. Outbound audio is resampled from 16 kHz to 48 kHz before transmission, while inbound audio is downsampled from 48 kHz to 16 kHz after decoding.

### How does the WebRTC implementation differ from the WebSocket transport?

While both implement the `SessionTransport` interface and support the same OpenAI Realtime API events, WebRTC establishes a **peer-to-peer connection** using aiortc rather than a persistent HTTP socket. This reduces latency by eliminating server hops for media relay and separates audio streams (via RTP) from control events (via the `oai-events` data channel), whereas WebSocket multiplexes everything through a single connection.

### What happens if the aiortc dependency is missing?

If you attempt to use WebRTC endpoints without installing the `[webrtc]` extra, the system gracefully degrades. The router in [`websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/websocket_router.py) detects the missing dependency and returns a clear error message to the client ([source line 590](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/websocket_router.py#L590)), preventing runtime exceptions while indicating that the transport is unavailable.

### How does the session handle connection failures or timeouts?

The **`_connect_watchdog`** background task monitors the connection state during initialization. If the peer connection fails to reach the "connected" state within the configured timeout, the watchdog triggers cleanup. Additionally, the **`negotiate()`** method includes ICE gathering timeouts, and the **`close()`** method ensures all tasks are cancelled and the peer connection is destroyed even if the session terminates unexpectedly.