WebRTC Transport Setup with STUN/TURN Servers for Production Deployment in Speech-to-Speech

Configure STUN/TURN servers in the huggingface/speech-to-speech repository by setting the SPEECH_TO_SPEECH_ICE_SERVERS environment variable with a JSON list of RTCIceServer objects, then instantiate WebRTCSession with rtc_configuration_from_env() to enable production-grade peer-to-peer connectivity.

The huggingface/speech-to-speech repository provides a self-hosted implementation of the OpenAI Realtime API supporting both WebSocket and WebRTC transports. For WebRTC transport setup with STUN/TURN servers for production deployment, the codebase uses aiortc to handle peer connections, RTP media streaming, and ICE negotiation. This guide walks through the configuration, negotiation flow, and operational considerations for running at scale.

---

Core WebRTC Architecture

The transport layer centers on four key components defined in src/speech_to_speech/api/openai_realtime/webrtc_session.py:

Component	Role	Source Location
WebRTCSession	Implements SessionTransport interface; manages RTCPeerConnection, audio pipelines, and lifecycle	Lines 1–250+
PipelineAudioTrack	MediaStreamTrack that emits paced 20 ms, 48 kHz audio frames from server-side PCM buffers	Lines 99–152
PcmResampler	Stateful wrapper around av.AudioResampler converting between 16 kHz pipeline and 48 kHz WebRTC formats	Lines 70–98
"oai-events" DataChannel	Carries all OpenAI Realtime JSON messages (response.created, session.update, etc.)	Lines 94–98

---

Configuring ICE Servers for Production

Environment Variable Configuration

The rtc_configuration_from_env() helper (lines 50–67) reads ICE server configuration exclusively from the environment:

export SPEECH_TO_SPEECH_ICE_SERVERS='[
    {"urls": "stun:stun.l.google.com:19302"},
    {"urls": "turn:turn.example.com:3478",
     "username": "myuser",
     "credential": "mysecret"}
]'

Critical requirements:

• Must be valid JSON array of RTCIceServer-compatible dictionaries
• Supports multiple STUN and TURN entries
• Authentication via username/credential for TURN relay
• If absent or malformed, aiortc falls back to defaults (tested in tests/openai_realtime/test_webrtc.py)

Runtime Instantiation

The router layer normally handles session creation. For custom deployments:

from aiortc import RTCPeerConnection
from speech_to_speech.api.openai_realtime.webrtc_session import (
    WebRTCSession,
    rtc_configuration_from_env,
)

# Build configuration from environment

rtc_config = rtc_configuration_from_env()  # Returns RTCConfiguration or None

# Create peer connection with ICE servers

pc = RTCPeerConnection(rtc_config)

# Initialize session with callbacks

session = WebRTCSession(
    pc,
    on_client_event=handle_client_event,   # JSON messages from data-channel

    on_audio=handle_inbound_audio,         # PCM buffers from client mic

    on_open=handle_open,                   # Connection established

    on_closed=handle_closed,               # Cleanup and metrics

)
session.setup()  # Attaches track and data-channel handlers

---

SDP Negotiation Flow

After session setup, complete the handshake with session.negotiate():

# Client (browser) generates offer; send to server

answer_sdp = await session.negotiate(client_offer_sdp)

# Return answer_sdp to client

The negotiation method performs four operations sequentially (lines 180–220):

1. Set remote description from client's SDP offer
2. Create answer with local media capabilities
3. Gather ICE candidates with ICE_GATHERING_TIMEOUT_S (5 s default)
4. Return SDP answer containing STUN/TURN candidates for client connectivity

---

Production Reliability Mechanisms

Connection Timeouts

Timeout	Default	Purpose
ICE_GATHERING_TIMEOUT_S	5 s	Maximum wait for candidate collection; returns partial SDP if exceeded
CONNECT_TIMEOUT_S	30 s	_connect_watchdog task monitors RTCPeerConnection.state; auto-closes if never reaches "connected"

The watchdog prevents pipeline units from being held indefinitely when NAT traversal fails or clients disconnect mid-negotiation.

Network Deployment Checklist

• TLS termination: Run behind Nginx or similar proxy forwarding /v1/realtime/webrtc/calls
• UDP port access: Open firewall ranges for TURN relay traffic (typically 3478, plus ephemeral ports)
• Horizontally scaled instances: Each server instance reads identical SPEECH_TO_SPEECH_ICE_SERVERS; TURN relay coordinates media routing across the pool

---

Browser Client Integration

Match the server-side data-channel label exactly:

const pc = new RTCPeerConnection({
  iceServers: [{ urls: "stun:stun.l.google.com:19302" }]
});

// Must use label "oai-events" — hardcoded in webrtc_session.py line 94
const dc = pc.createDataChannel("oai-events");
dc.onmessage = e => console.log(JSON.parse(e.data));

// Add audio transceiver for bidirectional streaming
pc.addTransceiver('audio', { direction: 'sendrecv' });

---

Key Source Files Reference

File	Purpose
src/speech_to_speech/api/openai_realtime/webrtc_session.py	Core transport: ICE handling, audio resampling, watchdog logic
src/speech_to_speech/api/openai_realtime/transports.py	Abstract SessionTransport interface
src/speech_to_speech/api/openai_realtime/websocket_router.py	HTTP routing and transport dispatch
src/speech_to_speech/api/openai_realtime/server.py	FastAPI service entry point
demo/server.py	Local development; contains _webrtc_calls_url() helper
tests/openai_realtime/test_webrtc.py	Test coverage including invalid ICE config handling

---

Summary

• ICE servers are configured exclusively via the SPEECH_TO_SPEECH_ICE_SERVERS environment variable as JSON-serialized RTCIceServer arrays
• WebRTCSession wraps aiortc's RTCPeerConnection with production safeguards: ICE timeouts, connection watchdogs, and automatic cleanup
• Bidirectional audio flows over RTP tracks at 48 kHz while JSON events travel on the "oai-events" data-channel
• Scaling requires consistent ICE configuration across all instances and UDP port accessibility for TURN relay

---

Frequently Asked Questions

What happens if SPEECH_TO_SPEECH_ICE_SERVERS is malformed or missing?

The rtc_configuration_from_env() helper returns None, causing RTCPeerConnection to use aiortc's default STUN configuration. The test in tests/openai_realtime/test_webrtc.py explicitly verifies this fallback behavior for invalid JSON inputs.

Why does the audio resample from 16 kHz to 48 kHz?

The server-side speech pipeline operates at 16 kHz (optimized for ASR/TTS models), while WebRTC mandates 48 kHz for Opus codec compatibility. The PcmResampler class maintains stateful conversion using FFmpeg via av.AudioResampler to prevent frame boundary artifacts.

How does the watchdog prevent resource leaks?

The _connect_watchdog coroutine (spawned in setup()) monitors pc.connectionState and triggers session.close() if "connected" is not reached within CONNECT_TIMEOUT_S (30 seconds). This automatically releases the pipeline unit back to the worker pool.

Can I use a TURN server with TLS transport?

Yes. Include turns: (TLS) or turn: (UDP/TCP) URLs in the SPEECH_TO_SPEECH_ICE_SERVERS JSON. The aiortc library underlying WebRTCSession supports all standard RTCIceServer configurations including TLS-encrypted TURN relays on port 5349.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}