# How to Connect OpenAI Realtime Clients to the Speech-to-Speech WebSocket Server

> Connect OpenAI Realtime clients to the Speech-to-Speech WebSocket server using the RealtimeClient SDK. Stream audio and register callbacks for seamless voice interaction.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-06

---

**Use the `RealtimeClient` SDK from the huggingface/speech-to-speech repository to establish an authenticated WebSocket connection, register audio callbacks, and stream PCM audio frames to the OpenAI Realtime endpoint.**

The **huggingface/speech-to-speech** repository provides a complete OpenAI Realtime API implementation that bridges Python clients to a WebSocket server. Connecting OpenAI Realtime clients requires understanding three architectural layers: the WebSocket service layer that manages the protocol, the client SDK that abstracts low-level socket operations, and the application layer that wires everything together. This guide walks through each layer with code references from the actual source.

## WebSocket Service Architecture

The repository's Realtime stack is organized into distinct layers that handle different responsibilities.

### WebSocket Service Layer

The service layer follows the OpenAI Realtime protocol specification. It handles **JSON-based control messages**, **binary audio frames**, and **keep-alive pings** to maintain low-latency connections.

The demo server in [`demo/server.py`](https://github.com/huggingface/speech-to-speech/blob/main/demo/server.py) starts this service and exposes the `/realtime` WebSocket endpoint. This server is protocol-compatible with OpenAI's official Realtime API, meaning any compliant client can connect.

### Client SDK Layer

The `openai_realtime` package provides a thin Python wrapper that transforms raw WebSocket calls into high-level events. The `RealtimeClient` class manages handshake, authentication, and message routing.

Key events exposed by the SDK:

- `on_turn_start` — fired when the server detects the beginning of a conversational turn
- `on_turn_end` — fired when the server finishes generating its response
- `on_output_audio` — delivers synthesized speech bytes from the LLM
- `on_error` — handles protocol violations or connection failures

The implementation is validated in [`tests/openai_realtime/test_openai_client.py`](https://github.com/huggingface/speech-to-speech/blob/main/tests/openai_realtime/test_openai_client.py), which exercises the full client lifecycle.

### Application Layer

User-facing scripts glue the SDK to speech pipelines. The canonical reference is [`listen_and_play_realtime.py`](https://github.com/huggingface/speech-to-speech/blob/main/listen_and_play_realtime.py), which creates a `RealtimeClient`, streams microphone input, and plays back real-time responses.

## Step-by-Step Connection Process

### 1. Configure API Credentials

The client authenticates using the `OPENAI_API_KEY` environment variable. The constructor also accepts an explicit token parameter.

```python
import os

api_key = os.getenv("OPENAI_API_KEY")
if not api_key:
    raise RuntimeError("Set OPENAI_API_KEY environment variable")

```

This pattern appears in [`listen_and_play_realtime.py`](https://github.com/huggingface/speech-to-speech/blob/main/listen_and_play_realtime.py) at line referencing `os.getenv("OPENAI_API_KEY")`.

### 2. Initialize the RealtimeClient

Instantiate `RealtimeClient` with your API key. The endpoint URL (`wss://api.openai.com/v1/realtime`) is baked into the SDK default, though you can override it for custom deployments.

```python
from openai_realtime import RealtimeClient

client = RealtimeClient(api_key=api_key)

```

Under the hood, this creates an `aiohttp` WebSocket session and registers protocol handlers in [`openai_realtime/client.py`](https://github.com/huggingface/speech-to-speech/blob/main/openai_realtime/client.py).

### 3. Register Audio and Event Callbacks

Define coroutines that handle server events. At minimum, implement `on_output_audio` to receive synthesized speech.

```python
import numpy as np
import sounddevice as sd

async def on_output_audio(chunk: bytes):
    """Play incoming PCM audio at 16kHz."""
    audio_array = np.frombuffer(chunk, dtype=np.int16)
    sd.play(audio_array, samplerate=16000)

client.on_output_audio = on_output_audio

```

Optional callbacks include `on_turn_start`, `on_turn_end`, and `on_error` for UI synchronization and error recovery.

### 4. Connect and Perform Handshake

The `connect()` method establishes the WebSocket and waits for the `session.created` response containing the `session_id`.

```python
await client.connect()  # Blocks until handshake completes

```

This logic is exercised in [`tests/openai_realtime/test_realtime_service.py`](https://github.com/huggingface/speech-to-speech/blob/main/tests/openai_realtime/test_realtime_service.py), which validates the message exchange format.

### 5. Start a Turn and Stream Audio

Signal intent to speak with `start_turn()`, then stream PCM frames at 16 kHz. The SDK batches frames and respects the server's `max_input_frames` limit.

```python
await client.start_turn()

# Stream microphone frames (16-bit PCM, 16000 Hz)

async for frame in capture_microphone():
    await client.send_audio(frame)

await client.end_turn()  # Signal completion

```

The `send_audio()` method in [`openai_realtime/client.py`](https://github.com/huggingface/speech-to-speech/blob/main/openai_realtime/client.py) handles frame batching and protocol encoding.

### 6. Handle Server Responses

While your turn ends, the server begins streaming `output_audio` events. Your `on_output_audio` callback receives chunks as they arrive, enabling true real-time playback.

The test suite in [`tests/openai_realtime/test_smart_turn.py`](https://github.com/huggingface/speech-to-speech/blob/main/tests/openai_realtime/test_smart_turn.py) validates this turn-handling lifecycle.

## Complete Minimal Example

```python
import os
import asyncio
import numpy as np
import sounddevice as sd
from openai_realtime import RealtimeClient

async def main():
    # Authentication

    api_key = os.getenv("OPENAI_API_KEY")
    if not api_key:
        raise RuntimeError("Set OPENAI_API_KEY environment variable")

    # Initialize client

    client = RealtimeClient(api_key=api_key)

    # Define audio playback callback

    async def on_output_audio(chunk: bytes):
        audio = np.frombuffer(chunk, dtype=np.int16)
        sd.play(audio, samplerate=16000)
        sd.wait()

    client.on_output_audio = on_output_audio

    # Connect and start session

    await client.connect()
    await client.start_turn()

    # Stream 5 seconds of microphone audio

    stream = sd.RawInputStream(
        samplerate=16000,
        blocksize=1600,  # 100ms chunks

        dtype=np.int16,
        channels=1
    )
    stream.start()

    for _ in range(50):  # 5 seconds

        frame, _ = stream.read(1600)
        await client.send_audio(frame.tobytes())
        await asyncio.sleep(0.01)

    stream.stop()
    await client.end_turn()

    # Allow time for server response

    await asyncio.sleep(5)

if __name__ == "__main__":
    asyncio.run(main())

```

## Production Reference: listen_and_play_realtime.py

The repository ships a production-ready script that extends the minimal example with voice activity detection and pipeline integration:

```bash
export OPENAI_API_KEY=sk-************************************
python scripts/listen_and_play_realtime.py

```

This script in [`scripts/listen_and_play_realtime.py`](https://github.com/huggingface/speech-to-speech/blob/main/scripts/listen_and_play_realtime.py) adds:

- **Voice Activity Detection (VAD)** to auto-detect speech endpoints
- **Whisper streaming transcription** for real-time STT
- **Pipeline integration** with Parakeet, MEL-O, or other TTS backends

## Key Design Characteristics

**Async-first architecture** — The SDK uses `asyncio` and `aiohttp` for non-blocking I/O, essential for sub-100ms audio latency.

**Modular event dispatch** — Each protocol message type routes to a dedicated coroutine, enabling custom behaviors like logging or analytics injection.

**Audio pipeline abstraction** — Utilities in [`utils/audio.py`](https://github.com/huggingface/speech-to-speech/blob/main/utils/audio.py) and [`utils/thread_manager.py`](https://github.com/huggingface/speech-to-speech/blob/main/utils/thread_manager.py) decouple capture/playback from WebSocket logic for cross-platform portability.

## Error Handling and Resilience

Protocol errors trigger the `on_error` callback with structured error codes. The client optionally implements exponential backoff reconnection, validated in [`tests/openai_realtime/test_websocket_session_lifecycle.py`](https://github.com/huggingface/speech-to-speech/blob/main/tests/openai_realtime/test_websocket_session_lifecycle.py).

Common error scenarios include:

- Authentication failures (invalid or expired API key)
- Rate limiting (HTTP 429 equivalents over WebSocket)
- Protocol violations (malformed message sequences)

## File Reference Guide

| File | Purpose |
|------|---------|
| [`scripts/listen_and_play_realtime.py`](https://github.com/huggingface/speech-to-speech/blob/main/scripts/listen_and_play_realtime.py) | Production reference implementation with VAD and pipelines |
| [`tests/openai_realtime/test_openai_client.py`](https://github.com/huggingface/speech-to-speech/blob/main/tests/openai_realtime/test_openai_client.py) | Unit tests for `RealtimeClient` handshake and messaging |
| [`tests/openai_realtime/test_realtime_service.py`](https://github.com/huggingface/speech-to-speech/blob/main/tests/openai_realtime/test_realtime_service.py) | Integration tests for server-side protocol compliance |
| [`tests/openai_realtime/test_websocket_session_lifecycle.py`](https://github.com/huggingface/speech-to-speech/blob/main/tests/openai_realtime/test_websocket_session_lifecycle.py) | Connection resilience and reconnection logic |
| [`tests/openai_realtime/test_smart_turn.py`](https://github.com/huggingface/speech-to-speech/blob/main/tests/openai_realtime/test_smart_turn.py) | Turn lifecycle validation |
| [`demo/server.py`](https://github.com/huggingface/speech-to-speech/blob/main/demo/server.py) | Minimal WebSocket server for local testing |
| [`src/speech_to_speech/baseHandler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/baseHandler.py) | Core event routing to STT/TTS pipelines |

## Summary

- **Obtain an OpenAI API key** and pass it to `RealtimeClient` initialization
- **Register callbacks** for `on_output_audio` and optional turn management events
- **Call `connect()`** to perform the WebSocket handshake and receive `session_id`
- **Use `start_turn()` / `end_turn()`** to bracket audio streaming, sending PCM frames via `send_audio()`
- **Reference [`listen_and_play_realtime.py`](https://github.com/huggingface/speech-to-speech/blob/main/listen_and_play_realtime.py)** for production patterns including VAD and pipeline integration

## Frequently Asked Questions

### What audio format does the RealtimeClient expect?

The client expects **16-bit PCM audio at 16000 Hz**, delivered as `bytes` objects. The SDK handles internal framing and protocol encoding. Downsample or convert your source audio using [`utils/audio.py`](https://github.com/huggingface/speech-to-speech/blob/main/utils/audio.py) utilities if needed.

### Can I connect to a custom WebSocket server instead of OpenAI's endpoint?

Yes. Pass the `base_url` parameter to `RealtimeClient` to override the default `wss://api.openai.com/v1/realtime`. The server must implement the same JSON/binary message protocol verified in [`tests/openai_realtime/test_realtime_service.py`](https://github.com/huggingface/speech-to-speech/blob/main/tests/openai_realtime/test_realtime_service.py).

### How does the SDK handle connection drops?

The `RealtimeClient` includes optional exponential backoff reconnection, exercised in [`test_websocket_session_lifecycle.py`](https://github.com/huggingface/speech-to-speech/blob/main/test_websocket_session_lifecycle.py). Set `auto_reconnect=True` in the constructor and provide an `on_error` callback to monitor recovery status.

### Is WebRTC supported for browser clients?

The repository includes WebRTC forwarding in [`tests/openai_realtime/test_webrtc.py`](https://github.com/huggingface/speech-to-speech/blob/main/tests/openai_realtime/test_webrtc.py). This wraps the Realtime WebSocket with a media server bridge, enabling browser-based clients to use WebRTC audio tracks while the backend speaks the native Realtime protocol.