# How WebSocket Protocol Enables Real-Time Communication in the Hugging Face Speech-to-Speech Pipeline

> Discover how WebSocket protocol powers real-time communication in the Hugging Face speech-to-speech pipeline, enabling low-latency audio streaming and instant text output.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: internals
- Published: 2026-08-05

---

**WebSockets provide bidirectional, low-latency streaming that allows continuous audio input from clients and simultaneous audio/text output from the server without repeated HTTP handshakes.**

The **huggingface/speech-to-speech** repository implements a complete real-time speech-to-speech pipeline using WebSockets as its core transport mechanism. This architecture enables live conversation where users speak naturally and receive synthesized responses with minimal delay. The WebSocket protocol enables real-time communication through three tightly integrated components that handle connection management, data serialization, and pipeline orchestration.

## WebSocket Architecture Components

### WebSocketStreamer: The Core Server

The **`WebSocketStreamer`** class in [`src/speech_to_speech/connections/websocket_streamer.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/connections/websocket_streamer.py) acts as the bridge between raw WebSocket connections and the processing pipeline. It accepts a single client, manages the connection lifecycle, and uses asyncio queues to feed audio data into the pipeline.

Key responsibilities include:

- Accepting incoming WebSocket connections on a configurable host/port
- Forwarding audio chunks from the client to the pipeline's `audio_input_queue`
- Reading from `audio_output_queue` and `text_output_queue` to push responses back to the client
- Handling connection cleanup when the client disconnects

The streamer runs an async event loop with two primary coroutines: `_recv_audio` (line 94-129) for incoming data and `_send_audio`/`_send_text` (lines 129-165) for outgoing streams.

### WebSocketTransport: Data Serialization Layer

The **`WebSocketTransport`** in [`src/speech_to_speech/api/openai_realtime/transports.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/transports.py) abstracts WebSocket details behind a uniform "session transport" API. This layer implements the OpenAI Realtime specification for compatibility with third-party clients.

Critical implementation details:

- Audio deltas are **base-64 encoded** JSON frames: `{"event": "audio_delta", "audio": "<base64>"}`
- Text events use plain JSON: `{"event": "text", "text": "..."}`
- The `send_ws_event` method (lines 53-61) handles frame decoding and queue dispatch

This abstraction lets the rest of the pipeline operate on raw bytes and strings without knowing the transport mechanism.

### WebSocketRouter: FastAPI Integration

The **`WebSocketRouter`** in [`src/speech_to_speech/api/openai_realtime/websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/websocket_router.py) wires HTTP upgrade requests into the WebSocket pipeline. When a client connects to `/v1/realtime`, the router:

1. Creates a `WebSocketTransport` instance
2. Registers a fresh `PipelineUnit` with the global `RealtimeService`
3. Launches coroutines that shuttle events between client and pipeline (lines 460-514)

The router catches `WebSocketDisconnect` exceptions to trigger graceful shutdown, ensuring resources are released properly.

## Real-Time Data Flow

The WebSocket protocol enables real-time communication through four continuous stages:

### 1. Client to Server: Streaming Audio Input

The browser captures microphone audio and chops it into short PCM frames. Each frame is base-64 encoded and sent as JSON:

```python

# Client-side (conceptual)

{"event": "audio_delta", "audio": "fYl9jX2NfY19jX2Nf..."}

```

On the server, `WebSocketTransport.send_ws_event` receives this frame, decodes the base64, and places raw PCM bytes onto `audio_input_queue` for the STT handler.

### 2. Pipeline Processing: STT → LLM → TTS

Audio bytes flow through the pipeline defined in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) (lines 688-691). The `s2s_pipeline` creates a `WebSocketStreamer` when raw-WebSocket mode is requested.

Processing stages execute in parallel:

- **STT handler**: Converts PCM to text (streaming transcription)
- **LLM handler**: Generates responses from transcribed text
- **TTS handler**: Synthesizes speech from LLM output

The `LMOutputProcessor` at [`src/speech_to_speech/LLM/lm_output_processor.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/lm_output_processor.py) (line 56) forwards LLM-generated text and tool calls into `text_output_queue`.

### 3. Server to Client: Streaming Audio and Text Output

When TTS completes a chunk, it produces an `AudioEventItem` on `audio_output_queue`. The `WebSocketStreamer._send_audio` method (lines 129-149) reads this queue, re-encodes PCM as base-64, and emits:

```json
{"event": "audio", "audio": "fYl9jX2NfY19jX2Nf..."}

```

Simultaneously, `_send_text` (lines 149-165) streams transcription and response text, enabling **real-time captions** that appear before audio synthesis completes.

### 4. Connection Lifecycle Management

The WebSocket protocol's persistent connection simplifies lifecycle handling:

- Connection start: Logged and registered with `RealtimeService`
- Active session: Bidirectional streaming continues until client disconnect
- Disconnect: `WebSocketDisconnect` caught in [`websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/websocket_router.py) (lines 460-514) triggers pipeline termination and queue cleanup

## Why WebSockets Enable Real-Time Performance

| Characteristic | HTTP/REST | WebSocket (as implemented) |
|----------------|-----------|---------------------------|
| Connection | Per-request handshake | Single persistent connection |
| Latency | >100ms per round-trip | <50ms frame-to-frame |
| Direction | Client request → Server response | Simultaneous bidirectional streaming |
| Framing overhead | HTTP headers per request | Minimal binary framing + JSON payload |
| Server push | Requires polling or SSE | Native push capability |

The WebSocket protocol eliminates repeated TLS handshakes and HTTP header overhead. Frames are dispatched as soon as they are ready—critical for "live" conversation where the server must respond while the user is still speaking.

## Implementation Examples

### Starting the WebSocket Server

```python
from speech_to_speech.connections.websocket_streamer import WebSocketStreamer
from speech_to_speech.arguments_classes.websocket_streamer_arguments import (
    WebSocketStreamerArguments,
)

# Configure host and port

ws_args = WebSocketStreamerArguments(host="0.0.0.0", port=8765)

streamer = WebSocketStreamer(
    host=ws_args.host,
    port=ws_args.port,
    pipeline=my_s2s_pipeline,  # Pre-built pipeline instance

)

# Run server (async internally, callable from sync code)

streamer.start()

```

This server accepts a single client and manages the full duplex connection through `WebSocketStreamer._run` (line 83-94).

### Minimal Browser Client

```javascript
const ws = new WebSocket("ws://localhost:8765");

// Connection opened: start microphone capture
ws.addEventListener("open", async () => {
  const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
  const audioCtx = new AudioContext();
  const source = audioCtx.createMediaStreamSource(stream);
  
  const processor = audioCtx.createScriptProcessor(1024, 1, 1);
  source.connect(processor);
  processor.connect(audioCtx.destination);
  
  processor.onaudioprocess = (e) => {
    // Float32 → Int16 conversion
    const floatData = e.inputBuffer.getChannelData(0);
    const int16 = new Int16Array(floatData.length);
    for (let i = 0; i < floatData.length; i++) {
      int16[i] = floatData[i] * 32767;
    }
    
    // Send base64-encoded frame
    const b64 = btoa(String.fromCharCode(...new Uint8Array(int16.buffer)));
    ws.send(JSON.stringify({ event: "audio_delta", audio: b64 }));
  };
});

// Handle server responses
ws.addEventListener("message", (ev) => {
  const msg = JSON.parse(ev.data);
  
  if (msg.event === "audio") {
    // Decode and play synthesized speech
    const pcm = new Int16Array(
      atob(msg.audio).split("").map(c => c.charCodeAt(0))
    );
    // ... feed to AudioBufferSourceNode for playback
  } else if (msg.event === "text") {
    // Display real-time transcription or response
    console.log("Assistant:", msg.text);
  }
});

```

This client matches the JSON schema implemented in `WebSocketTransport.send_ws_event` (lines 53-61 of [`transports.py`](https://github.com/huggingface/speech-to-speech/blob/main/transports.py)).

### FastAPI Router Integration

```python
from fastapi import FastAPI
from speech_to_speech.api.openai_realtime.websocket_router import (
    router as realtime_router,
)

app = FastAPI()
app.include_router(realtime_router, prefix="/v1/realtime")

# Endpoint available at ws://host:port/v1/realtime

```

The router registers WebSocket handlers at `/v1/realtime` (lines 460-473 in [`websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/websocket_router.py)), creating the full pipeline for each connecting client.

## Key Source Files

- **[`src/speech_to_speech/connections/websocket_streamer.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/connections/websocket_streamer.py)**: Core WebSocket server bridging client and pipeline
- **[`src/speech_to_speech/api/openai_realtime/transports.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/transports.py)**: JSON-over-WebSocket serialization
- **[`src/speech_to_speech/api/openai_realtime/websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/websocket_router.py)**: FastAPI route and connection lifecycle
- **[`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py)**: Pipeline orchestration with WebSocket injection
- **[`src/speech_to_speech/LLM/lm_output_processor.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/lm_output_processor.py)**: Text/tool output routing to WebSocket
- **[`demo/server.py`](https://github.com/huggingface/speech-to-speech/blob/main/demo/server.py)**: Complete example with browser UI

## Summary

- **WebSocket protocol enables real-time communication** through persistent, bidirectional connections that eliminate per-request overhead
- **Three-layer architecture** separates concerns: `WebSocketStreamer` for connections, `WebSocketTransport` for serialization, `WebSocketRouter` for HTTP integration
- **Continuous streaming** allows audio input and output to flow simultaneously without waiting for full utterances
- **OpenAI Realtime compatibility** via JSON framing makes the implementation interchangeable with standard clients
- **Sub-50ms latency** achieved by avoiding repeated handshakes and processing frames as they arrive

## Frequently Asked Questions

### What audio format does the WebSocket protocol use in this pipeline?

The pipeline accepts **16-bit PCM audio** at the sample rate configured in the STT handler. Client-side JavaScript must convert Float32 Web Audio API output to Int16, then base-64 encode for JSON transport. The server decodes back to raw bytes for the STT model.

### Can multiple clients connect to one WebSocketStreamer instance?

No. Each `WebSocketStreamer` instance in [`websocket_streamer.py`](https://github.com/huggingface/speech-to-speech/blob/main/websocket_streamer.py) accepts **exactly one client**. For multi-user deployments, spawn multiple streamer instances or use the FastAPI router ([`websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/websocket_router.py)), which creates isolated `PipelineUnit` instances per connection.

### How does the pipeline handle client disconnections?

The `WebSocketStreamer` catches disconnect exceptions in its `_run` loop (lines 83-150), drains remaining queues, and signals pipeline shutdown. The FastAPI router additionally catches `WebSocketDisconnect` (lines 460-514) to ensure `RealtimeService` unregisters the session.

### Is the WebSocket implementation compatible with OpenAI's Realtime API?

Yes. The `WebSocketTransport` class in [`transports.py`](https://github.com/huggingface/speech-to-speech/blob/main/transports.py) follows the same event schema: `audio_delta` for input, `audio` for output, and `text` for transcriptions. Third-party clients designed for OpenAI's Realtime API can connect to this server with minimal or no changes.