How WebSocket Protocol Enables Real-Time Communication in the Hugging Face Speech-to-Speech Pipeline

WebSockets provide bidirectional, low-latency streaming that allows continuous audio input from clients and simultaneous audio/text output from the server without repeated HTTP handshakes.

The huggingface/speech-to-speech repository implements a complete real-time speech-to-speech pipeline using WebSockets as its core transport mechanism. This architecture enables live conversation where users speak naturally and receive synthesized responses with minimal delay. The WebSocket protocol enables real-time communication through three tightly integrated components that handle connection management, data serialization, and pipeline orchestration.

WebSocket Architecture Components

WebSocketStreamer: The Core Server

The WebSocketStreamer class in src/speech_to_speech/connections/websocket_streamer.py acts as the bridge between raw WebSocket connections and the processing pipeline. It accepts a single client, manages the connection lifecycle, and uses asyncio queues to feed audio data into the pipeline.

Key responsibilities include:

• Accepting incoming WebSocket connections on a configurable host/port
• Forwarding audio chunks from the client to the pipeline's audio_input_queue
• Reading from audio_output_queue and text_output_queue to push responses back to the client
• Handling connection cleanup when the client disconnects

The streamer runs an async event loop with two primary coroutines: _recv_audio (line 94-129) for incoming data and _send_audio/_send_text (lines 129-165) for outgoing streams.

WebSocketTransport: Data Serialization Layer

The WebSocketTransport in src/speech_to_speech/api/openai_realtime/transports.py abstracts WebSocket details behind a uniform "session transport" API. This layer implements the OpenAI Realtime specification for compatibility with third-party clients.

Critical implementation details:

• Audio deltas are base-64 encoded JSON frames: {"event": "audio_delta", "audio": "<base64>"}
• Text events use plain JSON: {"event": "text", "text": "..."}
• The send_ws_event method (lines 53-61) handles frame decoding and queue dispatch

This abstraction lets the rest of the pipeline operate on raw bytes and strings without knowing the transport mechanism.

WebSocketRouter: FastAPI Integration

The WebSocketRouter in src/speech_to_speech/api/openai_realtime/websocket_router.py wires HTTP upgrade requests into the WebSocket pipeline. When a client connects to /v1/realtime, the router:

1. Creates a WebSocketTransport instance
2. Registers a fresh PipelineUnit with the global RealtimeService
3. Launches coroutines that shuttle events between client and pipeline (lines 460-514)

The router catches WebSocketDisconnect exceptions to trigger graceful shutdown, ensuring resources are released properly.

Real-Time Data Flow

The WebSocket protocol enables real-time communication through four continuous stages:

1. Client to Server: Streaming Audio Input

The browser captures microphone audio and chops it into short PCM frames. Each frame is base-64 encoded and sent as JSON:

# Client-side (conceptual)

{"event": "audio_delta", "audio": "fYl9jX2NfY19jX2Nf..."}

On the server, WebSocketTransport.send_ws_event receives this frame, decodes the base64, and places raw PCM bytes onto audio_input_queue for the STT handler.

2. Pipeline Processing: STT → LLM → TTS

Audio bytes flow through the pipeline defined in src/speech_to_speech/s2s_pipeline.py (lines 688-691). The s2s_pipeline creates a WebSocketStreamer when raw-WebSocket mode is requested.

Processing stages execute in parallel:

• STT handler: Converts PCM to text (streaming transcription)
• LLM handler: Generates responses from transcribed text
• TTS handler: Synthesizes speech from LLM output

The LMOutputProcessor at src/speech_to_speech/LLM/lm_output_processor.py (line 56) forwards LLM-generated text and tool calls into text_output_queue.

3. Server to Client: Streaming Audio and Text Output

When TTS completes a chunk, it produces an AudioEventItem on audio_output_queue. The WebSocketStreamer._send_audio method (lines 129-149) reads this queue, re-encodes PCM as base-64, and emits:

{"event": "audio", "audio": "fYl9jX2NfY19jX2Nf..."}

Simultaneously, _send_text (lines 149-165) streams transcription and response text, enabling real-time captions that appear before audio synthesis completes.

4. Connection Lifecycle Management

The WebSocket protocol's persistent connection simplifies lifecycle handling:

• Connection start: Logged and registered with RealtimeService
• Active session: Bidirectional streaming continues until client disconnect
• Disconnect: WebSocketDisconnect caught in websocket_router.py (lines 460-514) triggers pipeline termination and queue cleanup

Why WebSockets Enable Real-Time Performance

Characteristic	HTTP/REST	WebSocket (as implemented)
Connection	Per-request handshake	Single persistent connection
Latency	>100ms per round-trip	<50ms frame-to-frame
Direction	Client request → Server response	Simultaneous bidirectional streaming
Framing overhead	HTTP headers per request	Minimal binary framing + JSON payload
Server push	Requires polling or SSE	Native push capability

The WebSocket protocol eliminates repeated TLS handshakes and HTTP header overhead. Frames are dispatched as soon as they are ready—critical for "live" conversation where the server must respond while the user is still speaking.

Implementation Examples

Starting the WebSocket Server

from speech_to_speech.connections.websocket_streamer import WebSocketStreamer
from speech_to_speech.arguments_classes.websocket_streamer_arguments import (
    WebSocketStreamerArguments,
)

# Configure host and port

ws_args = WebSocketStreamerArguments(host="0.0.0.0", port=8765)

streamer = WebSocketStreamer(
    host=ws_args.host,
    port=ws_args.port,
    pipeline=my_s2s_pipeline,  # Pre-built pipeline instance

)

# Run server (async internally, callable from sync code)

streamer.start()

This server accepts a single client and manages the full duplex connection through WebSocketStreamer._run (line 83-94).

Minimal Browser Client

const ws = new WebSocket("ws://localhost:8765");

// Connection opened: start microphone capture
ws.addEventListener("open", async () => {
  const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
  const audioCtx = new AudioContext();
  const source = audioCtx.createMediaStreamSource(stream);
  
  const processor = audioCtx.createScriptProcessor(1024, 1, 1);
  source.connect(processor);
  processor.connect(audioCtx.destination);
  
  processor.onaudioprocess = (e) => {
    // Float32 → Int16 conversion
    const floatData = e.inputBuffer.getChannelData(0);
    const int16 = new Int16Array(floatData.length);
    for (let i = 0; i < floatData.length; i++) {
      int16[i] = floatData[i] * 32767;
    }
    
    // Send base64-encoded frame
    const b64 = btoa(String.fromCharCode(...new Uint8Array(int16.buffer)));
    ws.send(JSON.stringify({ event: "audio_delta", audio: b64 }));
  };
});

// Handle server responses
ws.addEventListener("message", (ev) => {
  const msg = JSON.parse(ev.data);
  
  if (msg.event === "audio") {
    // Decode and play synthesized speech
    const pcm = new Int16Array(
      atob(msg.audio).split("").map(c => c.charCodeAt(0))
    );
    // ... feed to AudioBufferSourceNode for playback
  } else if (msg.event === "text") {
    // Display real-time transcription or response
    console.log("Assistant:", msg.text);
  }
});

This client matches the JSON schema implemented in WebSocketTransport.send_ws_event (lines 53-61 of transports.py).

FastAPI Router Integration

from fastapi import FastAPI
from speech_to_speech.api.openai_realtime.websocket_router import (
    router as realtime_router,
)

app = FastAPI()
app.include_router(realtime_router, prefix="/v1/realtime")

# Endpoint available at ws://host:port/v1/realtime

The router registers WebSocket handlers at /v1/realtime (lines 460-473 in websocket_router.py), creating the full pipeline for each connecting client.

Key Source Files

• src/speech_to_speech/connections/websocket_streamer.py: Core WebSocket server bridging client and pipeline
• src/speech_to_speech/api/openai_realtime/transports.py: JSON-over-WebSocket serialization
• src/speech_to_speech/api/openai_realtime/websocket_router.py: FastAPI route and connection lifecycle
• src/speech_to_speech/s2s_pipeline.py: Pipeline orchestration with WebSocket injection
• src/speech_to_speech/LLM/lm_output_processor.py: Text/tool output routing to WebSocket
• demo/server.py: Complete example with browser UI

Summary

• WebSocket protocol enables real-time communication through persistent, bidirectional connections that eliminate per-request overhead
• Three-layer architecture separates concerns: WebSocketStreamer for connections, WebSocketTransport for serialization, WebSocketRouter for HTTP integration
• Continuous streaming allows audio input and output to flow simultaneously without waiting for full utterances
• OpenAI Realtime compatibility via JSON framing makes the implementation interchangeable with standard clients
• Sub-50ms latency achieved by avoiding repeated handshakes and processing frames as they arrive

Frequently Asked Questions

What audio format does the WebSocket protocol use in this pipeline?

The pipeline accepts 16-bit PCM audio at the sample rate configured in the STT handler. Client-side JavaScript must convert Float32 Web Audio API output to Int16, then base-64 encode for JSON transport. The server decodes back to raw bytes for the STT model.

Can multiple clients connect to one WebSocketStreamer instance?

No. Each WebSocketStreamer instance in websocket_streamer.py accepts exactly one client. For multi-user deployments, spawn multiple streamer instances or use the FastAPI router (websocket_router.py), which creates isolated PipelineUnit instances per connection.

How does the pipeline handle client disconnections?

The WebSocketStreamer catches disconnect exceptions in its _run loop (lines 83-150), drains remaining queues, and signals pipeline shutdown. The FastAPI router additionally catches WebSocketDisconnect (lines 460-514) to ensure RealtimeService unregisters the session.

Is the WebSocket implementation compatible with OpenAI's Realtime API?

Yes. The WebSocketTransport class in transports.py follows the same event schema: audio_delta for input, audio for output, and text for transcriptions. Third-party clients designed for OpenAI's Realtime API can connect to this server with minimal or no changes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}