How to Run Speech-to-Speech Pipeline in 4 Modes: Realtime, Local, Socket, and Raw-WebSocket

The speech-to-speech pipeline runs in four modes—realtime (default), local, socket, and raw-websocket—each using different audio transport layers from direct in-process streaming to TCP sockets and WebSocket protocols.

The huggingface/speech-to-speech repository implements a modular pipeline that can be deployed across diverse environments, from edge devices to cloud servers. The --mode argument in ModuleArguments (defined in arguments_classes/module_arguments.py) determines how audio flows between microphone input, the processing chain (VAD → STT → LLM → TTS), and speaker output.

Realtime Mode: OpenAI-Compatible WebSocket API

Realtime mode implements the OpenAI Realtime API specification, exposing a /v1/realtime WebSocket endpoint that handles both audio streaming and control events.

The entry point is RealtimeServer in api/openai_realtime/server.py, which launches a uvicorn HTTP server. Each incoming WebSocket connection spawns an isolated pipeline unit via _build_realtime_pipeline_unit() containing dedicated queues and handlers for voice activity detection (VAD), speech-to-text (STT), language model (LM), and text-to-speech (TTS) processing.

The WebSocketRouter class in api/openai_realtime/websocket_router.py manages message routing, turn detection, barge-in handling, and optional LLM proxying.

Launch the realtime server:

python -m speech_to_speech \
    --mode realtime \
    --ws_host 0.0.0.0 \
    --ws_port 8765

Connect with the official client:

python scripts/listen_and_play_realtime.py \
    --host 127.0.0.1 --port 8765 \
    --model local

Local Mode: Direct In-Process Audio Streaming

Local mode eliminates network overhead by running audio I/O entirely within the Python process. The LocalAudioStreamer class in connections/local_audio_streamer.py creates paired in-memory queues that bridge sounddevice microphone input directly to the pipeline and synthesized audio back to the speaker.

This mode achieves minimal latency and requires no external dependencies beyond the sounddevice library.

Launch in local mode:

python -m speech_to_speech --mode local

Run the bundled local client:

python scripts/listen_and_play.py

Socket Mode: Plain TCP Socket Transport

Socket mode uses two independent TCP sockets for bidirectional audio streaming, enabling integration with external processes that implement custom socket protocols.

The architecture splits transport into:

• SocketReceiver (connections/socket_receiver.py) — listens on recv_host:recv_port, receives raw PCM bytes, and pushes to recv_audio_chunks_queue
• SocketSender (connections/socket_sender.py) — connects to send_host:send_port and streams synthesized PCM from send_audio_chunks_queue

Both components run in dedicated threads with configurable host/port parameters.

Start the socket server:

python -m speech_to_speech \
    --mode socket \
    --recv_host localhost --recv_port 12345 \
    --send_host localhost --send_port 12346

Connect with the socket client:

python scripts/listen_and_play.py \
    --host localhost \
    --send_port 12345 \
    --recv_port 12346

Raw-WebSocket Mode: Thin PCM Streaming

Raw-websocket mode provides a lightweight WebSocket transport that forwards raw PCM audio bytes without the full OpenAI Realtime protocol overhead. The WebSocketStreamer class in connections/websocket_streamer.py exposes three queues:

• input_queue → pipeline (microphone audio)
• output_queue ← pipeline (synthesized audio)
• text_output_queue ← pipeline (optional transcriptions or tool call events)

This mode suits browser-based clients or simple WebSocket integrations that don't require turn detection or barge-in handling.

Launch the raw WebSocket server:

python -m speech_to_speech \
    --mode raw-websocket \
    --ws_host 0.0.0.0 \
    --ws_port 8765

Clients connect to ws://<host>:8765/ and exchange raw PCM frames directly.

How Mode Selection Works

The mode dispatch logic resides in src/speech_to_speech/s2s_pipeline.py at lines 677-735. When build_pipeline() receives the module_kwargs.mode value, it instantiates the corresponding communication handler:

Mode	Handler Class	Source File
local	LocalAudioStreamer	connections/local_audio_streamer.py
socket	SocketReceiver + SocketSender	connections/socket_receiver.py, connections/socket_sender.py
raw-websocket	WebSocketStreamer	connections/websocket_streamer.py
realtime	RealtimeServer	api/openai_realtime/server.py

Mode Comparison: When to Use Each

Choose your deployment mode based on these characteristics:

• realtime — Production conversational agents requiring turn detection, barge-in, and OpenAI API compatibility
• local — Development, debugging, or edge deployment where network stack overhead is unacceptable
• socket — Integration with legacy systems or custom front-ends that speak TCP socket protocols
• raw-websocket — Browser-based demos, lightweight WebSocket clients, or when Realtime API features are unnecessary

Summary

• Four transport modes control how audio enters and exits the speech-to-speech pipeline: realtime, local, socket, and raw-websocket
• Mode selection occurs in s2s_pipeline.py via the --mode CLI argument, defaulting to realtime
• Realtime mode provides full OpenAI Realtime API compatibility with isolated pipeline units per connection
• Local mode offers lowest latency through direct in-process audio streaming via LocalAudioStreamer
• Socket mode enables TCP-based integration with configurable receiver/sender ports
• Raw-websocket mode delivers lightweight WebSocket PCM streaming without protocol overhead

Frequently Asked Questions

How do I switch between speech-to-speech pipeline modes?

Pass the --mode argument when launching the pipeline: python -m speech_to_speech --mode local for local audio, python -m speech_to_speech --mode socket for TCP sockets, python -m speech_to_speech --mode raw-websocket for simple WebSockets, or omit the flag for the default realtime mode.

Which mode has the lowest latency for local testing?

Local mode achieves minimal latency because LocalAudioStreamer bypasses all network layers and uses direct in-memory queues between sounddevice and the pipeline, eliminating serialization, socket buffers, and protocol handling overhead.

Can I use raw-websocket mode with a browser client?

Yes. Raw-websocket mode exposes a standard WebSocket endpoint at ws://<host>:<port>/ that accepts raw PCM bytes. Browser-based JavaScript clients can connect using the WebSocket API and stream audio without implementing the full OpenAI Realtime protocol, though you must handle PCM encoding/decoding in your client code.

What is the difference between realtime and raw-websocket modes?

Realtime mode implements the complete OpenAI Realtime API with turn detection, VAD events, barge-in support, and JSON message framing via RealtimeServer and WebSocketRouter. Raw-websocket mode uses WebSocketStreamer for simple bidirectional PCM streaming without automatic turn management or protocol-level features—ideal when you want full control over the conversation flow.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}