# How to Run Speech-to-Speech Pipeline in 4 Modes: Realtime, Local, Socket, and Raw-WebSocket

> Explore four modes for the speech-to-speech pipeline: realtime, local, socket, and raw-websocket. Learn how to use diverse audio transport layers for your needs.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-05

---

**The speech-to-speech pipeline runs in four modes—`realtime` (default), `local`, `socket`, and `raw-websocket`—each using different audio transport layers from direct in-process streaming to TCP sockets and WebSocket protocols.**

The **huggingface/speech-to-speech** repository implements a modular pipeline that can be deployed across diverse environments, from edge devices to cloud servers. The `--mode` argument in `ModuleArguments` (defined in [`arguments_classes/module_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/arguments_classes/module_arguments.py)) determines how audio flows between microphone input, the processing chain (VAD → STT → LLM → TTS), and speaker output.

## Realtime Mode: OpenAI-Compatible WebSocket API

**Realtime mode** implements the OpenAI Realtime API specification, exposing a `/v1/realtime` WebSocket endpoint that handles both audio streaming and control events.

The entry point is `RealtimeServer` in [`api/openai_realtime/server.py`](https://github.com/huggingface/speech-to-speech/blob/main/api/openai_realtime/server.py), which launches a `uvicorn` HTTP server. Each incoming WebSocket connection spawns an isolated pipeline unit via `_build_realtime_pipeline_unit()` containing dedicated queues and handlers for voice activity detection (VAD), speech-to-text (STT), language model (LM), and text-to-speech (TTS) processing.

The `WebSocketRouter` class in [`api/openai_realtime/websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/api/openai_realtime/websocket_router.py) manages message routing, turn detection, barge-in handling, and optional LLM proxying.

Launch the realtime server:

```bash
python -m speech_to_speech \
    --mode realtime \
    --ws_host 0.0.0.0 \
    --ws_port 8765

```

Connect with the official client:

```bash
python scripts/listen_and_play_realtime.py \
    --host 127.0.0.1 --port 8765 \
    --model local

```

## Local Mode: Direct In-Process Audio Streaming

**Local mode** eliminates network overhead by running audio I/O entirely within the Python process. The `LocalAudioStreamer` class in [`connections/local_audio_streamer.py`](https://github.com/huggingface/speech-to-speech/blob/main/connections/local_audio_streamer.py) creates paired in-memory queues that bridge `sounddevice` microphone input directly to the pipeline and synthesized audio back to the speaker.

This mode achieves minimal latency and requires no external dependencies beyond the `sounddevice` library.

Launch in local mode:

```bash
python -m speech_to_speech --mode local

```

Run the bundled local client:

```bash
python scripts/listen_and_play.py

```

## Socket Mode: Plain TCP Socket Transport

**Socket mode** uses two independent TCP sockets for bidirectional audio streaming, enabling integration with external processes that implement custom socket protocols.

The architecture splits transport into:
- **`SocketReceiver`** ([`connections/socket_receiver.py`](https://github.com/huggingface/speech-to-speech/blob/main/connections/socket_receiver.py)) — listens on `recv_host:recv_port`, receives raw PCM bytes, and pushes to `recv_audio_chunks_queue`
- **`SocketSender`** ([`connections/socket_sender.py`](https://github.com/huggingface/speech-to-speech/blob/main/connections/socket_sender.py)) — connects to `send_host:send_port` and streams synthesized PCM from `send_audio_chunks_queue`

Both components run in dedicated threads with configurable host/port parameters.

Start the socket server:

```bash
python -m speech_to_speech \
    --mode socket \
    --recv_host localhost --recv_port 12345 \
    --send_host localhost --send_port 12346

```

Connect with the socket client:

```bash
python scripts/listen_and_play.py \
    --host localhost \
    --send_port 12345 \
    --recv_port 12346

```

## Raw-WebSocket Mode: Thin PCM Streaming

**Raw-websocket mode** provides a lightweight WebSocket transport that forwards raw PCM audio bytes without the full OpenAI Realtime protocol overhead. The `WebSocketStreamer` class in [`connections/websocket_streamer.py`](https://github.com/huggingface/speech-to-speech/blob/main/connections/websocket_streamer.py) exposes three queues:

- `input_queue` → pipeline (microphone audio)
- `output_queue` ← pipeline (synthesized audio)
- `text_output_queue` ← pipeline (optional transcriptions or tool call events)

This mode suits browser-based clients or simple WebSocket integrations that don't require turn detection or barge-in handling.

Launch the raw WebSocket server:

```bash
python -m speech_to_speech \
    --mode raw-websocket \
    --ws_host 0.0.0.0 \
    --ws_port 8765

```

Clients connect to `ws://<host>:8765/` and exchange raw PCM frames directly.

## How Mode Selection Works

The mode dispatch logic resides in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) at lines 677-735. When `build_pipeline()` receives the `module_kwargs.mode` value, it instantiates the corresponding communication handler:

| Mode | Handler Class | Source File |
|------|---------------|-------------|
| `local` | `LocalAudioStreamer` | [`connections/local_audio_streamer.py`](https://github.com/huggingface/speech-to-speech/blob/main/connections/local_audio_streamer.py) |
| `socket` | `SocketReceiver` + `SocketSender` | [`connections/socket_receiver.py`](https://github.com/huggingface/speech-to-speech/blob/main/connections/socket_receiver.py), [`connections/socket_sender.py`](https://github.com/huggingface/speech-to-speech/blob/main/connections/socket_sender.py) |
| `raw-websocket` | `WebSocketStreamer` | [`connections/websocket_streamer.py`](https://github.com/huggingface/speech-to-speech/blob/main/connections/websocket_streamer.py) |
| `realtime` | `RealtimeServer` | [`api/openai_realtime/server.py`](https://github.com/huggingface/speech-to-speech/blob/main/api/openai_realtime/server.py) |

## Mode Comparison: When to Use Each

Choose your deployment mode based on these characteristics:

- **`realtime`** — Production conversational agents requiring turn detection, barge-in, and OpenAI API compatibility
- **`local`** — Development, debugging, or edge deployment where network stack overhead is unacceptable
- **`socket`** — Integration with legacy systems or custom front-ends that speak TCP socket protocols
- **`raw-websocket`** — Browser-based demos, lightweight WebSocket clients, or when Realtime API features are unnecessary

## Summary

- **Four transport modes** control how audio enters and exits the speech-to-speech pipeline: `realtime`, `local`, `socket`, and `raw-websocket`
- **Mode selection** occurs in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) via the `--mode` CLI argument, defaulting to `realtime`
- **Realtime mode** provides full OpenAI Realtime API compatibility with isolated pipeline units per connection
- **Local mode** offers lowest latency through direct in-process audio streaming via `LocalAudioStreamer`
- **Socket mode** enables TCP-based integration with configurable receiver/sender ports
- **Raw-websocket mode** delivers lightweight WebSocket PCM streaming without protocol overhead

## Frequently Asked Questions

### How do I switch between speech-to-speech pipeline modes?

Pass the `--mode` argument when launching the pipeline: `python -m speech_to_speech --mode local` for local audio, `python -m speech_to_speech --mode socket` for TCP sockets, `python -m speech_to_speech --mode raw-websocket` for simple WebSockets, or omit the flag for the default `realtime` mode.

### Which mode has the lowest latency for local testing?

**Local mode** achieves minimal latency because `LocalAudioStreamer` bypasses all network layers and uses direct in-memory queues between `sounddevice` and the pipeline, eliminating serialization, socket buffers, and protocol handling overhead.

### Can I use raw-websocket mode with a browser client?

Yes. Raw-websocket mode exposes a standard WebSocket endpoint at `ws://<host>:<port>/` that accepts raw PCM bytes. Browser-based JavaScript clients can connect using the WebSocket API and stream audio without implementing the full OpenAI Realtime protocol, though you must handle PCM encoding/decoding in your client code.

### What is the difference between realtime and raw-websocket modes?

**Realtime mode** implements the complete OpenAI Realtime API with turn detection, VAD events, barge-in support, and JSON message framing via `RealtimeServer` and `WebSocketRouter`. **Raw-websocket mode** uses `WebSocketStreamer` for simple bidirectional PCM streaming without automatic turn management or protocol-level features—ideal when you want full control over the conversation flow.