# How to Handle Bidirectional Streaming Audio in the Speech-to-Speech Pipeline

> Learn to handle bidirectional streaming audio in the Speech-to-Speech pipeline using Hugging Face. Discover the WebSocketStreamer, SpeechToSpeech pipeline, and utility modules for seamless audio processing.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-05

---

**The Hugging Face Speech-to-Speech library implements bidirectional streaming audio through three coordinated components: `WebSocketStreamer` for network I/O, the `SpeechToSpeech` pipeline for STT→LLM→TTS orchestration, and utility modules for chunking and back-pressure management.**

Real-time speech-to-speech systems require simultaneous audio capture and playback without blocking either direction. The `huggingface/speech-to-speech` repository solves this with a full-duplex architecture that keeps latency minimal while maintaining synchronization between client and server. This guide explains how bidirectional streaming audio works and how to implement it in your own applications.

## Core Architecture for Bidirectional Streaming

The library's streaming design separates incoming and outgoing audio into independent async queues while coordinating their flow through shared control flags.

### WebSocketStreamer: The Networking Layer

The `WebSocketStreamer` class in [`src/speech_to_speech/connections/websocket_streamer.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/connections/websocket_streamer.py) manages the WebSocket connection and buffers audio in both directions.

It exposes two primary asyncio queues:

- `input_queue` — receives binary audio frames from the client
- `output_queue` — holds generated audio chunks destined for the client

Three coroutines run concurrently to handle bidirectional streaming audio:

1. **`_receive_loop`** — reads binary WebSocket frames from the client and enqueues them to `input_queue`
2. **`_process_loop`** — pulls audio from `input_queue`, feeds it through the pipeline, and writes results to `output_queue`
3. **`_send_loop`** — dequeues generated audio from `output_queue` and transmits it back to the client

Control flags coordinate the session lifecycle. The `should_listen` flag indicates when the client should stream input, and the special `SESSION_END` marker signals termination.

### SpeechToSpeech Pipeline: Streaming Orchestration

The pipeline in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) processes audio through three stages with streaming support at each step.

**Input handling:** The pipeline receives an **asynchronous generator** of audio frames from the `WebSocketStreamer`. Each STT handler yields **`Transcription`** events as soon as chunks are recognized, allowing the LLM to begin processing before the user finishes speaking.

**LLM streaming:** The LLM emits text deltas in real time, which immediately flow to the TTS handler without waiting for complete sentences.

**TTS output modes:** The TTS handler supports two operating modes:

- **Streaming mode** (`stream=True`) — returns an async iterator yielding `(audio_chunk, sample_rate, metadata)` tuples as they become available
- **Non-streaming mode** (`stream=False`) — buffers the entire utterance before emitting a single chunk

For bidirectional streaming audio, always enable streaming mode to minimize latency.

### Chunking and Synchronization Utilities

The [`utils.py`](https://github.com/huggingface/speech-to-speech/blob/main/utils.py) and [`thread_manager.py`](https://github.com/huggingface/speech-to-speech/blob/main/thread_manager.py) modules ensure smooth data flow without overwhelming either endpoint.

Key parameters include:

- `streaming_chunk_size` — default of 4–8 frames determines samples per network packet
- `should_listen` event — cleared when the server finishes sending audio, preventing client over-transmission
- Back-pressure handling — keeps client and server synchronized during variable-rate processing

## Bidirectional Streaming Audio Flow

Understanding the end-to-end flow helps debug latency issues and optimize for your use case.

### Client → Server Direction

The client opens a WebSocket and streams raw PCM audio (typically 16 kHz, 16-bit little-endian). The `_receive_loop` pushes each binary frame into `input_queue`. The pipeline immediately processes available audio through the STT handler and starts LLM inference as soon as partial transcriptions arrive.

### Server → Client Direction

As the LLM generates text, the TTS handler produces audio chunks in parallel. Each chunk enters `output_queue`, and `_send_loop` transmits them to the client for real-time playback. This overlapping generation means the client hears audio before the full LLM response completes.

### Session Termination

When the LLM finishes, the pipeline enqueues `SESSION_END`. The `WebSocketStreamer` detects this marker, clears `should_listen`, and optionally closes the socket after flushing remaining audio. This clean shutdown prevents truncated playback.

## Implementation Example

Below are minimal working examples for both client and server.

### Async WebSocket Client

```python

# example_client.py – captures microphone input and plays server responses

import asyncio
import websockets
import sounddevice as sd
import numpy as np

async def stream_audio(uri):
    async with websockets.connect(uri) as ws:
        def callback(indata, frames, time, status):
            ws.send_nowait(indata.tobytes())

        with sd.InputStream(samplerate=16000, channels=1, callback=callback):
            while True:
                data = await ws.recv()
                audio = np.frombuffer(data, dtype=np.int16)
                sd.play(audio, samplerate=16000)

asyncio.run(stream_audio("ws://localhost:8000/realtime"))

```

### Server with Bidirectional Streaming

```python

# server_side.py – launches the pipeline with WebSocketStreamer

import asyncio
from speech_to_speech.connections.websocket_streamer import WebSocketStreamer

async def run():
    streamer = WebSocketStreamer(
        host="0.0.0.0",
        port=8000,
        path="/realtime",
        stt_stream=True,         # enable streaming STT

        tts_stream=True,         # enable streaming TTS

        streaming_chunk_size=4,  # 4 frames per packet

    )
    await streamer.start()

asyncio.run(run())

```

The `WebSocketStreamer` automatically instantiates the `SpeechToSpeech` pipeline, wires `input_queue` to the STT handler, and routes TTS output back to the client.

## Key Implementation Files

| File | Purpose |
|------|---------|
| [`src/speech_to_speech/connections/websocket_streamer.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/connections/websocket_streamer.py) | WebSocket I/O, queue management, session lifecycle |
| [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) | Streaming pipeline connecting STT → LLM → TTS |
| [`src/speech_to_speech/utils/utils.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/utils/utils.py) | Audio chunking, timing, back-pressure handling |
| [`src/speech_to_speech/utils/thread_manager.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/utils/thread_manager.py) | Background thread pools for streaming handlers |

## Summary

- **Bidirectional streaming audio** in the Speech-to-Speech library relies on separate `input_queue` and `output_queue` with coordinated control flags.
- **`WebSocketStreamer`** manages the network layer with three concurrent coroutines for receive, process, and send operations.
- **`SpeechToSpeech` pipeline** enables streaming at each stage: STT yields partial transcriptions, LLM streams text deltas, and TTS outputs audio chunks immediately.
- **`streaming_chunk_size`** and **`should_listen`** provide tunable latency and flow control for real-time synchronization.
- All components use **asyncio primitives** to prevent blocking in either direction.

## Frequently Asked Questions

### What audio format does the WebSocketStreamer expect?

Raw PCM audio at 16 kHz sample rate with 16-bit little-endian encoding. The client must handle resampling and format conversion before transmission. The server assumes this format in [`src/speech_to_speech/connections/websocket_streamer.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/connections/websocket_streamer.py) for consistent processing across STT handlers.

### How does the server prevent the client from sending audio while it's still speaking?

The `should_listen` flag controls this. When the TTS handler generates audio, the server clears `should_listen`, signaling the client to pause transmission. Once output completes and `SESSION_END` processes, `should_listen` resets. This back-pressure mechanism prevents buffer overflow and desynchronization.

### Can I use non-streaming TTS with bidirectional streaming?

Technically yes, but this defeats the purpose. Non-streaming TTS (`stream=False`) buffers the complete utterance before returning audio, adding latency proportional to response length. For true bidirectional streaming audio, always set `tts_stream=True` as implemented in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py).

### What happens if the network latency exceeds the audio chunk duration?

The `streaming_chunk_size` parameter provides a buffer. Larger values increase latency tolerance but also increase playback delay. If latency persists, the client playback buffer may underrun, causing audible gaps. Monitor the `output_queue` depth and consider adaptive chunk sizing for variable network conditions.