How to Handle Bidirectional Streaming Audio in the Speech-to-Speech Pipeline

The Hugging Face Speech-to-Speech library implements bidirectional streaming audio through three coordinated components: WebSocketStreamer for network I/O, the SpeechToSpeech pipeline for STT→LLM→TTS orchestration, and utility modules for chunking and back-pressure management.

Real-time speech-to-speech systems require simultaneous audio capture and playback without blocking either direction. The huggingface/speech-to-speech repository solves this with a full-duplex architecture that keeps latency minimal while maintaining synchronization between client and server. This guide explains how bidirectional streaming audio works and how to implement it in your own applications.

Core Architecture for Bidirectional Streaming

The library's streaming design separates incoming and outgoing audio into independent async queues while coordinating their flow through shared control flags.

WebSocketStreamer: The Networking Layer

The WebSocketStreamer class in src/speech_to_speech/connections/websocket_streamer.py manages the WebSocket connection and buffers audio in both directions.

It exposes two primary asyncio queues:

• input_queue — receives binary audio frames from the client
• output_queue — holds generated audio chunks destined for the client

Three coroutines run concurrently to handle bidirectional streaming audio:

1. _receive_loop — reads binary WebSocket frames from the client and enqueues them to input_queue
2. _process_loop — pulls audio from input_queue, feeds it through the pipeline, and writes results to output_queue
3. _send_loop — dequeues generated audio from output_queue and transmits it back to the client

Control flags coordinate the session lifecycle. The should_listen flag indicates when the client should stream input, and the special SESSION_END marker signals termination.

SpeechToSpeech Pipeline: Streaming Orchestration

The pipeline in src/speech_to_speech/s2s_pipeline.py processes audio through three stages with streaming support at each step.

Input handling: The pipeline receives an asynchronous generator of audio frames from the WebSocketStreamer. Each STT handler yields Transcription events as soon as chunks are recognized, allowing the LLM to begin processing before the user finishes speaking.

LLM streaming: The LLM emits text deltas in real time, which immediately flow to the TTS handler without waiting for complete sentences.

TTS output modes: The TTS handler supports two operating modes:

• Streaming mode (stream=True) — returns an async iterator yielding (audio_chunk, sample_rate, metadata) tuples as they become available
• Non-streaming mode (stream=False) — buffers the entire utterance before emitting a single chunk

For bidirectional streaming audio, always enable streaming mode to minimize latency.

Chunking and Synchronization Utilities

The utils.py and thread_manager.py modules ensure smooth data flow without overwhelming either endpoint.

Key parameters include:

• streaming_chunk_size — default of 4–8 frames determines samples per network packet
• should_listen event — cleared when the server finishes sending audio, preventing client over-transmission
• Back-pressure handling — keeps client and server synchronized during variable-rate processing

Bidirectional Streaming Audio Flow

Understanding the end-to-end flow helps debug latency issues and optimize for your use case.

Client → Server Direction

The client opens a WebSocket and streams raw PCM audio (typically 16 kHz, 16-bit little-endian). The _receive_loop pushes each binary frame into input_queue. The pipeline immediately processes available audio through the STT handler and starts LLM inference as soon as partial transcriptions arrive.

Server → Client Direction

As the LLM generates text, the TTS handler produces audio chunks in parallel. Each chunk enters output_queue, and _send_loop transmits them to the client for real-time playback. This overlapping generation means the client hears audio before the full LLM response completes.

Session Termination

When the LLM finishes, the pipeline enqueues SESSION_END. The WebSocketStreamer detects this marker, clears should_listen, and optionally closes the socket after flushing remaining audio. This clean shutdown prevents truncated playback.

Implementation Example

Below are minimal working examples for both client and server.

Async WebSocket Client

# example_client.py – captures microphone input and plays server responses

import asyncio
import websockets
import sounddevice as sd
import numpy as np

async def stream_audio(uri):
    async with websockets.connect(uri) as ws:
        def callback(indata, frames, time, status):
            ws.send_nowait(indata.tobytes())

        with sd.InputStream(samplerate=16000, channels=1, callback=callback):
            while True:
                data = await ws.recv()
                audio = np.frombuffer(data, dtype=np.int16)
                sd.play(audio, samplerate=16000)

asyncio.run(stream_audio("ws://localhost:8000/realtime"))

Server with Bidirectional Streaming

# server_side.py – launches the pipeline with WebSocketStreamer

import asyncio
from speech_to_speech.connections.websocket_streamer import WebSocketStreamer

async def run():
    streamer = WebSocketStreamer(
        host="0.0.0.0",
        port=8000,
        path="/realtime",
        stt_stream=True,         # enable streaming STT

        tts_stream=True,         # enable streaming TTS

        streaming_chunk_size=4,  # 4 frames per packet

    )
    await streamer.start()

asyncio.run(run())

The WebSocketStreamer automatically instantiates the SpeechToSpeech pipeline, wires input_queue to the STT handler, and routes TTS output back to the client.

Key Implementation Files

File	Purpose
src/speech_to_speech/connections/websocket_streamer.py	WebSocket I/O, queue management, session lifecycle
src/speech_to_speech/s2s_pipeline.py	Streaming pipeline connecting STT → LLM → TTS
src/speech_to_speech/utils/utils.py	Audio chunking, timing, back-pressure handling
src/speech_to_speech/utils/thread_manager.py	Background thread pools for streaming handlers

Summary

• Bidirectional streaming audio in the Speech-to-Speech library relies on separate input_queue and output_queue with coordinated control flags.
• WebSocketStreamer manages the network layer with three concurrent coroutines for receive, process, and send operations.
• SpeechToSpeech pipeline enables streaming at each stage: STT yields partial transcriptions, LLM streams text deltas, and TTS outputs audio chunks immediately.
• streaming_chunk_size and should_listen provide tunable latency and flow control for real-time synchronization.
• All components use asyncio primitives to prevent blocking in either direction.

Frequently Asked Questions

What audio format does the WebSocketStreamer expect?

Raw PCM audio at 16 kHz sample rate with 16-bit little-endian encoding. The client must handle resampling and format conversion before transmission. The server assumes this format in src/speech_to_speech/connections/websocket_streamer.py for consistent processing across STT handlers.

How does the server prevent the client from sending audio while it's still speaking?

The should_listen flag controls this. When the TTS handler generates audio, the server clears should_listen, signaling the client to pause transmission. Once output completes and SESSION_END processes, should_listen resets. This back-pressure mechanism prevents buffer overflow and desynchronization.

Can I use non-streaming TTS with bidirectional streaming?

Technically yes, but this defeats the purpose. Non-streaming TTS (stream=False) buffers the complete utterance before returning audio, adding latency proportional to response length. For true bidirectional streaming audio, always set tts_stream=True as implemented in src/speech_to_speech/s2s_pipeline.py.

What happens if the network latency exceeds the audio chunk duration?

The streaming_chunk_size parameter provides a buffer. Larger values increase latency tolerance but also increase playback delay. If latency persists, the client playback buffer may underrun, causing audible gaps. Monitor the output_queue depth and consider adaptive chunk sizing for variable network conditions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}