How to Optimize Performance with Multiple Pipeline Instances Using `--num_pipelines` in Hugging Face Speech-to-Speech

Run the speech-to-speech server with --num_pipelines N to spawn N independent pipeline units, each handling a separate websocket session with isolated state and dedicated thread resources.

The huggingface/speech-to-speech repository implements a modular pipeline unit that chains Voice Activity Detection (VAD) → Speech-to-Text (STT) → Large Language Model (LLM) → Text-to-Speech (TTS) into a single realtime processing chain. By default, only one pipeline unit runs, limiting the server to a single concurrent client. The --num_pipelines argument removes this bottleneck by instantiating a managed pool of pipeline units.

How --num_pipelines Works Under the Hood

The --num_pipelines flag is defined in src/speech_to_speech/arguments_classes/module_arguments.py at lines 88-95. When specified, the build_pipeline function in src/speech_to_speech/s2s_pipeline.py (lines 63-66) iterates over the requested count and constructs each unit via _build_pipeline_unit (lines 20-31).

Each pipeline unit receives deep-copied handler arguments (lines 28-35), ensuring complete state isolation:

• Independent queues for audio chunks and responses
• Separate cancellation scopes per session
• No shared speculative turn state between clients

The uvicorn server listens on a single port and distributes incoming websocket connections round-robin to the first available pipeline.

Architecture Benefits of Multiple Pipeline Instances

Feature	Implementation Detail	Source Location
Concurrency scaling	Thread manager spawns handlers per pipeline	s2s_pipeline.py
State isolation	deepcopy() of all handler argument classes	s2s_pipeline.py#L28-L35
Argument validation	Ensures num_pipelines >= 1	s2s_pipeline.py#L52-L55
Platform adaptation	Auto-disables live transcription on macOS	s2s_pipeline.py#L75-L81

Platform-Specific Limitations on Apple Silicon

On macOS (platform == "darwin"), MLX inference uses a global lock implemented in src/speech_to_speech/utils/mlx_lock.py. Running multiple pipelines causes contention and repeated STT warnings.

The framework automatically applies this safeguard (lines 75-81):

# Simplified logic from s2s_pipeline.py

if platform.system() == "Darwin" and num_pipelines > 1:
    logger.warning("Live transcription disabled due to MLX global lock")
    stt_handler_kwargs["live_transcription"] = False

Linux and Windows do not have this limitation. You can run multiple pipelines with full live transcription support.

Running the Server with Multiple Pipelines

Command-Line Usage

Start the realtime server with 4 concurrent pipeline units:

speech-to-speech serve --num_pipelines 4 --log_level info

Run in local loopback mode with 2 pipelines for testing:

speech-to-speech local --num_pipelines 2 --log_level debug

Programmatic Configuration

Invoke the pipeline directly from Python:

from speech_to_speech.s2s_pipeline import run_pipeline_command

run_pipeline_command(
    command="serve",
    argv=["--num_pipelines", "3", "--log_level", "warning"]
)

Performance Optimization Guidelines

Match pipeline count to hardware capacity. Each pipeline spawns independent threads and may instantiate separate model instances. Monitor GPU memory and CPU utilization when scaling beyond 2-4 pipelines.

Disable live transcription on macOS or accept single-pipeline operation. The MLX global lock in utils/mlx_lock.py serializes inference, negating concurrency benefits.

Tune per-handler parameters to prevent resource starvation:

• Reduce --chunk-size for lower latency per pipeline
• Set explicit --device placement to distribute across GPUs
• Adjust STT batch sizes to balance throughput and memory

Summary

• --num_pipelines is defined in module_arguments.py and validated in s2s_pipeline.py
• Each pipeline unit is constructed with deep-copied arguments for complete state isolation
• Concurrency scales linearly on Linux/Windows; macOS requires trade-offs due to MLX locking
• The server distributes websocket connections automatically across the pipeline pool
• Monitor hardware resources when increasing pipeline count to maintain realtime performance

Frequently Asked Questions

What happens if I set --num_pipelines higher than my CPU cores?

The server will spawn the requested number of pipeline units, but context-switching overhead and model contention will degrade performance. Each pipeline maintains its own thread pool, so exceeding physical core count typically causes latency spikes rather than improved throughput.

Why does live transcription stop working when I increase pipelines on my Mac?

Apple Silicon uses a global MLX lock (utils/mlx_lock.py) that serializes neural network inference. The framework automatically disables live_transcription in s2s_pipeline.py lines 75-81 when num_pipelines > 1 on Darwin to prevent log flooding from STT timeout warnings. Use Linux or single-pipeline mode to preserve this feature.

Can different pipelines use different models or configurations?

No. All pipeline units receive identical deep copies of the handler arguments specified at startup. To serve different model configurations simultaneously, you must launch separate server instances on different ports.

How does the server choose which pipeline handles a new connection?

The uvicorn websocket handler queries the pipeline pool for the first unit with available capacity. This round-robin distribution happens automatically without client-side configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}