# How to Optimize Performance with Multiple Pipeline Instances Using `--num_pipelines` in Hugging Face Speech-to-Speech

> Optimize Hugging Face Speech-to-Speech performance by running multiple pipeline instances using --num_pipelines. Boost throughput with isolated state and dedicated threads.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: performance
- Published: 2026-08-07

---

**Run the `speech-to-speech` server with `--num_pipelines N` to spawn N independent pipeline units, each handling a separate websocket session with isolated state and dedicated thread resources.**

The `huggingface/speech-to-speech` repository implements a modular **pipeline unit** that chains Voice Activity Detection (VAD) → Speech-to-Text (STT) → Large Language Model (LLM) → Text-to-Speech (TTS) into a single realtime processing chain. By default, only one pipeline unit runs, limiting the server to a single concurrent client. The `--num_pipelines` argument removes this bottleneck by instantiating a managed pool of pipeline units.

## How `--num_pipelines` Works Under the Hood

The `--num_pipelines` flag is defined in [`src/speech_to_speech/arguments_classes/module_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/module_arguments.py) at lines 88-95. When specified, the `build_pipeline` function in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) (lines 63-66) iterates over the requested count and constructs each unit via `_build_pipeline_unit` (lines 20-31).

Each pipeline unit receives **deep-copied handler arguments** (lines 28-35), ensuring complete state isolation:

- Independent queues for audio chunks and responses
- Separate cancellation scopes per session
- No shared speculative turn state between clients

The uvicorn server listens on a single port and distributes incoming websocket connections round-robin to the first available pipeline.

## Architecture Benefits of Multiple Pipeline Instances

| Feature | Implementation Detail | Source Location |
|---------|----------------------|-----------------|
| **Concurrency scaling** | Thread manager spawns handlers per pipeline | [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) |
| **State isolation** | `deepcopy()` of all handler argument classes | `s2s_pipeline.py#L28-L35` |
| **Argument validation** | Ensures `num_pipelines >= 1` | `s2s_pipeline.py#L52-L55` |
| **Platform adaptation** | Auto-disables live transcription on macOS | `s2s_pipeline.py#L75-L81` |

## Platform-Specific Limitations on Apple Silicon

On macOS (`platform == "darwin"`), MLX inference uses a **global lock** implemented in [`src/speech_to_speech/utils/mlx_lock.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/utils/mlx_lock.py). Running multiple pipelines causes contention and repeated STT warnings.

The framework automatically applies this safeguard (lines 75-81):

```python

# Simplified logic from s2s_pipeline.py

if platform.system() == "Darwin" and num_pipelines > 1:
    logger.warning("Live transcription disabled due to MLX global lock")
    stt_handler_kwargs["live_transcription"] = False

```

**Linux and Windows do not have this limitation.** You can run multiple pipelines with full live transcription support.

## Running the Server with Multiple Pipelines

### Command-Line Usage

Start the realtime server with 4 concurrent pipeline units:

```bash
speech-to-speech serve --num_pipelines 4 --log_level info

```

Run in local loopback mode with 2 pipelines for testing:

```bash
speech-to-speech local --num_pipelines 2 --log_level debug

```

### Programmatic Configuration

Invoke the pipeline directly from Python:

```python
from speech_to_speech.s2s_pipeline import run_pipeline_command

run_pipeline_command(
    command="serve",
    argv=["--num_pipelines", "3", "--log_level", "warning"]
)

```

## Performance Optimization Guidelines

**Match pipeline count to hardware capacity.** Each pipeline spawns independent threads and may instantiate separate model instances. Monitor GPU memory and CPU utilization when scaling beyond 2-4 pipelines.

**Disable live transcription on macOS** or accept single-pipeline operation. The MLX global lock in [`utils/mlx_lock.py`](https://github.com/huggingface/speech-to-speech/blob/main/utils/mlx_lock.py) serializes inference, negating concurrency benefits.

**Tune per-handler parameters** to prevent resource starvation:

- Reduce `--chunk-size` for lower latency per pipeline
- Set explicit `--device` placement to distribute across GPUs
- Adjust STT batch sizes to balance throughput and memory

## Summary

- **`--num_pipelines`** is defined in [`module_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/module_arguments.py) and validated in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py)
- Each pipeline unit is constructed with **deep-copied arguments** for complete state isolation
- **Concurrency scales linearly** on Linux/Windows; **macOS requires trade-offs** due to MLX locking
- The server distributes websocket connections automatically across the pipeline pool
- Monitor hardware resources when increasing pipeline count to maintain realtime performance

## Frequently Asked Questions

### What happens if I set `--num_pipelines` higher than my CPU cores?

The server will spawn the requested number of pipeline units, but context-switching overhead and model contention will degrade performance. Each pipeline maintains its own thread pool, so exceeding physical core count typically causes latency spikes rather than improved throughput.

### Why does live transcription stop working when I increase pipelines on my Mac?

Apple Silicon uses a global MLX lock ([`utils/mlx_lock.py`](https://github.com/huggingface/speech-to-speech/blob/main/utils/mlx_lock.py)) that serializes neural network inference. The framework automatically disables `live_transcription` in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) lines 75-81 when `num_pipelines > 1` on Darwin to prevent log flooding from STT timeout warnings. Use Linux or single-pipeline mode to preserve this feature.

### Can different pipelines use different models or configurations?

No. All pipeline units receive identical deep copies of the handler arguments specified at startup. To serve different model configurations simultaneously, you must launch separate server instances on different ports.

### How does the server choose which pipeline handles a new connection?

The uvicorn websocket handler queries the pipeline pool for the first unit with available capacity. This round-robin distribution happens automatically without client-side configuration.