# Memory Optimization for Running Multiple Pipeline Instances in Hugging Face Speech-to-Speech

> Optimize memory for multiple Hugging Face speech-to-speech pipeline instances by sharing models, using lightweight backends, and bounding queues to avoid GPU/CPU exhaustion.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: performance
- Published: 2026-08-03

---

**Running multiple realtime pipeline instances requires sharing model instances, selecting lightweight MLX backends, and bounding queue sizes to prevent GPU/CPU memory exhaustion when scaling beyond a single pipeline unit.**

The Hugging Face Speech-to-Speech repository enables realtime voice conversations through a modular pipeline connecting a Voice Activity Detector (VAD), Speech-to-Text (STT) model, Large Language Model (LLM), and Text-to-Speech (TTS) model. When deploying production workloads with the `--mode realtime` flag and `--num_pipelines` greater than 1, naive implementations load separate model instances per pipeline, multiplying memory consumption by the number of concurrent units. Effective memory optimization for running multiple pipeline instances involves leveraging the framework's built-in safeguards and implementing strategic model sharing to maintain performance without exhausting system resources.

## Understanding the Per-Pipeline Memory Footprint

Each realtime pipeline unit constructs independent handlers for every component in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) between lines 1085-1120. When `setup()` is called on these handlers, they load dedicated model weights into memory:

| Component | Source Location | Typical Memory Cost |
|-----------|----------------|---------------------|
| **STT** (Whisper, Faster-Whisper, Parakeet) | [`speech_to_speech/STT/..._handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/speech_to_speech/STT/..._handler.py) → `setup()` | ~1-2 GB |
| **LLM** (Transformers or MLX) | [`speech_to_speech/LLM/language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/speech_to_speech/LLM/language_model.py) → `_load_model()` (lines 642-660) | ~3-8 GB |
| **TTS** (Qwen-3, Pocket, Kokoro) | [`speech_to_speech/TTS/..._handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/speech_to_speech/TTS/..._handler.py) → `setup()` | ~1-2 GB |

Running three concurrent pipelines without optimization therefore requires 15-36 GB of additional memory just for model weights, excluding activation buffers and queue overhead.

## Built-in Memory Safeguards

The repository includes two critical mechanisms to mitigate memory pressure when pooling pipelines.

### Automatic Live Transcription Disabling

When `num_pipelines > 1` on macOS, the framework automatically disables `enable_live_transcription` in the `main()` function (lines 1070-1082 of [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py)). The progressive STT path contends on a global MLX lock, generating warning logs and allocating temporary buffers that compound across threads. This guard prevents unnecessary memory contention on Apple Silicon devices.

### Thread Lifecycle Management

The `ThreadManager` class in [`utils/thread_manager.py`](https://github.com/huggingface/speech-to-speech/blob/main/utils/thread_manager.py) (lines 9-25) creates daemon-less threads and ensures clean joining on shutdown. This prevents stray Python objects and thread-local storage from lingering in memory after pipeline termination, eliminating gradual memory leaks during high-churn deployments.

## Proven Memory-Saving Strategies

Beyond the built-in safeguards, implement these four strategies to minimize the memory footprint of multi-pipeline deployments.

### Share Model Instances Across Pipeline Units

Instead of allowing each pipeline to call `setup()` independently, load models once outside the construction loop and inject them via `setup_kwargs`. Each handler's `setup()` method accepts a `model` key, enabling reuse of a single instance across multiple pipeline units.

Modify `_build_realtime_pipeline_unit()` in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) (around line 1085) to accept pre-initialized handlers:

```python

# Initialize shared LLM once before the pool comprehension

if module_kwargs.llm_backend == "mlx-lm":
    from speech_to_speech.LLM.language_model import LanguageModelHandler
    shared_llm = LanguageModelHandler()
    shared_llm.setup(
        model_name=MLX_DEFAULT_LM_MODEL,
        device=module_kwargs.device,
        torch_dtype="float16",
        gen_kwargs={"max_new_tokens": 512},
        backend="mlx",
    )

# Inject into each pipeline unit

pool = [
    _build_realtime_pipeline_unit(
        index=i,
        stop_event=stop_event,
        module_kwargs=module_kwargs,
        language_model_handler_kwargs={"model": shared_llm},  # Reuse instance

        # ... other handlers ...

    )
    for i in range(pool_size)
]

```

This pattern eliminates redundant LLM loading while maintaining isolated VAD, STT, and TTS handlers per pipeline.

### Select Lightweight MLX Backends

For Apple Silicon deployments, specify `--llm_backend mlx-lm` to use the MLX backend instead of PyTorch. MLX models serialize execution through a global lock (implemented in [`utils/mlx_lock.py`](https://github.com/huggingface/speech-to-speech/blob/main/utils/mlx_lock.py)), dramatically reducing per-pipeline memory overhead compared to `torch` models. Configure this in `prepare_module_args()` (lines 60-73) or via CLI:

```bash
python -m speech_to_speech.s2s_pipeline \
    --llm_backend mlx-lm \
    --device mps

```

### Disable Optional Components

Remove unnecessary handlers to reclaim 1-2 GB per pipeline. Set `--stt none` if bypassing speech recognition, or omit TTS loading when using downstream audio APIs. The argument parser in `ParsedArguments` (lines 91-110) validates these configurations, though ensure your use case supports the omitted component.

### Bound Queue Sizes to Prevent Memory Backpressure

Unbounded `Queue` objects retain every intermediate chunk until consumed, keeping large tensors alive in RAM. Modify `initialize_queues_and_events()` (lines 66-78) to specify `maxsize`:

```python
from queue import Queue
from threading import Event

def initialize_queues_and_events():
    return {
        "stop_event": Event(),
        "should_listen": Event(),
        "recv_audio_chunks_queue": Queue(maxsize=5),
        "send_audio_chunks_queue": Queue(maxsize=5),
        "spoken_prompt_queue": Queue(maxsize=5),
        "stt_output_queue": Queue(maxsize=5),
        "text_prompt_queue": Queue(maxsize=5),
        "lm_response_queue": Queue(maxsize=5),
        "lm_processed_queue": Queue(maxsize=5),
        "text_output_queue": Queue(maxsize=5),
    }

```

Setting `maxsize=5` or `10` forces back-pressure when consumers lag, preventing accumulation of audio frames or text chunks.

## Implementation Examples

### Running a 2-Pipeline Realtime Server with Minimal Memory

This configuration uses MLX for minimal LLM footprint, disables STT to save ~2 GB, and leverages automatic transcription disabling on macOS:

```bash
python -m speech_to_speech.s2s_pipeline \
    --mode realtime \
    --num_pipelines 2 \
    --llm_backend mlx-lm \
    --device mps \
    --stt none \
    --tts qwen3

```

The `main()` function validates that `--num_pipelines > 1` requires realtime mode and disables live transcription on macOS (lines 1070-1082). `ThreadManager` spawns isolated threads per handler without loading duplicate STT models.

### Sharing a CUDA-Based LLM Across Three Pipelines

For NVIDIA GPU deployments, the Transformers backend serializes generation via `_transformers_lock` (lines 188-189 in `BaseLanguageModelHandler`), preventing concurrent forward passes that would duplicate GPU memory. However, each pipeline still loads separate STT and TTS models unless shared:

```bash
python -m speech_to_speech.s2s_pipeline \
    --mode realtime \
    --num_pipelines 3 \
    --llm_backend transformers \
    --device cuda \
    --stt whisper \
    --tts qwen3

```

Combine this with the shared model pattern shown above to keep the LLM resident once while allowing pipeline-specific audio handlers.

## Key Source Files for Memory Optimization

| File | Function | Memory Relevance |
|------|----------|------------------|
| [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) | Orchestrates argument parsing, validates `num_pipelines`, builds the realtime pool (lines 1085-1120), and implements macOS live transcription guards (lines 1070-1082). | [View Source](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) |
| [`src/speech_to_speech/utils/thread_manager.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/utils/thread_manager.py) | Manages daemon-less thread lifecycle (lines 9-25) to prevent zombie threads from leaking memory. | [View Source](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/utils/thread_manager.py) |
| [`src/speech_to_speech/LLM/language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py) | Loads models via `_load_model()` (lines 642-660) and implements serialization locks for Transformers and MLX backends. | [View Source](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py) |
| [`src/speech_to_speech/utils/mlx_lock.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/utils/mlx_lock.py) | Provides `MLXLockContext` to serialize MLX model calls across threads, preventing contention-related memory spikes on Apple Silicon. | [View Source](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/utils/mlx_lock.py) |

## Summary

- **Model sharing** is the highest-impact optimization: load STT, LLM, or TTS models once and inject them into multiple pipeline units via `setup_kwargs` to eliminate redundant memory allocation.
- **Backend selection** significantly affects footprint: use `--llm_backend mlx-lm` on Apple Silicon to reduce per-pipeline overhead through serialized execution.
- **Automatic safeguards** exist for macOS deployments: live transcription disables automatically when `num_pipelines > 1` to prevent MLX lock contention.
- **Queue bounding** prevents memory backpressure: set `maxsize` parameters in `initialize_queues_and_events()` to limit intermediate buffer retention.
- **Component omission** saves 1-2 GB per pipeline: use `--stt none` or similar flags when specific handlers aren't required for your use case.

## Frequently Asked Questions

### How much memory does each additional pipeline instance consume?

Each naive pipeline instance loads separate model weights totaling approximately **5-12 GB** per unit (1-2 GB for STT, 3-8 GB for LLM, 1-2 GB for TTS), plus activation memory for audio buffers and queue overhead. Sharing model instances reduces this to the base model size plus minimal per-pipeline thread overhead (~100-500 MB).

### Can I share models between pipelines on both CUDA and Apple Silicon?

Yes. The shared model pattern works across both backends. On CUDA with `transformers`, the framework already serializes LLM inference via `_transformers_lock` (lines 188-189 in [`language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/language_model.py)), making sharing safe. On Apple Silicon, MLX uses a global lock in [`mlx_lock.py`](https://github.com/huggingface/speech-to-speech/blob/main/mlx_lock.py) to serialize access to shared model weights.

### Why does live transcription get disabled automatically on macOS?

When `num_pipelines > 1` on macOS, the `main()` function (lines 1070-1082 of [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py)) forces `enable_live_transcription=False` because the progressive STT path contends on a global MLX lock. This contention generates excessive warning logs and allocates temporary buffers that compound memory usage across threads, potentially destabilizing multi-pipeline deployments.

### What is the maximum recommended number of pipelines per GPU?

The practical limit depends on your GPU's VRAM and which models you share. With full model sharing (LLM, STT, TTS loaded once), you can theoretically run 10-20 pipeline threads on a 24 GB GPU, as each thread primarily consumes activation memory rather than model weights. Without sharing, limit yourself to **1-2 pipelines per 12 GB of VRAM** to avoid out-of-memory errors during inference peaks.