Memory Optimization for Running Multiple Pipeline Instances in Hugging Face Speech-to-Speech

Running multiple realtime pipeline instances requires sharing model instances, selecting lightweight MLX backends, and bounding queue sizes to prevent GPU/CPU memory exhaustion when scaling beyond a single pipeline unit.

The Hugging Face Speech-to-Speech repository enables realtime voice conversations through a modular pipeline connecting a Voice Activity Detector (VAD), Speech-to-Text (STT) model, Large Language Model (LLM), and Text-to-Speech (TTS) model. When deploying production workloads with the --mode realtime flag and --num_pipelines greater than 1, naive implementations load separate model instances per pipeline, multiplying memory consumption by the number of concurrent units. Effective memory optimization for running multiple pipeline instances involves leveraging the framework's built-in safeguards and implementing strategic model sharing to maintain performance without exhausting system resources.

Understanding the Per-Pipeline Memory Footprint

Each realtime pipeline unit constructs independent handlers for every component in s2s_pipeline.py between lines 1085-1120. When setup() is called on these handlers, they load dedicated model weights into memory:

Component	Source Location	Typical Memory Cost
STT (Whisper, Faster-Whisper, Parakeet)	speech_to_speech/STT/..._handler.py → setup()	~1-2 GB
LLM (Transformers or MLX)	speech_to_speech/LLM/language_model.py → _load_model() (lines 642-660)	~3-8 GB
TTS (Qwen-3, Pocket, Kokoro)	speech_to_speech/TTS/..._handler.py → setup()	~1-2 GB

Running three concurrent pipelines without optimization therefore requires 15-36 GB of additional memory just for model weights, excluding activation buffers and queue overhead.

Built-in Memory Safeguards

The repository includes two critical mechanisms to mitigate memory pressure when pooling pipelines.

Automatic Live Transcription Disabling

When num_pipelines > 1 on macOS, the framework automatically disables enable_live_transcription in the main() function (lines 1070-1082 of s2s_pipeline.py). The progressive STT path contends on a global MLX lock, generating warning logs and allocating temporary buffers that compound across threads. This guard prevents unnecessary memory contention on Apple Silicon devices.

Thread Lifecycle Management

The ThreadManager class in utils/thread_manager.py (lines 9-25) creates daemon-less threads and ensures clean joining on shutdown. This prevents stray Python objects and thread-local storage from lingering in memory after pipeline termination, eliminating gradual memory leaks during high-churn deployments.

Proven Memory-Saving Strategies

Beyond the built-in safeguards, implement these four strategies to minimize the memory footprint of multi-pipeline deployments.

Share Model Instances Across Pipeline Units

Instead of allowing each pipeline to call setup() independently, load models once outside the construction loop and inject them via setup_kwargs. Each handler's setup() method accepts a model key, enabling reuse of a single instance across multiple pipeline units.

Modify _build_realtime_pipeline_unit() in s2s_pipeline.py (around line 1085) to accept pre-initialized handlers:

# Initialize shared LLM once before the pool comprehension

if module_kwargs.llm_backend == "mlx-lm":
    from speech_to_speech.LLM.language_model import LanguageModelHandler
    shared_llm = LanguageModelHandler()
    shared_llm.setup(
        model_name=MLX_DEFAULT_LM_MODEL,
        device=module_kwargs.device,
        torch_dtype="float16",
        gen_kwargs={"max_new_tokens": 512},
        backend="mlx",
    )

# Inject into each pipeline unit

pool = [
    _build_realtime_pipeline_unit(
        index=i,
        stop_event=stop_event,
        module_kwargs=module_kwargs,
        language_model_handler_kwargs={"model": shared_llm},  # Reuse instance

        # ... other handlers ...

    )
    for i in range(pool_size)
]

This pattern eliminates redundant LLM loading while maintaining isolated VAD, STT, and TTS handlers per pipeline.

Select Lightweight MLX Backends

For Apple Silicon deployments, specify --llm_backend mlx-lm to use the MLX backend instead of PyTorch. MLX models serialize execution through a global lock (implemented in utils/mlx_lock.py), dramatically reducing per-pipeline memory overhead compared to torch models. Configure this in prepare_module_args() (lines 60-73) or via CLI:

python -m speech_to_speech.s2s_pipeline \
    --llm_backend mlx-lm \
    --device mps

Disable Optional Components

Remove unnecessary handlers to reclaim 1-2 GB per pipeline. Set --stt none if bypassing speech recognition, or omit TTS loading when using downstream audio APIs. The argument parser in ParsedArguments (lines 91-110) validates these configurations, though ensure your use case supports the omitted component.

Bound Queue Sizes to Prevent Memory Backpressure

Unbounded Queue objects retain every intermediate chunk until consumed, keeping large tensors alive in RAM. Modify initialize_queues_and_events() (lines 66-78) to specify maxsize:

from queue import Queue
from threading import Event

def initialize_queues_and_events():
    return {
        "stop_event": Event(),
        "should_listen": Event(),
        "recv_audio_chunks_queue": Queue(maxsize=5),
        "send_audio_chunks_queue": Queue(maxsize=5),
        "spoken_prompt_queue": Queue(maxsize=5),
        "stt_output_queue": Queue(maxsize=5),
        "text_prompt_queue": Queue(maxsize=5),
        "lm_response_queue": Queue(maxsize=5),
        "lm_processed_queue": Queue(maxsize=5),
        "text_output_queue": Queue(maxsize=5),
    }

Setting maxsize=5 or 10 forces back-pressure when consumers lag, preventing accumulation of audio frames or text chunks.

Implementation Examples

Running a 2-Pipeline Realtime Server with Minimal Memory

This configuration uses MLX for minimal LLM footprint, disables STT to save ~2 GB, and leverages automatic transcription disabling on macOS:

python -m speech_to_speech.s2s_pipeline \
    --mode realtime \
    --num_pipelines 2 \
    --llm_backend mlx-lm \
    --device mps \
    --stt none \
    --tts qwen3

The main() function validates that --num_pipelines > 1 requires realtime mode and disables live transcription on macOS (lines 1070-1082). ThreadManager spawns isolated threads per handler without loading duplicate STT models.

Sharing a CUDA-Based LLM Across Three Pipelines

For NVIDIA GPU deployments, the Transformers backend serializes generation via _transformers_lock (lines 188-189 in BaseLanguageModelHandler), preventing concurrent forward passes that would duplicate GPU memory. However, each pipeline still loads separate STT and TTS models unless shared:

python -m speech_to_speech.s2s_pipeline \
    --mode realtime \
    --num_pipelines 3 \
    --llm_backend transformers \
    --device cuda \
    --stt whisper \
    --tts qwen3

Combine this with the shared model pattern shown above to keep the LLM resident once while allowing pipeline-specific audio handlers.

Key Source Files for Memory Optimization

File	Function	Memory Relevance
src/speech_to_speech/s2s_pipeline.py	Orchestrates argument parsing, validates num_pipelines, builds the realtime pool (lines 1085-1120), and implements macOS live transcription guards (lines 1070-1082).	View Source
src/speech_to_speech/utils/thread_manager.py	Manages daemon-less thread lifecycle (lines 9-25) to prevent zombie threads from leaking memory.	View Source
src/speech_to_speech/LLM/language_model.py	Loads models via _load_model() (lines 642-660) and implements serialization locks for Transformers and MLX backends.	View Source
src/speech_to_speech/utils/mlx_lock.py	Provides MLXLockContext to serialize MLX model calls across threads, preventing contention-related memory spikes on Apple Silicon.	View Source

Summary

• Model sharing is the highest-impact optimization: load STT, LLM, or TTS models once and inject them into multiple pipeline units via setup_kwargs to eliminate redundant memory allocation.
• Backend selection significantly affects footprint: use --llm_backend mlx-lm on Apple Silicon to reduce per-pipeline overhead through serialized execution.
• Automatic safeguards exist for macOS deployments: live transcription disables automatically when num_pipelines > 1 to prevent MLX lock contention.
• Queue bounding prevents memory backpressure: set maxsize parameters in initialize_queues_and_events() to limit intermediate buffer retention.
• Component omission saves 1-2 GB per pipeline: use --stt none or similar flags when specific handlers aren't required for your use case.

Frequently Asked Questions

How much memory does each additional pipeline instance consume?

Each naive pipeline instance loads separate model weights totaling approximately 5-12 GB per unit (1-2 GB for STT, 3-8 GB for LLM, 1-2 GB for TTS), plus activation memory for audio buffers and queue overhead. Sharing model instances reduces this to the base model size plus minimal per-pipeline thread overhead (~100-500 MB).

Can I share models between pipelines on both CUDA and Apple Silicon?

Yes. The shared model pattern works across both backends. On CUDA with transformers, the framework already serializes LLM inference via _transformers_lock (lines 188-189 in language_model.py), making sharing safe. On Apple Silicon, MLX uses a global lock in mlx_lock.py to serialize access to shared model weights.

Why does live transcription get disabled automatically on macOS?

When num_pipelines > 1 on macOS, the main() function (lines 1070-1082 of s2s_pipeline.py) forces enable_live_transcription=False because the progressive STT path contends on a global MLX lock. This contention generates excessive warning logs and allocates temporary buffers that compound memory usage across threads, potentially destabilizing multi-pipeline deployments.

What is the maximum recommended number of pipelines per GPU?

The practical limit depends on your GPU's VRAM and which models you share. With full model sharing (LLM, STT, TTS loaded once), you can theoretically run 10-20 pipeline threads on a 24 GB GPU, as each thread primarily consumes activation memory rather than model weights. Without sharing, limit yourself to 1-2 pipelines per 12 GB of VRAM to avoid out-of-memory errors during inference peaks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}