# Optimizing Apple Silicon Performance with mlx-lm and mlx-audio Backends

> Boost Apple Silicon performance using mlx-lm and mlx-audio backends for speech-to-speech. Optimize inference speed and accelerate your ML workflows.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: performance
- Published: 2026-08-09

---

**To maximize inference speed on Apple Silicon, configure the speech-to-speech pipeline to use the `mlx-lm` backend for language modeling and `mlx-audio-whisper` for speech-to-text, wrapped by the global `MLXLockContext` to serialize Metal command buffer access.**

The `huggingface/speech-to-speech` repository provides a unified backend registry that lets you swap implementations for each pipeline stage (STT, LLM, TTS). On macOS, leveraging the MLX ecosystem—specifically *mlx-lm* and *mlx-audio*—unlocks native Metal acceleration while avoiding the compilation overhead typical of PyTorch CUDA kernels.

## Backend Registration and Selection

The system uses `BackendSpec` objects to define available implementations across three registries: `STT_BACKENDS`, `LLM_BACKENDS`, and `TTS_BACKENDS`. Selecting an MLX backend is the first step toward Apple Silicon optimization.

### Registering mlx-lm for Language Modeling

In [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py), the `mlx-lm` backend is registered for the LLM kind:

```python
BackendSpec(
    "mlx-lm",                     # ← name

    "llm",                        # ← kind

    LanguageModelHandlerArguments,
    _create_local_llm("mlx-lm"),
    config_prefix="llm",
    required_extra="mlx-lm",
)

```

This specification binds the CLI argument `--llm_backend mlx-lm` to the `LanguageModelHandlerArguments` class and a factory that instantiates the MLX-specific handler.

### Registering mlx-audio-whisper for STT

Similarly, the Whisper STT backend using MLX audio is registered at lines 21-30 of the same file:

```python
BackendSpec(
    "mlx-audio-whisper",
    "stt",
    MLXAudioWhisperSTTHandlerArguments,
    _simple_handler_factory(
        "speech_to_speech.STT.mlx_audio_whisper_handler",
        "MLXAudioWhisperSTTHandler",
        attach_speculative_turns=True,
    ),
    config_prefix="mlx_audio_whisper",
)

```

When the CLI receives `--stt mlx-audio-whisper`, the parser constructs a `ParsedArguments` object containing `BackendSelection` instances that point to these specifications. The pipeline later materializes handlers via `create_backend_handler`.

## Serializing Metal Access with a Global MLX Lock

Metal, Apple’s GPU driver, cannot process multiple concurrent command buffers from different Python threads. To prevent race conditions and "Completed handler provided after commit call" errors, the repository implements a **re-entrant global lock** in [`src/speech_to_speech/utils/mlx_lock.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/utils/mlx_lock.py):

```python
_mlx_lock = RLock()
...
def acquire_mlx_lock(...):
    ...
    _mlx_lock.acquire(...)

```

All MLX-based handlers wrap their inference calls in an `MLXLockContext`, which acquires this lock on entry and releases it on exit.

### LLM Handler Lock Implementation

In [`src/speech_to_speech/LLM/language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py), text generation is guarded as follows:

```python
with MLXLockContext(handler_name="MLX-LLM", timeout=10.0):
    token_iter = mlx_stream_generate(...)

```

This ensures that only one thread accesses the Metal command queue at a time during LLM inference.

### STT Handler Lock Implementation

The Whisper STT handler in [`src/speech_to_speech/STT/mlx_audio_whisper_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/mlx_audio_whisper_handler.py) applies the same pattern:

```python
with MLXLockContext(handler_name=self.__class__.__name__):
    result = self.model.generate(audio_input, verbose=False, **gen_kwargs)

```

Because the lock is **re-entrant**, the same thread (for example, the STT handler) may acquire it multiple times without deadlocking, while other threads (LLM or TTS) wait until the resource is free.

## One-Command Optimization with mac-optimal-settings

For a zero-configuration experience on Apple Silicon, the CLI provides `--mac-optimal-settings`. When enabled, the parser invokes `_mac_preset_defaults` in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) (lines 76-92) to inject sensible defaults:

```python
def _mac_preset_defaults(llm_backend: str) -> dict[str, Any]:
    defaults = {
        "stt": "parakeet-tdt",
        "llm_backend": "mlx-lm",
        "tts": "qwen3",
        "stt_device": "mps",
        "paraformer_stt_device": "mps",
        "facebook_mms_device": "mps",
        "qwen3_tts_device": "mps",
    }
    if llm_backend not in {"responses-api", "chat-completions"}:
        defaults["llm_device"] = "mps"
        if llm_backend == "mlx-lm":
            defaults["model_name"] = MLX_DEFAULT_LM_MODEL
    return defaults

```

Running `speech-to-speech serve --mac-optimal-settings` automatically selects the MLX-accelerated LLM, MLX-audio Whisper STT, and Qwen3-TTS (which uses `mlx-audio` on macOS). The preset forces `mps` (Metal Performance Shaders) as the device, ensuring every component runs on the GPU.

## Why MLX Backends Deliver Superior Performance

Optimizing Apple Silicon performance with `mlx-lm` and `mlx-audio` yields measurable gains through four architectural advantages:

* **Metal-native kernels**: Both backends compile kernels once and reuse them across inference calls, eliminating the warm-up overhead associated with PyTorch JIT compilation.
* **Zero-copy data transfer**: Audio tensors remain as `float32` NumPy arrays, which MLX models accept directly without expensive tensor conversions.
* **Aggressive memory management**: MLX releases GPU caches immediately after each generation via `mx.clear_cache()`, preventing out-of-memory errors during long-running sessions.
* **Deterministic execution**: The global lock serializes access to the Metal command queue, eliminating cryptic driver errors that would otherwise abort generation when multiple threads contend for GPU resources.

These characteristics allow a single-pipeline (or multi-pipeline) server to sustain real-time latency on M1, M2, and M3 chips.

## Practical Implementation Examples

### Launch with the mac-optimal Preset

The fastest way to start an optimized server:

```bash
speech-to-speech serve --mac-optimal-settings

```

This selects `mlx-lm` for the LLM, `mlx-audio-whisper` for STT, and `qwen3` for TTS, binding all stages to the Metal backend.

### Manual MLX Backend Specification

For explicit control without the preset:

```bash
speech-to-speech serve \
    --stt mlx-audio-whisper \
    --llm_backend mlx-lm \
    --tts qwen3 \
    --device mps \
    --model_name mlx-community/Qwen3-4B-Instruct-2507-bf16

```

### Programmatic Pipeline Construction

Embed the optimized pipeline in a Python application:

```python
from speech_to_speech.s2s_pipeline import parse_arguments, build_pipeline, prepare_all_args
from threading import Event

# Parse CLI-style arguments

args = parse_arguments([
    "--stt", "mlx-audio-whisper",
    "--llm_backend", "mlx-lm",
    "--tts", "qwen3",
    "--device", "mps",
])

# Apply mac defaults and device propagation

prepare_all_args(args)

# Create and start the pipeline

stop_event = Event()
pipeline = build_pipeline(args, stop_event)
pipeline.start()
pipeline.wait()  # Blocks until SIGINT/SIGTERM

```

This exercises the same registry and lock mechanisms while allowing integration into larger systems.

## Summary

* The **backend registry** in [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py) defines `mlx-lm` and `mlx-audio-whisper` as first-class backends for LLM and STT stages.
* A **global re-entrant lock** ([`utils/mlx_lock.py`](https://github.com/huggingface/speech-to-speech/blob/main/utils/mlx_lock.py)) serializes Metal command buffer access across threads, preventing driver race conditions.
* The **`--mac-optimal-settings`** preset automatically configures MLX backends and `mps` devices for all pipeline stages.
* **MLX backends** provide zero-copy inference, immediate cache clearing, and kernel reuse, delivering real-time latency on Apple Silicon.
* All handlers use **`MLXLockContext`** to safely coordinate GPU access between STT, LLM, and TTS components.

## Frequently Asked Questions

### What is the purpose of the global MLX lock in speech-to-speech?

The global MLX lock is a re-entrant `RLock` defined in [`utils/mlx_lock.py`](https://github.com/huggingface/speech-to-speech/blob/main/utils/mlx_lock.py) that prevents multiple Python threads from submitting concurrent command buffers to Metal. Because Apple's GPU driver requires serialized access, wrapping MLX inference calls in `MLXLockContext` eliminates "commit call" errors and ensures deterministic execution across the STT, LLM, and TTS handlers.

### How does the mac-optimal-settings preset configure the pipeline?

The `_mac_preset_defaults` function in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) sets `--llm_backend mlx-lm`, `--stt mlx-audio-whisper`, and `--tts qwen3`, while forcing all device parameters to `mps`. If the LLM backend is not a cloud API (responses-api or chat-completions), it also sets the default model to `MLX_DEFAULT_LM_MODEL`, ensuring local Metal acceleration.

### Can I use mlx-lm and mlx-audio-whisper on non-Apple hardware?

No. The `mlx-lm` and `mlx-audio` libraries are specifically built for Apple Silicon, leveraging the Metal Performance Shaders (MPS) framework. Attempting to instantiate these backends on Linux or Windows will fail due to missing MLX dependencies and Metal drivers. Use the standard `transformers` or `faster-whisper` backends for cross-platform compatibility.

### Why is serialized execution necessary if MLX is fast?

While MLX compiles efficient GPU kernels, Metal cannot interleave command buffers from different threads. Without the global lock, simultaneous inference requests (e.g., STT transcribing while the LLM generates) would submit overlapping commands, causing the Metal driver to raise fatal errors. Serializing access via `MLXLockContext` maintains low latency by ensuring the GPU processes one kernel at a time without synchronization overhead.