# How to Select Different LLM Backends for Speech-to-Speech: A Complete Guide

> Master selecting LLM backends for speech-to-speech with this guide. Easily switch between transformers, mlx-lm, and more using CLI flags or Python functions.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-11

---

**Use the `--llm_backend` CLI flag or programmatically call `select_backend()` from the backend registry to switch between transformers, mlx-lm, responses-api, and chat-completions handlers.**

The Hugging Face speech-to-speech repository implements a flexible backend registry system that allows you to select different LLM backends depending on your deployment environment and latency requirements. This modular architecture supports both local inference via Transformers or MLX, and API-based inference through OpenAI-compatible endpoints. Understanding how to configure these backends enables you to optimize your speech-to-speech pipeline for specific hardware constraints and model capabilities.

## Available LLM Backends

The system currently supports four distinct LLM backends registered in [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py) (lines 365-408):

- **transformers**: Local inference using Hugging Face Transformers
- **mlx-lm**: Optimized inference for Apple Silicon via MLX
- **responses-api**: OpenAI Responses API endpoint (default)
- **chat-completions**: OpenAI Chat Completions API with audio input support

Each backend exposes capability flags that determine pipeline compatibility. The **chat-completions** backend sets `supports_audio_input=True` and `supports_llm_proxy=True`, enabling direct audio ingestion and proxy functionality.

## How the Backend Registry Works

Speech-to-Speech uses a **backend registry** pattern to discover and instantiate language model handlers. Each entry is a `BackendSpec` containing the backend name, kind (`"llm"`), argument dataclass, a factory function, and optional capability flags.

The `LLM_BACKENDS` dictionary is built at import time in [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py). When constructing a pipeline, the system calls `select_backend()` (lines 62-69) to retrieve the appropriate specification and normalize user-provided configuration. Subsequently, `create_backend_handler()` (lines 182-188) builds the concrete handler instance.

## Selecting LLM Backends via CLI

The simplest method to switch backends uses the `--llm_backend` argument defined in [`src/speech_to_speech/arguments_classes/module_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/module_arguments.py) (lines 35-41). While the default value is `"responses-api"`, you can specify any registered name:

```bash

# Use OpenAI Chat Completions with GPT-4o-mini

speech-to-speech \
    --llm_backend chat-completions \
    --model_name gpt-4o-mini \
    --stt whisper \
    --tts qwen3

```

## Programmatic Backend Selection

For Python applications, import the registry functions to manually instantiate handlers:

```python
from speech_to_speech.backend_registry import LLM_BACKENDS, select_backend, create_backend_handler
from speech_to_speech.arguments_classes.language_model_arguments import LanguageModelHandlerArguments

# Configure backend name and arguments

backend_name = "transformers"
llm_args = LanguageModelHandlerArguments(
    model_name="mistralai/Mistral-7B-Instruct-v0.2",
    device="cpu",
)

# Resolve specification and normalize configuration

selection = select_backend(LLM_BACKENDS, backend_name, llm_args)

```

The `select_backend` function validates the configuration against the handler's `BackendSpec` requirements.

## Instantiating the Handler

After selecting the backend, use `create_backend_handler` to build the concrete instance:

```python
from speech_to_speech.pipeline.handler_types import HandlerContext
import threading
import queue

# Create minimal HandlerContext (normally provided by the pipeline)

context = HandlerContext(
    stop_event=threading.Event(),
    queue_in=queue.Queue(),
    queue_out=queue.Queue(),
    text_output_queue=queue.Queue(),
    should_listen=threading.Event(),
    cancel_scope=None,
    speculative_turns=None,
    pipeline_index=0,
    sample_rate=16000,
    enable_live_transcription=False,
    live_transcription_update_interval=0.5,
)

# Instantiate handler with helpful error messages for missing dependencies

handler = create_backend_handler(selection, context)

```

This pattern converts optional import errors into explicit `ImportError` exceptions suggesting the required extras package (e.g., installing `openai` for the chat-completions backend).

## Inspecting Backend Capabilities

Before selecting a backend, verify its capabilities programmatically to ensure compatibility with your pipeline:

```python
from speech_to_speech.backend_registry import LLM_BACKENDS

for name, spec in LLM_BACKENDS.items():
    caps = spec.capabilities
    print(f"{name}: proxy={caps.supports_llm_proxy}, audio_input={caps.supports_audio_input}")

```

This inspection is crucial when your pipeline requires specific features like `supports_llm_proxy` for intermediate routing or `supports_audio_input` for direct audio processing without text intermediates.

## Key Implementation Files

- **[`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py)**: Defines `BackendSpec`, builds `LLM_BACKENDS`, and provides `select_backend` and `create_backend_handler` utilities.
- **[`src/speech_to_speech/arguments_classes/module_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/module_arguments.py)**: Declares the `--llm_backend` CLI flag with default `"responses-api"`.
- **[`src/speech_to_speech/LLM/responses_api_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/responses_api_language_model.py)**: OpenAI Responses API implementation optimized for proxy compatibility.
- **[`src/speech_to_speech/LLM/chat_completions_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/chat_completions_language_model.py)**: Chat Completions implementation supporting audio input and proxy modes.
- **[`src/speech_to_speech/LLM/language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py)**: Core abstraction used by `transformers` and `mlx-lm` backends for local inference.

## Summary

- The speech-to-speech repository uses a **backend registry pattern** to manage LLM handler discovery and instantiation.
- Four backends are available: **transformers**, **mlx-lm**, **responses-api**, and **chat-completions**, each with distinct capability flags.
- Use **`--llm_backend <name>`** for CLI selection or **`select_backend()`** for programmatic configuration.
- The **`create_backend_handler()`** function converts missing dependency errors into actionable installation instructions.
- Check **`supports_llm_proxy`** and **`supports_audio_input`** capabilities before integrating backends into complex pipelines.

## Frequently Asked Questions

### What is the default LLM backend in speech-to-speech?

The default backend is **responses-api**, configured in [`src/speech_to_speech/arguments_classes/module_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/module_arguments.py) (lines 35-41). This backend connects to OpenAI's Responses API and requires the appropriate API key environment variable to be set.

### Can I use local models instead of API-based LLMs?

Yes. Select the **transformers** backend for standard Hugging Face models or **mlx-lm** for optimized inference on Apple Silicon. These backends instantiate models locally using the `LanguageModelHandlerArguments` configuration and do not require external API calls.

### How do I handle missing dependencies for specific backends?

The `create_backend_handler()` function in [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py) (lines 182-188) automatically catches import errors and raises an `ImportError` with specific installation instructions. For example, selecting `chat-completions` without the `openai` package installed will prompt you to install the required extra.

### Which backend supports direct audio input?

The **chat-completions** backend explicitly sets `supports_audio_input=True` in its capabilities, allowing the pipeline to pass raw audio directly to the LLM without intermediate text transcription. This is implemented in [`src/speech_to_speech/LLM/chat_completions_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/chat_completions_language_model.py).