# LLM Backends Supported by the Hugging Face Speech-to-Speech Pipeline: Complete Configuration Guide

> Discover LLM backends for Hugging Face speech-to-speech: transformers, mlx-lm, responses-api, and chat-completions. Get your complete configuration guide here.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-05

---

**The Hugging Face speech-to-speech pipeline supports four LLM backends: `transformers` for local Hugging Face models, `mlx-lm` for Apple Silicon, `responses-api` for OpenAI Realtime endpoints, and `chat-completions` for OpenAI-compatible APIs.**

You select your backend using the `--llm_backend` command-line flag (or the `llm_backend` field in `ModuleArguments`), which determines how the pipeline processes text generation. This article explains each backend's implementation, configuration options, and best use cases based on the official `huggingface/speech-to-speech` source code.

## The Four LLM Backends Explained

The pipeline's backend selection happens early in initialization. According to [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) (lines 68-84), the code maps string identifiers to specialized argument classes:

```python
_backend_lm_class = {
    "responses-api": ResponsesApiLanguageModelHandlerArguments,
    "chat-completions": ChatCompletionsLanguageModelHandlerArguments,
}
_lm_class = _backend_lm_class.get(_backend, LanguageModelHandlerArguments)

```

Unrecognized backends fall back to `LanguageModelHandlerArguments`, which handles both **transformers** and **mlx-lm** through an internal `backend` parameter.

### Transformers Backend

The **transformers** backend runs Hugging Face models locally using the standard 🤗 Transformers inference stack. This works on any CPU or CUDA-capable GPU.

Configuration flags:
- `--llm_backend transformers`
- `--model_name <HF-model-id>` (default: `Qwen/Qwen3-4B-Instruct-2507`)
- `--device <cpu|cuda|...>` (auto-detected from hardware)

Implementation location: [`src/speech_to_speech/LLM/language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py) — class `LanguageModelHandler` with `backend="transformers"`

### MLX-LM Backend

The **mlx-lm** backend provides optimized inference on Apple Silicon through the **MLX** library. It uses the same handler class as transformers but routes generation through MLX's accelerated kernels.

Configuration flags:
- `--llm_backend mlx-lm`
- `--model_name <MLX-model-id>` (default: `mlx-community/Qwen3-4B-Instruct-2507-bf16`)
- `--device mps` (optional; MLX auto-selects optimal device)

Implementation location: [`src/speech_to_speech/LLM/language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py) — class `LanguageModelHandler` with `backend="mlx"`

### Responses-API Backend

The **responses-api** backend connects to **OpenAI Realtime /v1/responses** endpoints. This targets remote servers implementing OpenAI's Realtime API specification for streaming audio responses.

Configuration flags:
- `--llm_backend responses-api`
- `--model_name <remote-model-id>`
- Authentication: set `OPENAI_API_KEY` environment variable, or pass `api_key` and `base_url` via `ResponsesApiLanguageModelHandlerArguments`

Implementation location: [`src/speech_to_speech/LLM/responses_api_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/responses_api_language_model.py) — class `ResponsesApiModelHandler`

### Chat-Completions Backend

The **chat-completions** backend uses the standard **OpenAI /v1/chat/completions** protocol. This is the most widely supported remote API format.

Configuration flags:
- `--llm_backend chat-completions`
- `--model_name <remote-model-id>`
- Authentication: set `OPENAI_API_KEY` environment variable, or pass `api_key` and `base_url` via `ChatCompletionsLanguageModelHandlerArguments`

Implementation location: [`src/speech_to_speech/LLM/chat_completions_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/chat_completions_language_model.py) — class `ChatCompletionsApiModelHandler`

**Critical constraint:** When running with `--stt none` (audio-only mode, bypassing speech-to-text), you **must** use the `chat-completions` backend. The pipeline enforces this at runtime ([`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) lines 324-326):

```python

# Runtime validation for audio-only mode

if self.stt == "none" and self.llm_backend != "chat-completions":
    raise ValueError("--stt none requires --llm_backend chat-completions")

```

## Configuration Examples by Use Case

### Local GPU with Transformers

Run a 4B parameter model on CUDA:

```bash
python -m speech_to_speech.s2s_pipeline \
    --llm_backend transformers \
    --model_name Qwen/Qwen3-4B-Instruct-2507 \
    --device cuda

```

### Apple Silicon with MLX

Optimized inference on M-series Macs:

```bash
python -m speech_to_speech.s2s_pipeline \
    --llm_backend mlx-lm \
    --model_name mlx-community/Qwen3-4B-Instruct-2507-bf16 \
    --device mps

```

### Remote OpenAI Realtime Endpoint

Connect to a hosted Realtime API:

```bash
export OPENAI_API_KEY="sk-..."

python -m speech_to_speech.s2s_pipeline \
    --llm_backend responses-api \
    --model_name gpt-4o-mini \
    --device cpu

```

### Audio-Only Mode with Chat Completions

Send raw audio directly to the LLM without STT:

```bash
export OPENAI_API_KEY="sk-..."

python -m speech_to_speech.s2s_pipeline \
    --stt none \
    --llm_backend chat-completions \
    --model_name gpt-4o-mini

```

## Backend Selection Decision Framework

| Your Environment | Recommended Backend |
|-----------------|---------------------|
| Linux/Windows with NVIDIA GPU | `transformers` |
| macOS with Apple Silicon (M1/M2/M3/M4) | `mlx-lm` |
| CPU-only servers | `transformers` |
| OpenAI GPT-4o / GPT-4o-mini with Realtime API | `responses-api` |
| Standard OpenAI or compatible endpoints | `chat-completions` |
| Audio-only inference (no STT) | `chat-completions` (required) |

## Key Source Files for Backend Implementation

| File | Purpose |
|------|---------|
| [`src/speech_to_speech/arguments_classes/module_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/module_arguments.py) (lines 44-48) | Defines `llm_backend` literal options: `"transformers"`, `"mlx-lm"`, `"responses-api"`, `"chat-completions"` |
| [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) (lines 68-84) | Parses backend flag and instantiates correct argument class |
| [`src/speech_to_speech/LLM/language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py) | Base handler for local transformers and MLX inference |
| [`src/speech_to_speech/LLM/responses_api_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/responses_api_language_model.py) | OpenAI Realtime API remote handler |
| [`src/speech_to_speech/LLM/chat_completions_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/chat_completions_language_model.py) | OpenAI Chat Completions API remote handler |
| [`src/speech_to_speech/LLM/base_openai_compatible_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/base_openai_compatible_language_model.py) | Shared utilities for remote backends (auth, retries, streaming) |

## Summary

- **Four backends** cover all deployment scenarios: local (transformers, mlx-lm) and remote (responses-api, chat-completions)
- **Single flag** `--llm_backend` controls selection, parsed in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py)
- **Apple Silicon optimization** requires `mlx-lm` with MLX-formatted models
- **Audio-only mode** (`--stt none`) strictly requires `chat-completions` backend
- **Authentication** for remote backends uses `OPENAI_API_KEY` environment variable or explicit argument class fields

## Frequently Asked Questions

### What is the default LLM backend if I don't specify one?

The `huggingface/speech-to-speech` pipeline defaults to `transformers` when no `--llm_backend` is provided. However, the default model (`Qwen/Qwen3-4B-Instruct-2507`) changes based on backend—for `mlx-lm`, the default switches to `mlx-community/Qwen3-4B-Instruct-2507-bf16`. Always verify your target backend matches your hardware and model availability.

### Can I use a custom OpenAI-compatible server instead of OpenAI's API?

Yes. Both `responses-api` and `chat-completions` backends support custom base URLs. Pass `--base_url` via `ResponsesApiLanguageModelHandlerArguments` or `ChatCompletionsLanguageModelHandlerArguments`, or set it in your configuration. The [`base_openai_compatible_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/base_openai_compatible_language_model.py) file contains the shared HTTP client logic that handles non-OpenAI endpoints.

### Why does MLX-LM require a different model ID than Transformers?

MLX models use a different weight format optimized for Apple Silicon memory architecture. The `mlx-community` organization on Hugging Face provides pre-converted models. While you *can* convert standard Transformers models to MLX format, using official `mlx-community` releases ensures compatibility and performance. The `LanguageModelHandler` automatically selects the correct inference path based on the `backend` parameter.