# How to Swap LLM Backends in Speech-to-Speech: OpenAI, vLLM, llama.cpp, and Hugging Face

> Easily swap LLM backends like OpenAI, vLLM, llama.cpp, and Hugging Face for your speech-to-speech models using the --llm_backend CLI flag and configure your endpoint or model path.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-07

---

**Use the `--llm_backend` CLI flag to switch between remote APIs (`responses-api`, `chat-completions`) and local inference engines (`transformers`, `mlx-lm`), then configure the endpoint or model path accordingly.**

The `huggingface/speech-to-speech` pipeline decouples the LLM logic from the audio processing layers through a unified handler interface. By changing a single argument, you can route inference to OpenAI’s cloud API, a self-hosted llama.cpp server, or a local Hugging Face model without modifying the core pipeline code.

## Architecture of the LLM Backend System

The system relies on a common base class that abstracts provider-specific implementations. This design ensures that audio streaming, buffering, and TTS generation remain unchanged regardless of which LLM generates the text.

### The Base Handler Interface

All backends inherit from `BaseLanguageModelHandler`, which defines standard methods such as `setup()`, `warmup()`, and `process()`. When the pipeline starts, [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) calls `get_llm_handler` (invoked via `prepare_all_args`) to instantiate the concrete handler based on your `--llm_backend` selection. The rest of the pipeline interacts only with this interface, remaining oblivious to whether the model runs locally or remotely.

### Backend-Specific Implementations

The four supported backends map to distinct handler classes:

- **`responses-api`** → `ResponsesApiModelHandler` in [[`responses_api_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/responses_api_language_model.py)](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/responses_api_language_model.py)
- **`chat-completions`** → `ChatCompletionsLanguageModelHandler` in [[`chat_completions_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/chat_completions_language_model.py)](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/chat_completions_language_model.py)
- **`transformers`** → `LanguageModelHandler` in [[`language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/language_model.py)](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py)
- **`mlx-lm`** → `LanguageModelHandler` (MLX path) in the same [[`language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/language_model.py)](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py)

The remote backends (`responses-api` and `chat-completions`) use the OpenAI Python client to stream responses via `_request` and `_iter_stream_events`, translating provider-specific formats into internal `ProviderEvent` streams. Local backends load models via `pipeline` (Transformers) or `mlx_lm.load` (MLX), handling token streaming through `TextIteratorStreamer`. The MLX path additionally uses `MLXLockContext` to prevent race conditions on Apple Silicon devices.

## How to Configure Each Backend

All backends share common arguments defined in [[`language_model_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/language_model_arguments.py)](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/language_model_arguments.py), such as `--model_name` and `--llm_device`. Specific connection parameters vary by backend.

### OpenAI Responses API

Use this for the native OpenAI `/v1/responses` endpoint.

```bash
speech-to-speech serve \
  --llm_backend responses-api \
  --model_name gpt-4o-mini \
  --responses_api_api_key $OPENAI_API_KEY \
  --responses_api_base_url https://api.openai.com/v1

```

### OpenAI-Compatible Chat Completions (vLLM, llama.cpp)

Use this backend for any server exposing the OpenAI Chat Completions schema, including vLLM, TGI, or llama.cpp’s `llama-server`.

```bash
speech-to-speech serve \
  --llm_backend chat-completions \
  --model_name meta-llama/Llama-2-7b-chat-hf \
  --responses_api_base_url http://127.0.0.1:8080/v1 \
  --responses_api_api_key ""

```

The pipeline sends requests to `/v1/chat/completions` via the `ChatCompletionsLanguageModelHandler`, making it compatible with any OpenAI-compatible local server.

### Local Hugging Face Transformers

For direct inference on CUDA or CPU without an external API server.

```bash
speech-to-speech local \
  --llm_backend transformers \
  --model_name Qwen/Qwen2.5-7B-Instruct \
  --llm_device cuda \
  --llm_torch_dtype float16

```

The handler loads the model using Hugging Face `pipeline` and manages device placement and data types via arguments in [[`language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/language_model.py)](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py).

### Apple Silicon with MLX-LM

Optimized for Apple Silicon devices using the MLX framework.

```bash
speech-to-speech local \
  --llm_backend mlx-lm \
  --model_name mlx-community/Llama-3.1-8B-Instruct \
  --llm_device mps

```

This selects the MLX code path within `LanguageModelHandler`, invoking `mlx_load` and `mlx_generate` while maintaining the same public interface.

### Programmatic Configuration

You can also swap backends programmatically by constructing arguments before pipeline initialization:

```python
from speech_to_speech.s2s_pipeline import parse_arguments
from speech_to_speech.arguments_classes.module_arguments import ModuleArguments

args = parse_arguments([
    "--llm_backend", "chat-completions",
    "--model_name", "local-model",
    "--responses_api_base_url", "http://localhost:8080/v1"
])

# Use args to build your pipeline

```

## Key Source Files

| File | Purpose |
|------|---------|
| [[`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py)](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) | CLI parsing, backend selection via `get_llm_handler`, pipeline orchestration |
| [[`src/speech_to_speech/LLM/language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py)](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py) | Base handler for `transformers` and `mlx-lm` backends |
| [[`src/speech_to_speech/LLM/responses_api_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/responses_api_language_model.py)](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/responses_api_language_model.py) | OpenAI Responses API implementation |
| [[`src/speech_to_speech/LLM/chat_completions_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/chat_completions_language_model.py)](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/chat_completions_language_model.py) | OpenAI Chat Completions client (supports vLLM/llama.cpp) |
| [[`src/speech_to_speech/arguments_classes/language_model_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/language_model_arguments.py)](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/language_model_arguments.py) | Shared CLI arguments (`--model_name`, `--llm_gen_*`) |
| [[`src/speech_to_speech/LLM/README.md`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/README.md)](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/README.md) | Comprehensive backend documentation and flag reference |

## Summary

- The `--llm_backend` flag selects the concrete implementation: `responses-api`, `chat-completions`, `transformers`, or `mlx-lm`.
- Remote backends use OpenAI-compatible clients, while local backends load models directly via Transformers or MLX.
- All handlers inherit from `BaseLanguageModelHandler`, ensuring the audio pipeline remains agnostic to the LLM source.
- Configuration is centralized in [[`language_model_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/language_model_arguments.py)](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/language_model_arguments.py), with backend-specific handlers registered in [[`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py)](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py).

## Frequently Asked Questions

### Can I use vLLM with the speech-to-speech pipeline?

Yes. vLLM exposes an OpenAI-compatible Chat Completions endpoint. Set `--llm_backend chat-completions` and point `--responses_api_base_url` to your vLLM server (e.g., `http://localhost:8000/v1`). The `ChatCompletionsLanguageModelHandler` treats vLLM identically to the OpenAI API.

### How do I connect to a local llama.cpp server?

Start `llama-server` with the `--api-key` option (or without), then launch the pipeline with `--llm_backend chat-completions --responses_api_base_url http://127.0.0.1:8080/v1 --responses_api_api_key ""`. The handler sends requests to the `/v1/chat/completions` endpoint that llama.cpp provides.

### Does the pipeline support Apple Silicon for local inference?

Yes. Use `--llm_backend mlx-lm` to load models via the MLX framework. This backend uses `MLXLockContext` to manage device locking on Apple Silicon and supports quantized MLX models from the Hugging Face Hub.

### Can I switch LLM backends without restarting the service?

No. The backend selection occurs during argument parsing in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) at startup. To change backends, you must restart the process with a different `--llm_backend` value and the corresponding configuration arguments.