How to Swap LLM Backends in Speech-to-Speech: OpenAI, vLLM, llama.cpp, and Hugging Face

Use the --llm_backend CLI flag to switch between remote APIs (responses-api, chat-completions) and local inference engines (transformers, mlx-lm), then configure the endpoint or model path accordingly.

The huggingface/speech-to-speech pipeline decouples the LLM logic from the audio processing layers through a unified handler interface. By changing a single argument, you can route inference to OpenAI’s cloud API, a self-hosted llama.cpp server, or a local Hugging Face model without modifying the core pipeline code.

Architecture of the LLM Backend System

The system relies on a common base class that abstracts provider-specific implementations. This design ensures that audio streaming, buffering, and TTS generation remain unchanged regardless of which LLM generates the text.

The Base Handler Interface

All backends inherit from BaseLanguageModelHandler, which defines standard methods such as setup(), warmup(), and process(). When the pipeline starts, s2s_pipeline.py calls get_llm_handler (invoked via prepare_all_args) to instantiate the concrete handler based on your --llm_backend selection. The rest of the pipeline interacts only with this interface, remaining oblivious to whether the model runs locally or remotely.

Backend-Specific Implementations

The four supported backends map to distinct handler classes:

• responses-api → ResponsesApiModelHandler in [responses_api_language_model.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/responses_api_language_model.py)
• chat-completions → ChatCompletionsLanguageModelHandler in [chat_completions_language_model.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/chat_completions_language_model.py)
• transformers → LanguageModelHandler in [language_model.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py)
• mlx-lm → LanguageModelHandler (MLX path) in the same [language_model.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py)

The remote backends (responses-api and chat-completions) use the OpenAI Python client to stream responses via _request and _iter_stream_events, translating provider-specific formats into internal ProviderEvent streams. Local backends load models via pipeline (Transformers) or mlx_lm.load (MLX), handling token streaming through TextIteratorStreamer. The MLX path additionally uses MLXLockContext to prevent race conditions on Apple Silicon devices.

How to Configure Each Backend

All backends share common arguments defined in [language_model_arguments.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/language_model_arguments.py), such as --model_name and --llm_device. Specific connection parameters vary by backend.

OpenAI Responses API

Use this for the native OpenAI /v1/responses endpoint.

speech-to-speech serve \
  --llm_backend responses-api \
  --model_name gpt-4o-mini \
  --responses_api_api_key $OPENAI_API_KEY \
  --responses_api_base_url https://api.openai.com/v1

OpenAI-Compatible Chat Completions (vLLM, llama.cpp)

Use this backend for any server exposing the OpenAI Chat Completions schema, including vLLM, TGI, or llama.cpp’s llama-server.

speech-to-speech serve \
  --llm_backend chat-completions \
  --model_name meta-llama/Llama-2-7b-chat-hf \
  --responses_api_base_url http://127.0.0.1:8080/v1 \
  --responses_api_api_key ""

The pipeline sends requests to /v1/chat/completions via the ChatCompletionsLanguageModelHandler, making it compatible with any OpenAI-compatible local server.

Local Hugging Face Transformers

For direct inference on CUDA or CPU without an external API server.

speech-to-speech local \
  --llm_backend transformers \
  --model_name Qwen/Qwen2.5-7B-Instruct \
  --llm_device cuda \
  --llm_torch_dtype float16

The handler loads the model using Hugging Face pipeline and manages device placement and data types via arguments in [language_model.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py).

Apple Silicon with MLX-LM

Optimized for Apple Silicon devices using the MLX framework.

speech-to-speech local \
  --llm_backend mlx-lm \
  --model_name mlx-community/Llama-3.1-8B-Instruct \
  --llm_device mps

This selects the MLX code path within LanguageModelHandler, invoking mlx_load and mlx_generate while maintaining the same public interface.

Programmatic Configuration

You can also swap backends programmatically by constructing arguments before pipeline initialization:

from speech_to_speech.s2s_pipeline import parse_arguments
from speech_to_speech.arguments_classes.module_arguments import ModuleArguments

args = parse_arguments([
    "--llm_backend", "chat-completions",
    "--model_name", "local-model",
    "--responses_api_base_url", "http://localhost:8080/v1"
])

# Use args to build your pipeline

Key Source Files

File	Purpose
[src/speech_to_speech/s2s_pipeline.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py)	CLI parsing, backend selection via get_llm_handler, pipeline orchestration
[src/speech_to_speech/LLM/language_model.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py)	Base handler for transformers and mlx-lm backends
[src/speech_to_speech/LLM/responses_api_language_model.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/responses_api_language_model.py)	OpenAI Responses API implementation
[src/speech_to_speech/LLM/chat_completions_language_model.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/chat_completions_language_model.py)	OpenAI Chat Completions client (supports vLLM/llama.cpp)
[src/speech_to_speech/arguments_classes/language_model_arguments.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/language_model_arguments.py)	Shared CLI arguments (--model_name, --llm_gen_*)
[src/speech_to_speech/LLM/README.md](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/README.md)	Comprehensive backend documentation and flag reference

Summary

• The --llm_backend flag selects the concrete implementation: responses-api, chat-completions, transformers, or mlx-lm.
• Remote backends use OpenAI-compatible clients, while local backends load models directly via Transformers or MLX.
• All handlers inherit from BaseLanguageModelHandler, ensuring the audio pipeline remains agnostic to the LLM source.
• Configuration is centralized in [language_model_arguments.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/language_model_arguments.py), with backend-specific handlers registered in [s2s_pipeline.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py).

Frequently Asked Questions

Can I use vLLM with the speech-to-speech pipeline?

Yes. vLLM exposes an OpenAI-compatible Chat Completions endpoint. Set --llm_backend chat-completions and point --responses_api_base_url to your vLLM server (e.g., http://localhost:8000/v1). The ChatCompletionsLanguageModelHandler treats vLLM identically to the OpenAI API.

How do I connect to a local llama.cpp server?

Start llama-server with the --api-key option (or without), then launch the pipeline with --llm_backend chat-completions --responses_api_base_url http://127.0.0.1:8080/v1 --responses_api_api_key "". The handler sends requests to the /v1/chat/completions endpoint that llama.cpp provides.

Does the pipeline support Apple Silicon for local inference?

Yes. Use --llm_backend mlx-lm to load models via the MLX framework. This backend uses MLXLockContext to manage device locking on Apple Silicon and supports quantized MLX models from the Hugging Face Hub.

Can I switch LLM backends without restarting the service?

No. The backend selection occurs during argument parsing in s2s_pipeline.py at startup. To change backends, you must restart the process with a different --llm_backend value and the corresponding configuration arguments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}