LLM Backends Supported by the Hugging Face Speech-to-Speech Pipeline: Complete Configuration Guide

The Hugging Face speech-to-speech pipeline supports four LLM backends: transformers for local Hugging Face models, mlx-lm for Apple Silicon, responses-api for OpenAI Realtime endpoints, and chat-completions for OpenAI-compatible APIs.

You select your backend using the --llm_backend command-line flag (or the llm_backend field in ModuleArguments), which determines how the pipeline processes text generation. This article explains each backend's implementation, configuration options, and best use cases based on the official huggingface/speech-to-speech source code.

The Four LLM Backends Explained

The pipeline's backend selection happens early in initialization. According to src/speech_to_speech/s2s_pipeline.py (lines 68-84), the code maps string identifiers to specialized argument classes:

_backend_lm_class = {
    "responses-api": ResponsesApiLanguageModelHandlerArguments,
    "chat-completions": ChatCompletionsLanguageModelHandlerArguments,
}
_lm_class = _backend_lm_class.get(_backend, LanguageModelHandlerArguments)

Unrecognized backends fall back to LanguageModelHandlerArguments, which handles both transformers and mlx-lm through an internal backend parameter.

Transformers Backend

The transformers backend runs Hugging Face models locally using the standard 🤗 Transformers inference stack. This works on any CPU or CUDA-capable GPU.

Configuration flags:

• --llm_backend transformers
• --model_name <HF-model-id> (default: Qwen/Qwen3-4B-Instruct-2507)
• --device <cpu|cuda|...> (auto-detected from hardware)

Implementation location: src/speech_to_speech/LLM/language_model.py — class LanguageModelHandler with backend="transformers"

MLX-LM Backend

The mlx-lm backend provides optimized inference on Apple Silicon through the MLX library. It uses the same handler class as transformers but routes generation through MLX's accelerated kernels.

Configuration flags:

• --llm_backend mlx-lm
• --model_name <MLX-model-id> (default: mlx-community/Qwen3-4B-Instruct-2507-bf16)
• --device mps (optional; MLX auto-selects optimal device)

Implementation location: src/speech_to_speech/LLM/language_model.py — class LanguageModelHandler with backend="mlx"

Responses-API Backend

The responses-api backend connects to OpenAI Realtime /v1/responses endpoints. This targets remote servers implementing OpenAI's Realtime API specification for streaming audio responses.

Configuration flags:

• --llm_backend responses-api
• --model_name <remote-model-id>
• Authentication: set OPENAI_API_KEY environment variable, or pass api_key and base_url via ResponsesApiLanguageModelHandlerArguments

Implementation location: src/speech_to_speech/LLM/responses_api_language_model.py — class ResponsesApiModelHandler

Chat-Completions Backend

The chat-completions backend uses the standard OpenAI /v1/chat/completions protocol. This is the most widely supported remote API format.

Configuration flags:

• --llm_backend chat-completions
• --model_name <remote-model-id>
• Authentication: set OPENAI_API_KEY environment variable, or pass api_key and base_url via ChatCompletionsLanguageModelHandlerArguments

Implementation location: src/speech_to_speech/LLM/chat_completions_language_model.py — class ChatCompletionsApiModelHandler

Critical constraint: When running with --stt none (audio-only mode, bypassing speech-to-text), you must use the chat-completions backend. The pipeline enforces this at runtime (src/speech_to_speech/s2s_pipeline.py lines 324-326):

# Runtime validation for audio-only mode

if self.stt == "none" and self.llm_backend != "chat-completions":
    raise ValueError("--stt none requires --llm_backend chat-completions")

Configuration Examples by Use Case

Local GPU with Transformers

Run a 4B parameter model on CUDA:

python -m speech_to_speech.s2s_pipeline \
    --llm_backend transformers \
    --model_name Qwen/Qwen3-4B-Instruct-2507 \
    --device cuda

Apple Silicon with MLX

Optimized inference on M-series Macs:

python -m speech_to_speech.s2s_pipeline \
    --llm_backend mlx-lm \
    --model_name mlx-community/Qwen3-4B-Instruct-2507-bf16 \
    --device mps

Remote OpenAI Realtime Endpoint

Connect to a hosted Realtime API:

export OPENAI_API_KEY="sk-..."

python -m speech_to_speech.s2s_pipeline \
    --llm_backend responses-api \
    --model_name gpt-4o-mini \
    --device cpu

Audio-Only Mode with Chat Completions

Send raw audio directly to the LLM without STT:

export OPENAI_API_KEY="sk-..."

python -m speech_to_speech.s2s_pipeline \
    --stt none \
    --llm_backend chat-completions \
    --model_name gpt-4o-mini

Backend Selection Decision Framework

Your Environment	Recommended Backend
Linux/Windows with NVIDIA GPU	transformers
macOS with Apple Silicon (M1/M2/M3/M4)	mlx-lm
CPU-only servers	transformers
OpenAI GPT-4o / GPT-4o-mini with Realtime API	responses-api
Standard OpenAI or compatible endpoints	chat-completions
Audio-only inference (no STT)	chat-completions (required)

Key Source Files for Backend Implementation

File	Purpose
src/speech_to_speech/arguments_classes/module_arguments.py (lines 44-48)	Defines llm_backend literal options: "transformers", "mlx-lm", "responses-api", "chat-completions"
src/speech_to_speech/s2s_pipeline.py (lines 68-84)	Parses backend flag and instantiates correct argument class
src/speech_to_speech/LLM/language_model.py	Base handler for local transformers and MLX inference
src/speech_to_speech/LLM/responses_api_language_model.py	OpenAI Realtime API remote handler
src/speech_to_speech/LLM/chat_completions_language_model.py	OpenAI Chat Completions API remote handler
src/speech_to_speech/LLM/base_openai_compatible_language_model.py	Shared utilities for remote backends (auth, retries, streaming)

Summary

• Four backends cover all deployment scenarios: local (transformers, mlx-lm) and remote (responses-api, chat-completions)
• Single flag --llm_backend controls selection, parsed in s2s_pipeline.py
• Apple Silicon optimization requires mlx-lm with MLX-formatted models
• Audio-only mode (--stt none) strictly requires chat-completions backend
• Authentication for remote backends uses OPENAI_API_KEY environment variable or explicit argument class fields

Frequently Asked Questions

What is the default LLM backend if I don't specify one?

The huggingface/speech-to-speech pipeline defaults to transformers when no --llm_backend is provided. However, the default model (Qwen/Qwen3-4B-Instruct-2507) changes based on backend—for mlx-lm, the default switches to mlx-community/Qwen3-4B-Instruct-2507-bf16. Always verify your target backend matches your hardware and model availability.

Can I use a custom OpenAI-compatible server instead of OpenAI's API?

Yes. Both responses-api and chat-completions backends support custom base URLs. Pass --base_url via ResponsesApiLanguageModelHandlerArguments or ChatCompletionsLanguageModelHandlerArguments, or set it in your configuration. The base_openai_compatible_language_model.py file contains the shared HTTP client logic that handles non-OpenAI endpoints.

Why does MLX-LM require a different model ID than Transformers?

MLX models use a different weight format optimized for Apple Silicon memory architecture. The mlx-community organization on Hugging Face provides pre-converted models. While you can convert standard Transformers models to MLX format, using official mlx-community releases ensures compatibility and performance. The LanguageModelHandler automatically selects the correct inference path based on the backend parameter.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}