How to Select Different LLM Backends for Speech-to-Speech: A Complete Guide

Use the --llm_backend CLI flag or programmatically call select_backend() from the backend registry to switch between transformers, mlx-lm, responses-api, and chat-completions handlers.

The Hugging Face speech-to-speech repository implements a flexible backend registry system that allows you to select different LLM backends depending on your deployment environment and latency requirements. This modular architecture supports both local inference via Transformers or MLX, and API-based inference through OpenAI-compatible endpoints. Understanding how to configure these backends enables you to optimize your speech-to-speech pipeline for specific hardware constraints and model capabilities.

Available LLM Backends

The system currently supports four distinct LLM backends registered in src/speech_to_speech/backend_registry.py (lines 365-408):

• transformers: Local inference using Hugging Face Transformers
• mlx-lm: Optimized inference for Apple Silicon via MLX
• responses-api: OpenAI Responses API endpoint (default)
• chat-completions: OpenAI Chat Completions API with audio input support

Each backend exposes capability flags that determine pipeline compatibility. The chat-completions backend sets supports_audio_input=True and supports_llm_proxy=True, enabling direct audio ingestion and proxy functionality.

How the Backend Registry Works

Speech-to-Speech uses a backend registry pattern to discover and instantiate language model handlers. Each entry is a BackendSpec containing the backend name, kind ("llm"), argument dataclass, a factory function, and optional capability flags.

The LLM_BACKENDS dictionary is built at import time in src/speech_to_speech/backend_registry.py. When constructing a pipeline, the system calls select_backend() (lines 62-69) to retrieve the appropriate specification and normalize user-provided configuration. Subsequently, create_backend_handler() (lines 182-188) builds the concrete handler instance.

Selecting LLM Backends via CLI

The simplest method to switch backends uses the --llm_backend argument defined in src/speech_to_speech/arguments_classes/module_arguments.py (lines 35-41). While the default value is "responses-api", you can specify any registered name:

# Use OpenAI Chat Completions with GPT-4o-mini

speech-to-speech \
    --llm_backend chat-completions \
    --model_name gpt-4o-mini \
    --stt whisper \
    --tts qwen3

Programmatic Backend Selection

For Python applications, import the registry functions to manually instantiate handlers:

from speech_to_speech.backend_registry import LLM_BACKENDS, select_backend, create_backend_handler
from speech_to_speech.arguments_classes.language_model_arguments import LanguageModelHandlerArguments

# Configure backend name and arguments

backend_name = "transformers"
llm_args = LanguageModelHandlerArguments(
    model_name="mistralai/Mistral-7B-Instruct-v0.2",
    device="cpu",
)

# Resolve specification and normalize configuration

selection = select_backend(LLM_BACKENDS, backend_name, llm_args)

The select_backend function validates the configuration against the handler's BackendSpec requirements.

Instantiating the Handler

After selecting the backend, use create_backend_handler to build the concrete instance:

from speech_to_speech.pipeline.handler_types import HandlerContext
import threading
import queue

# Create minimal HandlerContext (normally provided by the pipeline)

context = HandlerContext(
    stop_event=threading.Event(),
    queue_in=queue.Queue(),
    queue_out=queue.Queue(),
    text_output_queue=queue.Queue(),
    should_listen=threading.Event(),
    cancel_scope=None,
    speculative_turns=None,
    pipeline_index=0,
    sample_rate=16000,
    enable_live_transcription=False,
    live_transcription_update_interval=0.5,
)

# Instantiate handler with helpful error messages for missing dependencies

handler = create_backend_handler(selection, context)

This pattern converts optional import errors into explicit ImportError exceptions suggesting the required extras package (e.g., installing openai for the chat-completions backend).

Inspecting Backend Capabilities

Before selecting a backend, verify its capabilities programmatically to ensure compatibility with your pipeline:

from speech_to_speech.backend_registry import LLM_BACKENDS

for name, spec in LLM_BACKENDS.items():
    caps = spec.capabilities
    print(f"{name}: proxy={caps.supports_llm_proxy}, audio_input={caps.supports_audio_input}")

This inspection is crucial when your pipeline requires specific features like supports_llm_proxy for intermediate routing or supports_audio_input for direct audio processing without text intermediates.

Key Implementation Files

• src/speech_to_speech/backend_registry.py: Defines BackendSpec, builds LLM_BACKENDS, and provides select_backend and create_backend_handler utilities.
• src/speech_to_speech/arguments_classes/module_arguments.py: Declares the --llm_backend CLI flag with default "responses-api".
• src/speech_to_speech/LLM/responses_api_language_model.py: OpenAI Responses API implementation optimized for proxy compatibility.
• src/speech_to_speech/LLM/chat_completions_language_model.py: Chat Completions implementation supporting audio input and proxy modes.
• src/speech_to_speech/LLM/language_model.py: Core abstraction used by transformers and mlx-lm backends for local inference.

Summary

• The speech-to-speech repository uses a backend registry pattern to manage LLM handler discovery and instantiation.
• Four backends are available: transformers, mlx-lm, responses-api, and chat-completions, each with distinct capability flags.
• Use --llm_backend <name> for CLI selection or select_backend() for programmatic configuration.
• The create_backend_handler() function converts missing dependency errors into actionable installation instructions.
• Check supports_llm_proxy and supports_audio_input capabilities before integrating backends into complex pipelines.

Frequently Asked Questions

What is the default LLM backend in speech-to-speech?

The default backend is responses-api, configured in src/speech_to_speech/arguments_classes/module_arguments.py (lines 35-41). This backend connects to OpenAI's Responses API and requires the appropriate API key environment variable to be set.

Can I use local models instead of API-based LLMs?

Yes. Select the transformers backend for standard Hugging Face models or mlx-lm for optimized inference on Apple Silicon. These backends instantiate models locally using the LanguageModelHandlerArguments configuration and do not require external API calls.

How do I handle missing dependencies for specific backends?

The create_backend_handler() function in src/speech_to_speech/backend_registry.py (lines 182-188) automatically catches import errors and raises an ImportError with specific installation instructions. For example, selecting chat-completions without the openai package installed will prompt you to install the required extra.

Which backend supports direct audio input?

The chat-completions backend explicitly sets supports_audio_input=True in its capabilities, allowing the pipeline to pass raw audio directly to the LLM without intermediate text transcription. This is implemented in src/speech_to_speech/LLM/chat_completions_language_model.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}