How to Integrate vLLM or llama.cpp Servers as an LLM Backend in Speech-to-Speech

You can integrate vLLM or llama.cpp servers as the LLM backend in Speech-to-Speech by setting --llm_backend to responses-api or chat-completions and pointing --responses_api_base_url to your server's OpenAI-compatible endpoint.

The huggingface/speech-to-speech repository provides a modular pipeline for real-time speech-to-speech conversation. To integrate vLLM or llama.cpp servers as the LLM backend, you leverage the built-in API abstraction layer that communicates via OpenAI-compatible REST endpoints, allowing you to swap the default local transformers backend for any self-hosted server.

Architectural Overview

The pipeline delegates all LLM operations to a pluggable backend system. When you configure an API-based backend, the system routes requests through HTTP to your self-hosted server.

The LLM Abstraction Layer

In src/speech_to_speech/LLM/language_model.py, the core interface selects the concrete client implementation based on the --llm_backend flag. This file handles the instantiation of either local model classes or remote API clients, ensuring the rest of the pipeline remains agnostic to where the LLM inference occurs.

Backend Registration

The src/speech_to_speech/backend_registry.py file maintains the registry mapping string identifiers to handler classes. It registers responses-api and chat-completions as valid options alongside local backends like transformers and mlx-lm. When you specify --llm_backend responses-api, the registry returns the appropriate client class for OpenAI-style API communication.

API Configuration Arguments

Configuration for remote servers is defined in src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py. This module exposes the CLI flags required to connect to external endpoints:

• --responses_api_base_url: The base URL of your vLLM or llama.cpp server
• --responses_api_api_key: Authentication header value (can be empty for local servers)
• --responses_api_stream: Boolean flag to enable token streaming

Integration Steps

Step 1: Launch the vLLM Server

Start your vLLM server with the model you want to use. Ensure it exposes the OpenAI-compatible API on a known port.

vllm serve gemma-2b-it --host 0.0.0.0 --port 8000

Step 2: Launch the llama.cpp Server

For llama.cpp, use the llama-server binary with your GGUF model. The server automatically provides the /v1/chat/completions and /v1/responses endpoints.

llama-server -hf ggml-org/gemma-4-E4B-it-GGUF -np 2 -c 65536 -fa on --port 8080

Step 3: Configure Speech-to-Speech

Run the pipeline with the API backend flags. You must provide a placeholder for --model_name even though the actual model is determined by the server configuration.

Complete Configuration Examples

vLLM Backend Configuration

Connect to a local vLLM instance using the chat-completions backend:

speech-to-speech serve \
    --stt parakeet-tdt \
    --llm_backend chat-completions \
    --model_name dummy \
    --responses_api_base_url http://127.0.0.1:8000/v1 \
    --responses_api_api_key "" \
    --responses_api_stream \
    --tts qwen3

llama.cpp Backend Configuration

Connect to a local llama.cpp server using the responses-api backend:

speech-to-speech serve \
    --stt parakeet-tdt \
    --llm_backend responses-api \
    --model_name dummy \
    --responses_api_base_url http://127.0.0.1:8080/v1 \
    --responses_api_api_key "" \
    --responses_api_stream \
    --tts qwen3

Both commands start the full pipeline (VAD → STT → LLM → TTS). The LLM requests route over HTTP to your local server, which returns streaming tokens exactly as the OpenAI Realtime client expects.

Key Configuration Parameters

• --llm_backend: Selects the client implementation. Use responses-api or chat-completions for OpenAI-compatible servers.
• --responses_api_base_url: The root URL of your server including the /v1 path prefix (e.g., http://localhost:8000/v1).
• --responses_api_api_key: Set to an empty string "" for local vLLM or llama.cpp instances that do not require authentication.
• --responses_api_stream: Enables streaming response handling, which is required for real-time speech-to-speech operation.
• --model_name: A placeholder value (e.g., dummy) required by the argument parser but ignored by the server when using API backends.

Summary

• Speech-to-Speech supports pluggable LLM backends via the --llm_backend flag, including responses-api and chat-completions for external servers.
• vLLM and llama.cpp expose OpenAI-compatible REST endpoints that work out-of-the-box with these backends.
• Configuration requires setting --responses_api_base_url to point at your server and providing a placeholder --model_name.
• No source code modifications are necessary; the integration works through the existing abstraction layers in language_model.py and backend_registry.py.

Frequently Asked Questions

Do I need to modify the Speech-to-Speech source code to use vLLM or llama.cpp?

No. The repository is designed with backend abstraction in mind. By using the --llm_backend and --responses_api_base_url flags, you can route all LLM traffic to external servers without touching the Python source files.

Which backend should I use, responses-api or chat-completions?

Both work with vLLM and llama.cpp. Use chat-completions if your server primarily implements the legacy /v1/chat/completions endpoint, or responses-api if you are targeting the newer OpenAI Realtime-compatible /v1/responses endpoint. Most modern llama.cpp builds support both.

Does the --model_name parameter affect which model the server uses?

No. When using API backends, the --model_name argument acts only as a placeholder to satisfy the CLI parser. The actual model loaded is determined by the server configuration (e.g., the model you passed to vllm serve or llama-server).

Can I use these integrations on Apple Silicon with the MLX backend?

No. The MLX backend (--llm_backend mlx-lm) is specifically for local Apple Silicon inference using the MLX framework. To use vLLM or llama.cpp, you must use the API backends (responses-api or chat-completions) and connect to the servers over HTTP, regardless of whether the server runs on the same machine or remotely.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}