# How to Integrate vLLM or llama.cpp Servers as an LLM Backend in Speech-to-Speech

> Learn to integrate vLLM or llama.cpp as your LLM backend in Speech-to-Speech. Set the llm_backend to responses-api or chat-completions and use your server's OpenAI compatible endpoint.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-10

---

**You can integrate vLLM or llama.cpp servers as the LLM backend in Speech-to-Speech by setting `--llm_backend` to `responses-api` or `chat-completions` and pointing `--responses_api_base_url` to your server's OpenAI-compatible endpoint.**

The **huggingface/speech-to-speech** repository provides a modular pipeline for real-time speech-to-speech conversation. To integrate vLLM or llama.cpp servers as the LLM backend, you leverage the built-in API abstraction layer that communicates via OpenAI-compatible REST endpoints, allowing you to swap the default local transformers backend for any self-hosted server.

## Architectural Overview

The pipeline delegates all LLM operations to a pluggable backend system. When you configure an API-based backend, the system routes requests through HTTP to your self-hosted server.

### The LLM Abstraction Layer

In [`src/speech_to_speech/LLM/language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py), the core interface selects the concrete client implementation based on the `--llm_backend` flag. This file handles the instantiation of either local model classes or remote API clients, ensuring the rest of the pipeline remains agnostic to where the LLM inference occurs.

### Backend Registration

The [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py) file maintains the registry mapping string identifiers to handler classes. It registers `responses-api` and `chat-completions` as valid options alongside local backends like `transformers` and `mlx-lm`. When you specify `--llm_backend responses-api`, the registry returns the appropriate client class for OpenAI-style API communication.

### API Configuration Arguments

Configuration for remote servers is defined in [`src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py). This module exposes the CLI flags required to connect to external endpoints:

- `--responses_api_base_url`: The base URL of your vLLM or llama.cpp server
- `--responses_api_api_key`: Authentication header value (can be empty for local servers)
- `--responses_api_stream`: Boolean flag to enable token streaming

## Integration Steps

### Step 1: Launch the vLLM Server

Start your vLLM server with the model you want to use. Ensure it exposes the OpenAI-compatible API on a known port.

```bash
vllm serve gemma-2b-it --host 0.0.0.0 --port 8000

```

### Step 2: Launch the llama.cpp Server

For llama.cpp, use the `llama-server` binary with your GGUF model. The server automatically provides the `/v1/chat/completions` and `/v1/responses` endpoints.

```bash
llama-server -hf ggml-org/gemma-4-E4B-it-GGUF -np 2 -c 65536 -fa on --port 8080

```

### Step 3: Configure Speech-to-Speech

Run the pipeline with the API backend flags. You must provide a placeholder for `--model_name` even though the actual model is determined by the server configuration.

## Complete Configuration Examples

### vLLM Backend Configuration

Connect to a local vLLM instance using the `chat-completions` backend:

```bash
speech-to-speech serve \
    --stt parakeet-tdt \
    --llm_backend chat-completions \
    --model_name dummy \
    --responses_api_base_url http://127.0.0.1:8000/v1 \
    --responses_api_api_key "" \
    --responses_api_stream \
    --tts qwen3

```

### llama.cpp Backend Configuration

Connect to a local llama.cpp server using the `responses-api` backend:

```bash
speech-to-speech serve \
    --stt parakeet-tdt \
    --llm_backend responses-api \
    --model_name dummy \
    --responses_api_base_url http://127.0.0.1:8080/v1 \
    --responses_api_api_key "" \
    --responses_api_stream \
    --tts qwen3

```

Both commands start the full pipeline (VAD → STT → LLM → TTS). The LLM requests route over HTTP to your local server, which returns streaming tokens exactly as the OpenAI Realtime client expects.

## Key Configuration Parameters

- **`--llm_backend`**: Selects the client implementation. Use `responses-api` or `chat-completions` for OpenAI-compatible servers.
- **`--responses_api_base_url`**: The root URL of your server including the `/v1` path prefix (e.g., `http://localhost:8000/v1`).
- **`--responses_api_api_key`**: Set to an empty string `""` for local vLLM or llama.cpp instances that do not require authentication.
- **`--responses_api_stream`**: Enables streaming response handling, which is required for real-time speech-to-speech operation.
- **`--model_name`**: A placeholder value (e.g., `dummy`) required by the argument parser but ignored by the server when using API backends.

## Summary

- **Speech-to-Speech** supports pluggable LLM backends via the `--llm_backend` flag, including `responses-api` and `chat-completions` for external servers.
- **vLLM** and **llama.cpp** expose OpenAI-compatible REST endpoints that work out-of-the-box with these backends.
- Configuration requires setting `--responses_api_base_url` to point at your server and providing a placeholder `--model_name`.
- No source code modifications are necessary; the integration works through the existing abstraction layers in [`language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/language_model.py) and [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py).

## Frequently Asked Questions

### Do I need to modify the Speech-to-Speech source code to use vLLM or llama.cpp?

No. The repository is designed with backend abstraction in mind. By using the `--llm_backend` and `--responses_api_base_url` flags, you can route all LLM traffic to external servers without touching the Python source files.

### Which backend should I use, `responses-api` or `chat-completions`?

Both work with vLLM and llama.cpp. Use `chat-completions` if your server primarily implements the legacy `/v1/chat/completions` endpoint, or `responses-api` if you are targeting the newer OpenAI Realtime-compatible `/v1/responses` endpoint. Most modern llama.cpp builds support both.

### Does the `--model_name` parameter affect which model the server uses?

No. When using API backends, the `--model_name` argument acts only as a placeholder to satisfy the CLI parser. The actual model loaded is determined by the server configuration (e.g., the model you passed to `vllm serve` or `llama-server`).

### Can I use these integrations on Apple Silicon with the MLX backend?

No. The MLX backend (`--llm_backend mlx-lm`) is specifically for local Apple Silicon inference using the MLX framework. To use vLLM or llama.cpp, you must use the API backends (`responses-api` or `chat-completions`) and connect to the servers over HTTP, regardless of whether the server runs on the same machine or remotely.