# How to Set Up Local Inference with llama.cpp or vLLM in Hugging Face Speech-to-Speech

> Set up local inference with llama.cpp or vLLM for Hugging Face Speech-to-Speech. Configure the responses_api_base_url for fully local LLM inference.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-06

---

**Configure the --responses_api_base_url flag to point at a self-hosted OpenAI-compatible server running llama.cpp or vLLM, then launch the pipeline with an empty or dummy API key for fully local LLM inference.**

The Hugging Face **speech-to-speech** repository provides an end-to-end voice conversation pipeline that decouples its language model (LLM) stage from any specific provider. Because the pipeline communicates through an **OpenAI-compatible API layer**, you can replace remote API calls with **local inference** by running your own LLM server. This guide walks through configuring the pipeline to use **llama.cpp** or **vLLM** as the backend, complete with the exact CLI flags and implementation details from the source code.

## Architecture: Why Local Inference Works

The speech-to-speech pipeline follows a four-stage design: **VAD → STT → LLM → TTS**, with each component operating in its own thread and passing data through queues. The LLM stage does not embed a model directly; instead, it sends HTTP requests to an external endpoint.

This abstraction lives in two model handler classes:

- `ResponsesApiModelHandler` in [`src/speech_to_speech/api/openai_responses/model_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_responses/model_handler.py)
- `ChatCompletionsApiModelHandler` in [`src/speech_to_speech/api/openai_chat_completions/model_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_chat_completions/model_handler.py)

Both handlers construct JSON payloads conforming to the OpenAI schema and forward them through a generic HTTP client defined in [`src/speech_to_speech/api/openai_responses/client.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_responses/client.py). The pipeline therefore accepts **any server** implementing `/v1/responses` or `/v1/chat/completions` — including locally-hosted llama.cpp and vLLM instances.

## CLI Flags That Control the LLM Connection

The connection parameters are exposed through argument classes in the source tree:

| File | Purpose |
|------|---------|
| [`src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py) | Defines `--responses_api_base_url`, `--responses_api_api_key`, `--responses_api_stream`, and `--responses_api_audio_content_type` for the Responses API backend |
| [`src/speech_to_speech/arguments_classes/chat_completions_language_model_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/chat_completions_language_model_arguments.py) | Defines equivalent flags for the Chat-Completions backend |

These flags are **shared across backends** — you use the same names regardless of whether you select `--llm_backend responses-api` or `--llm_backend chat-completions`.

## Setting Up llama.cpp for Local Inference

**llama.cpp** exposes an OpenAI-compatible server through the `llama-server` binary. When started, it binds to port 8080 by default and accepts requests at `/v1/chat/completions` or `/v1/responses`.

### Step 1: Start the llama.cpp Server

```bash
llama-server \
    -hf ggml-org/gemma-4-E4B-it-GGUF \
    -np 2 \
    -c 65536 \
    -fa on \
    --swa-full

```

- `-hf` loads a Hugging Face GGUF model
- `-np 2` permits parallel processing of two prompts
- `-c 65536` sets the context size
- `-fa on` enables FlashAttention
- `--swa-full` uses sliding window attention for the full context

### Step 2: Launch Speech-to-Speech Against the Local Endpoint

```bash
speech-to-speech \
    --mode realtime \
    --stt parakeet-tdt \
    --llm_backend responses-api \
    --tts qwen3 \
    --model_name "ggml-org/gemma-4-E4B-it-GGUF" \
    --responses_api_base_url "http://127.0.0.1:8080/v1" \
    --responses_api_api_key "" \
    --responses_api_stream \
    --enable_live_transcription

```

**Critical detail**: Pass an **empty string** for `--responses_api_api_key`. The llama.cpp server does not authenticate requests, and the pipeline requires this parameter to be present even when unused.

## Setting Up vLLM for Local Inference

**vLLM** similarly exposes an OpenAI-compatible API through its `api_server` entrypoint. The default port is 8000.

### Step 1: Start the vLLM Server

```bash
docker run -d -p 8000:8000 ghcr.io/vllm/vllm:latest \
    python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen3-4B-Instruct-2507 \
    --port 8000

```

For GPU acceleration, add `--gpus all` to the Docker command and ensure the NVIDIA Container Toolkit is installed.

### Step 2: Launch Speech-to-Speech Against vLLM

```bash
speech-to-speech \
    --mode realtime \
    --stt parakeet-tdt \
    --llm_backend chat-completions \
    --tts qwen3 \
    --model_name "Qwen/Qwen3-4B-Instruct-2507" \
    --responses_api_base_url "http://localhost:8000/v1" \
    --responses_api_api_key "any-string" \
    --responses_api_stream

```

vLLM does not validate the API key by default, so any string (or omission) suffices.

## Handling Audio Input in Local Mode

The pipeline supports **direct audio input** to the LLM, bypassing speech-to-text entirely. This uses the `input_audio` parameter in the OpenAI schema and requires the `--responses_api_audio_content_type` flag.

Enable this mode with:

```bash
speech-to-speech \
    --mode realtime \
    --stt none \
    --llm_backend chat-completions \
    --model_name "gpt-audio-1.5" \
    --responses_api_base_url "http://127.0.0.1:8080/v1" \
    --responses_api_api_key "" \
    --responses_api_audio_content_type input_audio

```

The model handler converts internal `Chat` objects to the correct payload shape — including base64-encoded WAV data for `input_audio` or data-URLs for `audio_url` — as verified in the test suite under `test_chat_messages_converts_audio_to_llama_cpp_shape`.

## Docker Compose for Automated Local Setup

The repository includes a [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml) that automates llama.cpp server startup. Review this file to understand the environment variables and volume mounts required for persistent model caching:

- [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml) — orchestrates the llama.cpp server alongside the pipeline containers

## Backend Selection: responses-api vs. chat-completions

| Backend | Endpoint | Best For |
|---------|----------|----------|
| `responses-api` | `/v1/responses` | Native audio-in/audio-out models, newer OpenAI-compatible servers |
| `chat-completions` | `/v1/chat/completions` | Maximum compatibility, vLLM, older llama.cpp builds, text-only fallback |

Both backends use identical connection flags. Select based on what your local server implements.

## Summary

- **Speech-to-speech decouples the LLM stage** through an OpenAI-compatible HTTP client, enabling local inference by changing two flags
- **Point `--responses_api_base_url` at your server** (`http://127.0.0.1:8080/v1` for llama.cpp, `http://localhost:8000/v1` for vLLM)
- **Supply any value for `--responses_api_api_key`** — empty for llama.cpp, dummy string for vLLM
- **Use `--responses_api_audio_content_type input_audio`** to send raw audio directly to multimodal models
- **Reference files**: [`responses_api_language_model_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/responses_api_language_model_arguments.py), [`chat_completions_language_model_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/chat_completions_language_model_arguments.py), and the respective [`model_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/model_handler.py) modules implement this behavior

## Frequently Asked Questions

### Does speech-to-speech require an internet connection for the LLM stage?

No. Once configured with `--responses_api_base_url` pointing to a local server, all LLM inference happens on your hardware. The only network requirement is initial model download if using Docker or Hugging Face Hub loaders.

### Can I use quantized models with this setup?

Yes. llama.cpp exclusively runs quantized GGUF models, and vLLM supports AWQ, GPTQ, and FP8 quantization. Pass the Hugging Face model identifier or local path to whichever server you choose, then reference the same name in `--model_name` for logging purposes.

### Why does the pipeline use `--responses_api_*` flags for both backends?

The argument classes share flag names for backward compatibility and ease of switching. The `responses-api` and `chat-completions` values for `--llm_backend` determine which handler class processes your requests, but the connection parameters remain identical.

### Is streaming supported with local servers?

Yes. Include `--responses_api_stream` to enable Server-Sent Events. Both llama.cpp and vLLM implement the OpenAI streaming format, and the pipeline's client consumes chunks incrementally through [`src/speech_to_speech/api/openai_responses/client.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_responses/client.py).