How to Set Up Local Inference with llama.cpp or vLLM in Hugging Face Speech-to-Speech

Configure the --responses_api_base_url flag to point at a self-hosted OpenAI-compatible server running llama.cpp or vLLM, then launch the pipeline with an empty or dummy API key for fully local LLM inference.

The Hugging Face speech-to-speech repository provides an end-to-end voice conversation pipeline that decouples its language model (LLM) stage from any specific provider. Because the pipeline communicates through an OpenAI-compatible API layer, you can replace remote API calls with local inference by running your own LLM server. This guide walks through configuring the pipeline to use llama.cpp or vLLM as the backend, complete with the exact CLI flags and implementation details from the source code.

Architecture: Why Local Inference Works

The speech-to-speech pipeline follows a four-stage design: VAD → STT → LLM → TTS, with each component operating in its own thread and passing data through queues. The LLM stage does not embed a model directly; instead, it sends HTTP requests to an external endpoint.

This abstraction lives in two model handler classes:

• ResponsesApiModelHandler in src/speech_to_speech/api/openai_responses/model_handler.py
• ChatCompletionsApiModelHandler in src/speech_to_speech/api/openai_chat_completions/model_handler.py

Both handlers construct JSON payloads conforming to the OpenAI schema and forward them through a generic HTTP client defined in src/speech_to_speech/api/openai_responses/client.py. The pipeline therefore accepts any server implementing /v1/responses or /v1/chat/completions — including locally-hosted llama.cpp and vLLM instances.

CLI Flags That Control the LLM Connection

The connection parameters are exposed through argument classes in the source tree:

File	Purpose
src/speech_to_speech/arguments_classes/responses_api_language_model_arguments.py	Defines --responses_api_base_url, --responses_api_api_key, --responses_api_stream, and --responses_api_audio_content_type for the Responses API backend
src/speech_to_speech/arguments_classes/chat_completions_language_model_arguments.py	Defines equivalent flags for the Chat-Completions backend

These flags are shared across backends — you use the same names regardless of whether you select --llm_backend responses-api or --llm_backend chat-completions.

Setting Up llama.cpp for Local Inference

llama.cpp exposes an OpenAI-compatible server through the llama-server binary. When started, it binds to port 8080 by default and accepts requests at /v1/chat/completions or /v1/responses.

Step 1: Start the llama.cpp Server

llama-server \
    -hf ggml-org/gemma-4-E4B-it-GGUF \
    -np 2 \
    -c 65536 \
    -fa on \
    --swa-full

• -hf loads a Hugging Face GGUF model
• -np 2 permits parallel processing of two prompts
• -c 65536 sets the context size
• -fa on enables FlashAttention
• --swa-full uses sliding window attention for the full context

Step 2: Launch Speech-to-Speech Against the Local Endpoint

speech-to-speech \
    --mode realtime \
    --stt parakeet-tdt \
    --llm_backend responses-api \
    --tts qwen3 \
    --model_name "ggml-org/gemma-4-E4B-it-GGUF" \
    --responses_api_base_url "http://127.0.0.1:8080/v1" \
    --responses_api_api_key "" \
    --responses_api_stream \
    --enable_live_transcription

Critical detail: Pass an empty string for --responses_api_api_key. The llama.cpp server does not authenticate requests, and the pipeline requires this parameter to be present even when unused.

Setting Up vLLM for Local Inference

vLLM similarly exposes an OpenAI-compatible API through its api_server entrypoint. The default port is 8000.

Step 1: Start the vLLM Server

docker run -d -p 8000:8000 ghcr.io/vllm/vllm:latest \
    python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen3-4B-Instruct-2507 \
    --port 8000

For GPU acceleration, add --gpus all to the Docker command and ensure the NVIDIA Container Toolkit is installed.

Step 2: Launch Speech-to-Speech Against vLLM

speech-to-speech \
    --mode realtime \
    --stt parakeet-tdt \
    --llm_backend chat-completions \
    --tts qwen3 \
    --model_name "Qwen/Qwen3-4B-Instruct-2507" \
    --responses_api_base_url "http://localhost:8000/v1" \
    --responses_api_api_key "any-string" \
    --responses_api_stream

vLLM does not validate the API key by default, so any string (or omission) suffices.

Handling Audio Input in Local Mode

The pipeline supports direct audio input to the LLM, bypassing speech-to-text entirely. This uses the input_audio parameter in the OpenAI schema and requires the --responses_api_audio_content_type flag.

Enable this mode with:

speech-to-speech \
    --mode realtime \
    --stt none \
    --llm_backend chat-completions \
    --model_name "gpt-audio-1.5" \
    --responses_api_base_url "http://127.0.0.1:8080/v1" \
    --responses_api_api_key "" \
    --responses_api_audio_content_type input_audio

The model handler converts internal Chat objects to the correct payload shape — including base64-encoded WAV data for input_audio or data-URLs for audio_url — as verified in the test suite under test_chat_messages_converts_audio_to_llama_cpp_shape.

Docker Compose for Automated Local Setup

The repository includes a docker-compose.yml that automates llama.cpp server startup. Review this file to understand the environment variables and volume mounts required for persistent model caching:

• docker-compose.yml — orchestrates the llama.cpp server alongside the pipeline containers

Backend Selection: responses-api vs. chat-completions

Backend	Endpoint	Best For
responses-api	/v1/responses	Native audio-in/audio-out models, newer OpenAI-compatible servers
chat-completions	/v1/chat/completions	Maximum compatibility, vLLM, older llama.cpp builds, text-only fallback

Both backends use identical connection flags. Select based on what your local server implements.

Summary

• Speech-to-speech decouples the LLM stage through an OpenAI-compatible HTTP client, enabling local inference by changing two flags
• Point --responses_api_base_url at your server (http://127.0.0.1:8080/v1 for llama.cpp, http://localhost:8000/v1 for vLLM)
• Supply any value for --responses_api_api_key — empty for llama.cpp, dummy string for vLLM
• Use --responses_api_audio_content_type input_audio to send raw audio directly to multimodal models
• Reference files: responses_api_language_model_arguments.py, chat_completions_language_model_arguments.py, and the respective model_handler.py modules implement this behavior

Frequently Asked Questions

Does speech-to-speech require an internet connection for the LLM stage?

No. Once configured with --responses_api_base_url pointing to a local server, all LLM inference happens on your hardware. The only network requirement is initial model download if using Docker or Hugging Face Hub loaders.

Can I use quantized models with this setup?

Yes. llama.cpp exclusively runs quantized GGUF models, and vLLM supports AWQ, GPTQ, and FP8 quantization. Pass the Hugging Face model identifier or local path to whichever server you choose, then reference the same name in --model_name for logging purposes.

Why does the pipeline use --responses_api_* flags for both backends?

The argument classes share flag names for backward compatibility and ease of switching. The responses-api and chat-completions values for --llm_backend determine which handler class processes your requests, but the connection parameters remain identical.

Is streaming supported with local servers?

Yes. Include --responses_api_stream to enable Server-Sent Events. Both llama.cpp and vLLM implement the OpenAI streaming format, and the pipeline's client consumes chunks incrementally through src/speech_to_speech/api/openai_responses/client.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}