How to Run Speech-to-Speech Fully Offline with a Local llama.cpp Server

You can run the Hugging Face Speech-to-Speech pipeline completely offline by pointing the LLM stage at a local llama.cpp server using the --responses_api_base_url and --responses_api_api_key flags.

The huggingface/speech-to-speech repository implements a modular, four-stage pipeline for real-time voice conversations. Because the LLM component communicates via an OpenAI-compatible HTTP API, you can replace cloud-based models with a self-hosted llama.cpp instance to run speech-to-speech fully offline with local llama.cpp server infrastructure.

Architecture Overview

The Speech-to-Speech (STS) system processes audio through four interchangeable stages that run in separate threads and communicate via queues:

• Voice Activity Detection (VAD) – detects when a user starts and stops speaking.
• Speech-to-Text (STT) – converts spoken audio into text.
• Large Language Model (LLM) – generates replies or tool calls via an HTTP API.
• Text-to-Speech (TTS) – synthesizes the reply back into audio.

The LLM slot expects an OpenAI-compatible HTTP interface. This design allows you to redirect API calls from hosted services to any local server implementing the same JSON schema—including a llama.cpp inference server—enabling fully offline operation.

Step-by-Step Offline Configuration

1. Start the llama.cpp Server

First, launch llama-server from the llama.cpp repository to expose a local OpenAI-compatible endpoint. The following example uses the Gemma-4 model:

llama-server -hf ggml-org/gemma-4-E4B-it-GGUF \
             -np 2 -c 65536 -fa on --swa-full

This command starts an HTTP service on port 8080 (default) that listens for /v1/completions requests. The server runs entirely on your local machine without requiring external API keys.

2. Configure STS to Use the Local Endpoint

Next, launch the STS server with flags that redirect LLM calls to your local instance. In src/speech_to_speech/cli.py, the parser handles --responses_api_base_url and --responses_api_api_key, passing them to the generic OpenAI-compatible wrapper in src/speech_to_speech/LLM/base_openai_compatible_language_model.py.

Run the following command to connect the pipeline to your local server:

speech-to-speech serve \
    --model_name "ggml-org/gemma-4-E4B-it-GGUF" \
    --responses_api_base_url "http://127.0.0.1:8080/v1" \
    --responses_api_api_key ""

• --model_name specifies the model identifier sent to the llama.cpp server (matching the -hf parameter used when starting llama-server).
• --responses_api_base_url points to the local llama.cpp endpoint.
• --responses_api_api_key is left empty because local servers typically do not require authentication.

3. Connect the Client

With the server running, interact with the pipeline using any OpenAI Realtime-compatible client. The built-in CLI client connects via WebSocket:

speech-to-speech talk --url ws://127.0.0.1:8765/v1/realtime

The client communicates with the STS WebSocket server, which internally routes LLM requests to the local llama.cpp instance. All audio processing—VAD, STT, and TTS—remains on the same machine, ensuring no external network traffic.

How the Integration Works

The integration relies on the backend registry and pipeline orchestration to instantiate the correct handlers. In src/speech_to_speech/backend_registry.py, the --llm_backend option maps to the appropriate handler class. The pipeline controller in src/speech_to_speech/s2s_pipeline.py wires together the VAD, STT, LLM, and TTS components.

When you specify --responses_api_base_url, the base_openai_compatible_language_model.py wrapper builds request payloads exactly like the official OpenAI API, sending them to your local URL instead of cloud endpoints. This transparent compatibility means any server following the OpenAI specification—such as llama.cpp—works without modification to the core pipeline logic.

Summary

• The STS pipeline uses an OpenAI-compatible HTTP API for its LLM stage, making it backend-agnostic.
• llama.cpp server provides a self-hosted implementation of this API via the /v1/completions endpoint.
• Use --responses_api_base_url "http://127.0.0.1:8080/v1" and --responses_api_api_key "" to redirect all LLM traffic to your local instance.
• All four pipeline stages (VAD, STT, LLM, TTS) can run on a single machine without internet access.

Frequently Asked Questions

Do I need an internet connection for any stage of the pipeline?

No. Once you have downloaded the required model files (GGUF format for the LLM and local weights for STT/TTS), the entire pipeline operates offline. The VAD, STT, LLM, and TTS components all run locally, and the LLM communicates with your self-hosted llama.cpp server via localhost.

What model formats does the local llama.cpp server support?

The llama.cpp server supports models in GGUF format. When starting the server with the -hf flag, you can reference Hugging Face repositories containing GGUF files (e.g., ggml-org/gemma-4-E4B-it-GGUF), or load local .gguf files directly from disk.

Can I use other local LLM servers besides llama.cpp?

Yes. Any inference server that implements the OpenAI-compatible /v1/completions endpoint will work. The base_openai_compatible_language_model.py wrapper sends standard HTTP requests, so alternatives like LocalAI, text-generation-webui, or custom implementations are compatible as long as they follow the same JSON schema.

How do I verify the pipeline is running offline?

Monitor your network traffic while the system is active. The speech-to-speech serve process should only open connections to 127.0.0.1:8080 (llama.cpp) and 0.0.0.0:8765 (WebSocket server). No DNS resolution or TCP connections to external IP addresses should occur during inference. You can also physically disconnect from the internet after loading models to confirm continuous operation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}