# How to Run Speech-to-Speech Fully Offline with a Local llama.cpp Server

> Learn to run speech-to-speech fully offline with a local llama.cpp server. Control Hugging Face’s pipeline locally using simple flags for enhanced privacy and speed.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-09

---

**You can run the Hugging Face Speech-to-Speech pipeline completely offline by pointing the LLM stage at a local llama.cpp server using the `--responses_api_base_url` and `--responses_api_api_key` flags.**

The huggingface/speech-to-speech repository implements a modular, four-stage pipeline for real-time voice conversations. Because the LLM component communicates via an OpenAI-compatible HTTP API, you can replace cloud-based models with a self-hosted llama.cpp instance to run speech-to-speech fully offline with local llama.cpp server infrastructure.

## Architecture Overview

The Speech-to-Speech (STS) system processes audio through four interchangeable stages that run in separate threads and communicate via queues:

- **Voice Activity Detection (VAD)** – detects when a user starts and stops speaking.
- **Speech-to-Text (STT)** – converts spoken audio into text.
- **Large Language Model (LLM)** – generates replies or tool calls via an HTTP API.
- **Text-to-Speech (TTS)** – synthesizes the reply back into audio.

The LLM slot expects an **OpenAI-compatible** HTTP interface. This design allows you to redirect API calls from hosted services to any local server implementing the same JSON schema—including a llama.cpp inference server—enabling fully offline operation.

## Step-by-Step Offline Configuration

### 1. Start the llama.cpp Server

First, launch `llama-server` from the llama.cpp repository to expose a local OpenAI-compatible endpoint. The following example uses the Gemma-4 model:

```bash
llama-server -hf ggml-org/gemma-4-E4B-it-GGUF \
             -np 2 -c 65536 -fa on --swa-full

```

This command starts an HTTP service on port `8080` (default) that listens for `/v1/completions` requests. The server runs entirely on your local machine without requiring external API keys.

### 2. Configure STS to Use the Local Endpoint

Next, launch the STS server with flags that redirect LLM calls to your local instance. In [`src/speech_to_speech/cli.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/cli.py), the parser handles `--responses_api_base_url` and `--responses_api_api_key`, passing them to the generic OpenAI-compatible wrapper in [`src/speech_to_speech/LLM/base_openai_compatible_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/base_openai_compatible_language_model.py).

Run the following command to connect the pipeline to your local server:

```bash
speech-to-speech serve \
    --model_name "ggml-org/gemma-4-E4B-it-GGUF" \
    --responses_api_base_url "http://127.0.0.1:8080/v1" \
    --responses_api_api_key ""

```

- `--model_name` specifies the model identifier sent to the llama.cpp server (matching the `-hf` parameter used when starting `llama-server`).
- `--responses_api_base_url` points to the local llama.cpp endpoint.
- `--responses_api_api_key` is left empty because local servers typically do not require authentication.

### 3. Connect the Client

With the server running, interact with the pipeline using any OpenAI Realtime-compatible client. The built-in CLI client connects via WebSocket:

```bash
speech-to-speech talk --url ws://127.0.0.1:8765/v1/realtime

```

The client communicates with the STS WebSocket server, which internally routes LLM requests to the local llama.cpp instance. All audio processing—VAD, STT, and TTS—remains on the same machine, ensuring no external network traffic.

## How the Integration Works

The integration relies on the backend registry and pipeline orchestration to instantiate the correct handlers. In [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py), the `--llm_backend` option maps to the appropriate handler class. The pipeline controller in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) wires together the VAD, STT, LLM, and TTS components.

When you specify `--responses_api_base_url`, the [`base_openai_compatible_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/base_openai_compatible_language_model.py) wrapper builds request payloads exactly like the official OpenAI API, sending them to your local URL instead of cloud endpoints. This transparent compatibility means any server following the OpenAI specification—such as llama.cpp—works without modification to the core pipeline logic.

## Summary

- The STS pipeline uses an **OpenAI-compatible HTTP API** for its LLM stage, making it backend-agnostic.
- **llama.cpp server** provides a self-hosted implementation of this API via the `/v1/completions` endpoint.
- Use **`--responses_api_base_url "http://127.0.0.1:8080/v1"`** and **`--responses_api_api_key ""`** to redirect all LLM traffic to your local instance.
- All four pipeline stages (VAD, STT, LLM, TTS) can run on a single machine without internet access.

## Frequently Asked Questions

### Do I need an internet connection for any stage of the pipeline?

No. Once you have downloaded the required model files (GGUF format for the LLM and local weights for STT/TTS), the entire pipeline operates offline. The VAD, STT, LLM, and TTS components all run locally, and the LLM communicates with your self-hosted llama.cpp server via localhost.

### What model formats does the local llama.cpp server support?

The llama.cpp server supports models in **GGUF** format. When starting the server with the `-hf` flag, you can reference Hugging Face repositories containing GGUF files (e.g., `ggml-org/gemma-4-E4B-it-GGUF`), or load local `.gguf` files directly from disk.

### Can I use other local LLM servers besides llama.cpp?

Yes. Any inference server that implements the OpenAI-compatible `/v1/completions` endpoint will work. The [`base_openai_compatible_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/base_openai_compatible_language_model.py) wrapper sends standard HTTP requests, so alternatives like **LocalAI**, **text-generation-webui**, or custom implementations are compatible as long as they follow the same JSON schema.

### How do I verify the pipeline is running offline?

Monitor your network traffic while the system is active. The `speech-to-speech serve` process should only open connections to `127.0.0.1:8080` (llama.cpp) and `0.0.0.0:8765` (WebSocket server). No DNS resolution or TCP connections to external IP addresses should occur during inference. You can also physically disconnect from the internet after loading models to confirm continuous operation.