# How to Set Up a Fully Local Voice Agent with llama.cpp and Gemma 4

> Set up a fully local voice agent with llama.cpp and Gemma 4. Run an offline voice assistant on your hardware, bypassing cloud APIs entirely for complete privacy and control.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-07

---

**Run a complete offline voice assistant on Apple Silicon or x86 by connecting the speech-to-speech pipeline to a local llama.cpp server serving Gemma 4, bypassing cloud APIs entirely.**

The **speech-to-speech** repository from Hugging Face enables a modular **VAD → STT → LLM → TTS** pipeline that can operate entirely on local hardware. By swapping the LLM component for a self-hosted **llama.cpp** server running **Gemma 4**, you create a fully offline voice agent with native audio support. This guide walks through the exact configuration, commands, and source files needed to deploy this stack.

---

## Understanding the Local Pipeline Architecture

The speech-to-speech library orchestrates four stages through threaded queues defined in `src/speech_to_speech/pipeline/*`. For a fully local deployment with Gemma 4, the architecture simplifies to:

- **Microphone → VAD**: Silero VAD detects speech boundaries without cloud dependencies
- **VAD → LLM**: Raw audio segments route directly to llama.cpp's OpenAI-compatible chat-completions endpoint—**no STT transcription step**
- **LLM → TTS**: Generated text streams to the Qwen3-TTS handler
- **TTS → Speakers**: Audio deltas return via WebSocket in OpenAI Realtime-compatible format

This "native audio" path leverages Gemma 4's multimodal capabilities to process audio directly rather than converting to text first.

---

## Installing and Configuring llama.cpp for Gemma 4

Gemma 4 requires a recent llama.cpp build with projector support for native audio processing.

### Step 1: Install llama.cpp

On macOS with Homebrew:

```bash
brew install llama.cpp
brew upgrade llama.cpp   # Critical: ensure Gemma-4 projector support

```

Linux users should build from source following the [llama.cpp repository](https://github.com/ggerganov/llama.cpp) instructions.

### Step 2: Launch the llama.cpp Server

Start `llama-server` with the Gemma 4 GGUF checkpoint and parameters tuned for voice interaction:

```bash
llama-server \
    -hf ggml-org/gemma-4-12B-it-GGUF:Q4_0 \
    -c 16384 \
    -np 1 \
    -fa on \
    --host 127.0.0.1 \
    --port 8080

```

| Flag | Purpose |
|------|---------|
| `-hf` | Downloads the specified GGUF from Hugging Face |
| `-c 16384` | Context window sized for conversation turns |
| `-np 1` | Single parallel sequence (optimizes for latency) |
| `-fa on` | Flash Attention for efficient inference |
| `--host/--port` | Exposes `/v1/chat/completions` locally |

The server now accepts OpenAI-compatible requests at `http://127.0.0.1:8080/v1/chat/completions`.

---

## Configuring the Speech-to-Speech Pipeline

With llama.cpp running, configure the pipeline to bypass cloud STT and point to your local LLM.

### Step 3: Start the Pipeline Server

```bash
uv run speech-to-speech serve \
    --stt none \
    --llm_backend chat-completions \
    --tts qwen3 \
    --model_name "ggml-org/gemma-4-12B-it-GGUF" \
    --responses_api_base_url "http://127.0.0.1:8080/v1" \
    --responses_api_api_key "" \
    --responses_api_audio_content_type input_audio \
    --responses_api_stream \
    --qwen3_tts_mlx_quantization 6bit \
    --min_silence_ms 300

```

### Critical Configuration Flags Explained

| Flag | Value | Why It Matters |
|------|-------|--------------|
| `--stt none` | Disables transcription | Audio flows directly to LLM as WAV bytes |
| `--llm_backend chat-completions` | Uses OpenAI-compatible endpoint | Required for native audio with llama.cpp |
| `--responses_api_base_url` | Local server address | Routes all LLM traffic to `127.0.0.1:8080` |
| `--responses_api_audio_content_type` | `input_audio` | Signals raw WAV payload format |
| `--responses_api_stream` | Enables streaming | Delivers TTS chunks without waiting for full response |

The `--model_name` flag must match the Gemma 4 identifier so the pipeline constructs correct API calls.

---

## Connecting a Client Interface

### Step 4a: Browser Demo (Recommended)

Launch the packaged web interface:

```bash
SPEECH_TO_SPEECH_URL="ws://localhost:8765/v1/realtime" \
uv run --with-requirements demo/requirements.txt \
    uvicorn --app-dir demo server:app --port 7860

```

Then open `http://localhost:7860` in your browser to start talking.

### Step 4b: Headless CLI Client

For terminal-only operation:

```bash
speech-to-speech talk --url ws://127.0.0.1:8765/v1/realtime

```

---

## Key Source Files and Their Roles

Understanding these files helps debug or extend your local setup:

| File | Function |
|------|----------|
| [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) | Main orchestrator that instantiates handlers based on `--llm_backend`; routes VAD output to the selected LLM module |
| [`src/speech_to_speech/LLM/chat_completions_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/chat_completions_language_model.py) | HTTP client implementing OpenAI chat-completions protocol; formats audio payloads for llama.cpp consumption |
| [`src/speech_to_speech/arguments_classes/chat_completions_language_model_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/chat_completions_language_model_arguments.py) | CLI argument definitions including `--responses_api_base_url` and audio content type options |
| [`examples/gemma4-12b-macos/README.md`](https://github.com/huggingface/speech-to-speech/blob/main/examples/gemma4-12b-macos/README.md) | Complete working example with macOS-specific tuning |
| [`README.md`](https://github.com/huggingface/speech-to-speech/blob/main/README.md) (Fully Local section) | Overview with alternate quantization options and performance tips |

---

## Performance Optimization for Local Deployment

### Apple Silicon Specifics

The [`examples/gemma4-12b-macos/README.md`](https://github.com/huggingface/speech-to-speech/blob/main/examples/gemma4-12b-macos/README.md) demonstrates MLX-optimized TTS through `--qwen3_tts_mlx_quantization`. Values of `4bit`, `6bit`, or `8bit` trade quality for speed.

### Context and Latency Trade-offs

| Scenario | Recommended `-c` Value |
|----------|------------------------|
| Short exchanges (Q&A) | 4096-8192 |
| Multi-turn conversations | 16384 (as shown) |
| Long-form narration | 32768+ with quantized KV cache |

Reduce `--min_silence_ms` below 300ms for snappier responses, or increase it to prevent mid-sentence interruptions.

---

## Troubleshooting Common Issues

**"Connection refused" to llama.cpp**
- Verify `llama-server` is listening: `curl http://127.0.0.1:8080/v1/models`
- Check firewall rules for localhost binding

**Audio not reaching the LLM**
- Confirm `--stt none` is set; any STT backend intercepts audio before the LLM
- Validate `--responses_api_audio_content_type input_audio` matches llama.cpp's expected format

**Gemma 4 projector errors**
- Upgrade llama.cpp: `brew upgrade llama.cpp` or rebuild from latest source
- Ensure the GGUF includes the multimodal projector: `ggml-org/gemma-4-12B-it-GGUF` variants

---

## Summary

- **llama.cpp** exposes Gemma 4 through an OpenAI-compatible local server using `llama-server -hf ggml-org/gemma-4-12B-it-GGUF`
- **`--stt none`** and **`--llm_backend chat-completions`** configure the speech-to-speech pipeline for native audio routing
- **`--responses_api_audio_content_type input_audio`** ensures raw WAV bytes flow directly to the multimodal LLM
- All components execute in threaded queues via `src/speech_to_speech/pipeline/*` with no external API dependencies

---

## Frequently Asked Questions

### How does native audio differ from traditional STT→LLM→TTS pipelines?

Traditional pipelines convert speech to text with a dedicated model (like Whisper), send text to the LLM, then synthesize response text to speech. Native audio skips the transcription step entirely—the multimodal Gemma 4 model processes raw audio waveforms directly and emits text responses, reducing latency and propagation errors.

### Can I use a different quantization than Q4_0 for Gemma 4?

Yes. The `llama-server -hf` flag accepts any GGUF variant in the `ggml-org/gemma-4-12B-it-GGUF` repository. Q5_K_M and Q6_K offer better quality at slower speeds; Q3_K_S reduces memory for edge devices. Match the `--model_name` in your pipeline command to the selected quantization.

### Why must I disable STT with `--stt none` for native audio?

The speech-to-speech pipeline normally transcribes audio before LLM processing. With native audio capabilities, Gemma 4 ingests raw WAV bytes directly—transcription would duplicate effort and strip prosodic information the multimodal model uses. Setting `--stt none` routes VAD output straight to the chat-completions handler in [`chat_completions_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/chat_completions_language_model.py).

### Does this work on Linux or Windows, or only macOS?

The pipeline and llama.cpp are cross-platform. The [`examples/gemma4-12b-macos/README.md`](https://github.com/huggingface/speech-to-speech/blob/main/examples/gemma4-12b-macos/README.md) provides Apple Silicon optimizations, but the same commands function on Linux with llama.cpp built from source. Windows users can run through WSL2 or native builds, though audio device handling may require additional configuration.