How to Set Up a Fully Local Voice Agent with llama.cpp and Gemma 4

Run a complete offline voice assistant on Apple Silicon or x86 by connecting the speech-to-speech pipeline to a local llama.cpp server serving Gemma 4, bypassing cloud APIs entirely.

The speech-to-speech repository from Hugging Face enables a modular VAD → STT → LLM → TTS pipeline that can operate entirely on local hardware. By swapping the LLM component for a self-hosted llama.cpp server running Gemma 4, you create a fully offline voice agent with native audio support. This guide walks through the exact configuration, commands, and source files needed to deploy this stack.

---

Understanding the Local Pipeline Architecture

The speech-to-speech library orchestrates four stages through threaded queues defined in src/speech_to_speech/pipeline/*. For a fully local deployment with Gemma 4, the architecture simplifies to:

• Microphone → VAD: Silero VAD detects speech boundaries without cloud dependencies
• VAD → LLM: Raw audio segments route directly to llama.cpp's OpenAI-compatible chat-completions endpoint—no STT transcription step
• LLM → TTS: Generated text streams to the Qwen3-TTS handler
• TTS → Speakers: Audio deltas return via WebSocket in OpenAI Realtime-compatible format

This "native audio" path leverages Gemma 4's multimodal capabilities to process audio directly rather than converting to text first.

---

Installing and Configuring llama.cpp for Gemma 4

Gemma 4 requires a recent llama.cpp build with projector support for native audio processing.

Step 1: Install llama.cpp

On macOS with Homebrew:

brew install llama.cpp
brew upgrade llama.cpp   # Critical: ensure Gemma-4 projector support

Linux users should build from source following the llama.cpp repository instructions.

Step 2: Launch the llama.cpp Server

Start llama-server with the Gemma 4 GGUF checkpoint and parameters tuned for voice interaction:

llama-server \
    -hf ggml-org/gemma-4-12B-it-GGUF:Q4_0 \
    -c 16384 \
    -np 1 \
    -fa on \
    --host 127.0.0.1 \
    --port 8080

Flag	Purpose
-hf	Downloads the specified GGUF from Hugging Face
-c 16384	Context window sized for conversation turns
-np 1	Single parallel sequence (optimizes for latency)
-fa on	Flash Attention for efficient inference
--host/--port	Exposes /v1/chat/completions locally

The server now accepts OpenAI-compatible requests at http://127.0.0.1:8080/v1/chat/completions.

---

Configuring the Speech-to-Speech Pipeline

With llama.cpp running, configure the pipeline to bypass cloud STT and point to your local LLM.

Step 3: Start the Pipeline Server

uv run speech-to-speech serve \
    --stt none \
    --llm_backend chat-completions \
    --tts qwen3 \
    --model_name "ggml-org/gemma-4-12B-it-GGUF" \
    --responses_api_base_url "http://127.0.0.1:8080/v1" \
    --responses_api_api_key "" \
    --responses_api_audio_content_type input_audio \
    --responses_api_stream \
    --qwen3_tts_mlx_quantization 6bit \
    --min_silence_ms 300

Critical Configuration Flags Explained

Flag	Value	Why It Matters
--stt none	Disables transcription	Audio flows directly to LLM as WAV bytes
--llm_backend chat-completions	Uses OpenAI-compatible endpoint	Required for native audio with llama.cpp
--responses_api_base_url	Local server address	Routes all LLM traffic to 127.0.0.1:8080
--responses_api_audio_content_type	input_audio	Signals raw WAV payload format
--responses_api_stream	Enables streaming	Delivers TTS chunks without waiting for full response

The --model_name flag must match the Gemma 4 identifier so the pipeline constructs correct API calls.

---

Connecting a Client Interface

Step 4a: Browser Demo (Recommended)

Launch the packaged web interface:

SPEECH_TO_SPEECH_URL="ws://localhost:8765/v1/realtime" \
uv run --with-requirements demo/requirements.txt \
    uvicorn --app-dir demo server:app --port 7860

Then open http://localhost:7860 in your browser to start talking.

Step 4b: Headless CLI Client

For terminal-only operation:

speech-to-speech talk --url ws://127.0.0.1:8765/v1/realtime

---

Key Source Files and Their Roles

Understanding these files helps debug or extend your local setup:

File	Function
src/speech_to_speech/s2s_pipeline.py	Main orchestrator that instantiates handlers based on --llm_backend; routes VAD output to the selected LLM module
src/speech_to_speech/LLM/chat_completions_language_model.py	HTTP client implementing OpenAI chat-completions protocol; formats audio payloads for llama.cpp consumption
src/speech_to_speech/arguments_classes/chat_completions_language_model_arguments.py	CLI argument definitions including --responses_api_base_url and audio content type options
examples/gemma4-12b-macos/README.md	Complete working example with macOS-specific tuning
README.md (Fully Local section)	Overview with alternate quantization options and performance tips

---

Performance Optimization for Local Deployment

Apple Silicon Specifics

The examples/gemma4-12b-macos/README.md demonstrates MLX-optimized TTS through --qwen3_tts_mlx_quantization. Values of 4bit, 6bit, or 8bit trade quality for speed.

Context and Latency Trade-offs

Scenario	Recommended -c Value
Short exchanges (Q&A)	4096-8192
Multi-turn conversations	16384 (as shown)
Long-form narration	32768+ with quantized KV cache

Reduce --min_silence_ms below 300ms for snappier responses, or increase it to prevent mid-sentence interruptions.

---

Troubleshooting Common Issues

"Connection refused" to llama.cpp

• Verify llama-server is listening: curl http://127.0.0.1:8080/v1/models
• Check firewall rules for localhost binding

Audio not reaching the LLM

• Confirm --stt none is set; any STT backend intercepts audio before the LLM
• Validate --responses_api_audio_content_type input_audio matches llama.cpp's expected format

Gemma 4 projector errors

• Upgrade llama.cpp: brew upgrade llama.cpp or rebuild from latest source
• Ensure the GGUF includes the multimodal projector: ggml-org/gemma-4-12B-it-GGUF variants

---

Summary

• llama.cpp exposes Gemma 4 through an OpenAI-compatible local server using llama-server -hf ggml-org/gemma-4-12B-it-GGUF
• --stt none and --llm_backend chat-completions configure the speech-to-speech pipeline for native audio routing
• --responses_api_audio_content_type input_audio ensures raw WAV bytes flow directly to the multimodal LLM
• All components execute in threaded queues via src/speech_to_speech/pipeline/* with no external API dependencies

---

Frequently Asked Questions

How does native audio differ from traditional STT→LLM→TTS pipelines?

Traditional pipelines convert speech to text with a dedicated model (like Whisper), send text to the LLM, then synthesize response text to speech. Native audio skips the transcription step entirely—the multimodal Gemma 4 model processes raw audio waveforms directly and emits text responses, reducing latency and propagation errors.

Can I use a different quantization than Q4_0 for Gemma 4?

Yes. The llama-server -hf flag accepts any GGUF variant in the ggml-org/gemma-4-12B-it-GGUF repository. Q5_K_M and Q6_K offer better quality at slower speeds; Q3_K_S reduces memory for edge devices. Match the --model_name in your pipeline command to the selected quantization.

Why must I disable STT with --stt none for native audio?

The speech-to-speech pipeline normally transcribes audio before LLM processing. With native audio capabilities, Gemma 4 ingests raw WAV bytes directly—transcription would duplicate effort and strip prosodic information the multimodal model uses. Setting --stt none routes VAD output straight to the chat-completions handler in chat_completions_language_model.py.

Does this work on Linux or Windows, or only macOS?

The pipeline and llama.cpp are cross-platform. The examples/gemma4-12b-macos/README.md provides Apple Silicon optimizations, but the same commands function on Linux with llama.cpp built from source. Windows users can run through WSL2 or native builds, though audio device handling may require additional configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}