How to Set Up a Docker Compose Environment with Local LLM Server for Speech-to-Speech

Use the pre-configured docker-compose.yml in the huggingface/speech-to-speech repository to launch a local LLM backend alongside the speech-to-speech pipeline with GPU support out of the box.

The speech-to-speech repository by Hugging Face provides a production-ready Docker Compose configuration that eliminates manual setup. According to the source code in docker-compose.yml, the stack deploys two interconnected services: a GPU-accelerated llama.cpp server for local LLM inference and the speech-to-speech pipeline that routes all generation requests to this backend.

Architecture Overview

The Docker Compose environment orchestrates two specialized containers:

Service	Purpose	Key Implementation Details
llama	Local LLM server	Runs ghcr.io/ggml-org/llama.cpp:server-cuda with OpenAI-compatible API on port 8080
pipeline	Speech-to-speech application	Custom build from repository Dockerfile, exposes public API on port 8765

Both services mount a shared ./cache/ volume at /root/.cache/ for model persistence and request NVIDIA GPU access through identical deploy.resources.reservations.devices configurations.

Prerequisites

Before deploying, verify your environment meets these requirements:

• Docker Engine 20.10+ with NVIDIA Container Toolkit installed
• Compatible NVIDIA driver for the CUDA runtime in the llama.cpp image
• At least 4 GB free disk space for the default GGUF model download
• Git for cloning the repository

Step-by-Step Setup

Follow these commands to launch your local speech-to-speech environment with integrated LLM backend:

# 1. Clone the repository

git clone https://github.com/huggingface/speech-to-speech.git
cd speech-to-speech

# 2. (Optional) Customize configuration

# Edit docker-compose.yml to change model_id or GPU device_ids

# 3. Pull the GPU-enabled LLM image

docker pull ghcr.io/ggml-org/llama.cpp:server-cuda

# 4. Build and start all services

docker compose up --build -d

# 5. Verify LLM server health

curl http://localhost:8080/v1/models

# 6. Test the complete speech-to-speech pipeline

curl -X POST http://localhost:8765/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"ggml-org/gemma-4-E4B-it-GGUF","messages":[{"role":"user","content":"Hello!"}]}'

Service Configuration Details

LLM Service (llama)

As defined in docker-compose.yml lines 5-31, the LLM container uses this configuration:

services:
  llama:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda
    command:
      - -hf
      - ggml-org/gemma-4-E4B-it-GGUF
      - -np
      - "2"
      - -c
      - "65536"
      - -fa
      - "on"
      - --swa-full
      - --host
      - 0.0.0.0
      - --port
      - "8080"
    ports:
      - 8080:8080/tcp
    volumes:
      - ./cache/:/root/.cache/
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]

Key parameters:

• -hf ggml-org/gemma-4-E4B-it-GGUF — Downloads and serves the Gemma 4B instruction-tuned model
• -c 65536 — Allocates 65,536 tokens of context window
• -np 2 — Enables 2 parallel completion slots
• device_ids: ['0'] — Binds to first available GPU (modify for multi-GPU systems)

Pipeline Service

The speech-to-speech container, configured in docker-compose.yml lines 32-70, connects to the LLM backend internally:

  pipeline:
    build:
      context: .
      dockerfile: ${DOCKERFILE:-Dockerfile}
    command:
      - speech-to-speech
      - serve
      - --host
      - 0.0.0.0
      - --port
      - "8765"
      - --llm_backend
      - responses-api
      - --model_name
      - ggml-org/gemma-4-E4B-it-GGUF
      - --responses_api_base_url
      - http://llama:8080/v1
    ports:
      - 8765:8765/tcp
    volumes:
      - ./cache/:/root/.cache/
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]

The --responses_api_base_url http://llama:8080/v1 argument routes all LLM calls to the internal Docker network address, keeping LLM traffic isolated from the host network while exposing only the speech-to-speech API endpoint.

How the Components Interact

Understanding the data flow helps with troubleshooting and customization:

1. Model caching — On first startup, the llama service downloads the GGUF model to ./cache/. Subsequent restarts reuse cached weights.

2. Internal networking — The pipeline service resolves http://llama:8080/v1 through Docker's embedded DNS, forwarding all generation requests to the local LLM.

3. Command entry point — In src/speech_to_speech/cli.py (lines 156-166), the speech-to-speech serve command implements the server logic invoked by the container's command array.

4. GPU scheduling — Both containers request the same GPU device, with llama.cpp handling inference and the pipeline managing STT/TTS acceleration.

Customization Options

Switch to CPU-Only Deployment

For systems without NVIDIA GPUs, override the Dockerfile selection:

DOCKERFILE=Dockerfile.cpu docker compose up --build -d

Use a Different GGUF Model

Modify the -hf argument in docker-compose.yml:

command:
  - -hf
  - your-org/your-model-GGUF  # Replace model identifier

Adjust GPU Device Assignment

Change device_ids in both services to target specific GPUs:

device_ids: ['1']  # Use second GPU

Verification and Health Checks

Confirm successful deployment with these diagnostic commands:

# Check LLM server model availability

curl -s http://localhost:8080/v1/models | jq '.data[].id'

# Verify speech-to-speech endpoint responsiveness

curl -s http://localhost:8765/v1/models

# Inspect container logs for errors

docker compose logs -f llama
docker compose logs -f pipeline

Troubleshooting Common Issues

Symptom	Cause	Resolution
nvidia-smi fails inside container	NVIDIA runtime not configured	Install NVIDIA Container Toolkit and restart Docker
Port 8080 already bound	Conflicting service on host	Change ports: mapping to 8081:8080/tcp
Model download hangs	Network or permission issue	Verify ./cache/ is writable and check connectivity
GPU memory exhausted	Model too large for VRAM	Select smaller GGUF quantization or reduce -c context size
Pipeline cannot reach LLM	DNS resolution failure	Ensure both containers on same Docker network: docker network inspect speech-to-speech_default

Summary

• The huggingface/speech-to-speech repository provides a complete docker-compose.yml for local LLM deployment
• Two services run: llama (LLM inference) and pipeline (speech-to-speech API)
• GPU acceleration requires NVIDIA Container Toolkit with matching driver versions
• Shared ./cache/ volume persists downloaded models across restarts
• The pipeline service routes LLM requests internally via http://llama:8080/v1
• Override DOCKERFILE environment variable to select alternative container builds

Frequently Asked Questions

What LLM models are compatible with this setup?

Any GGUF-format model hosted on Hugging Face works with the llama.cpp backend. The default gemma-4-E4B-it-GGUF provides a balance of quality and speed, but you can substitute larger models like Llama 3 or Mistral by changing the -hf parameter. Ensure your GPU has sufficient VRAM for the selected model's activation weights.

Do I need an OpenAI API key for local inference?

No. The --llm_backend responses-api configuration redirects all generation to the local llama container. The API compatibility layer mimics OpenAI's chat completion format, allowing tools designed for OpenAI to work with your local model without code changes.

How do I monitor GPU utilization across both containers?

Run docker exec combined with nvidia-smi to check allocation per container, or use nvidia-smi dmon on the host for continuous monitoring. Since both services request the same GPU device, their processes appear together in host-level GPU statistics.

Can I scale the LLM service to multiple replicas?

Docker Compose's default networking complicates multi-replica LLM deployments because the pipeline service expects a single llama hostname. For horizontal scaling, migrate to Kubernetes or implement a load balancer container that the pipeline targets instead of connecting directly to llama.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}