# How to Set Up a Docker Compose Environment with Local LLM Server for Speech-to-Speech

> Easily set up a Docker Compose environment with a local LLM server for speech-to-speech using the huggingface/speech-to-speech repository. Get GPU support out of the box.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-07

---

**Use the pre-configured [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml) in the huggingface/speech-to-speech repository to launch a local LLM backend alongside the speech-to-speech pipeline with GPU support out of the box.**

The **speech-to-speech** repository by Hugging Face provides a production-ready Docker Compose configuration that eliminates manual setup. According to the source code in [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml), the stack deploys two interconnected services: a GPU-accelerated [`llama.cpp`](https://github.com/huggingface/speech-to-speech/blob/main/llama.cpp) server for local LLM inference and the speech-to-speech pipeline that routes all generation requests to this backend.

## Architecture Overview

The Docker Compose environment orchestrates two specialized containers:

| Service | Purpose | Key Implementation Details |
|---------|---------|---------------------------|
| `llama` | **Local LLM server** | Runs `ghcr.io/ggml-org/llama.cpp:server-cuda` with OpenAI-compatible API on port 8080 |
| `pipeline` | **Speech-to-speech application** | Custom build from repository `Dockerfile`, exposes public API on port 8765 |

Both services mount a shared `./cache/` volume at `/root/.cache/` for model persistence and request NVIDIA GPU access through identical `deploy.resources.reservations.devices` configurations.

## Prerequisites

Before deploying, verify your environment meets these requirements:

- Docker Engine 20.10+ with **NVIDIA Container Toolkit** installed
- Compatible NVIDIA driver for the CUDA runtime in the [`llama.cpp`](https://github.com/huggingface/speech-to-speech/blob/main/llama.cpp) image
- At least 4 GB free disk space for the default GGUF model download
- Git for cloning the repository

## Step-by-Step Setup

Follow these commands to launch your local speech-to-speech environment with integrated LLM backend:

```bash

# 1. Clone the repository

git clone https://github.com/huggingface/speech-to-speech.git
cd speech-to-speech

# 2. (Optional) Customize configuration

# Edit docker-compose.yml to change model_id or GPU device_ids

# 3. Pull the GPU-enabled LLM image

docker pull ghcr.io/ggml-org/llama.cpp:server-cuda

# 4. Build and start all services

docker compose up --build -d

# 5. Verify LLM server health

curl http://localhost:8080/v1/models

# 6. Test the complete speech-to-speech pipeline

curl -X POST http://localhost:8765/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"ggml-org/gemma-4-E4B-it-GGUF","messages":[{"role":"user","content":"Hello!"}]}'

```

## Service Configuration Details

### LLM Service (`llama`)

As defined in [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml) lines 5-31, the LLM container uses this configuration:

```yaml
services:
  llama:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda
    command:
      - -hf
      - ggml-org/gemma-4-E4B-it-GGUF
      - -np
      - "2"
      - -c
      - "65536"
      - -fa
      - "on"
      - --swa-full
      - --host
      - 0.0.0.0
      - --port
      - "8080"
    ports:
      - 8080:8080/tcp
    volumes:
      - ./cache/:/root/.cache/
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]

```

**Key parameters:**
- `-hf ggml-org/gemma-4-E4B-it-GGUF` — Downloads and serves the Gemma 4B instruction-tuned model
- `-c 65536` — Allocates 65,536 tokens of context window
- `-np 2` — Enables 2 parallel completion slots
- `device_ids: ['0']` — Binds to first available GPU (modify for multi-GPU systems)

### Pipeline Service

The speech-to-speech container, configured in [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml) lines 32-70, connects to the LLM backend internally:

```yaml
  pipeline:
    build:
      context: .
      dockerfile: ${DOCKERFILE:-Dockerfile}
    command:
      - speech-to-speech
      - serve
      - --host
      - 0.0.0.0
      - --port
      - "8765"
      - --llm_backend
      - responses-api
      - --model_name
      - ggml-org/gemma-4-E4B-it-GGUF
      - --responses_api_base_url
      - http://llama:8080/v1
    ports:
      - 8765:8765/tcp
    volumes:
      - ./cache/:/root/.cache/
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]

```

The `--responses_api_base_url http://llama:8080/v1` argument routes all LLM calls to the internal Docker network address, keeping LLM traffic isolated from the host network while exposing only the speech-to-speech API endpoint.

## How the Components Interact

Understanding the data flow helps with troubleshooting and customization:

1. **Model caching** — On first startup, the `llama` service downloads the GGUF model to `./cache/`. Subsequent restarts reuse cached weights.

2. **Internal networking** — The `pipeline` service resolves `http://llama:8080/v1` through Docker's embedded DNS, forwarding all generation requests to the local LLM.

3. **Command entry point** — In [`src/speech_to_speech/cli.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/cli.py) (lines 156-166), the `speech-to-speech serve` command implements the server logic invoked by the container's `command` array.

4. **GPU scheduling** — Both containers request the same GPU device, with [`llama.cpp`](https://github.com/huggingface/speech-to-speech/blob/main/llama.cpp) handling inference and the pipeline managing STT/TTS acceleration.

## Customization Options

### Switch to CPU-Only Deployment

For systems without NVIDIA GPUs, override the Dockerfile selection:

```bash
DOCKERFILE=Dockerfile.cpu docker compose up --build -d

```

### Use a Different GGUF Model

Modify the `-hf` argument in [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml):

```yaml
command:
  - -hf
  - your-org/your-model-GGUF  # Replace model identifier

```

### Adjust GPU Device Assignment

Change `device_ids` in both services to target specific GPUs:

```yaml
device_ids: ['1']  # Use second GPU

```

## Verification and Health Checks

Confirm successful deployment with these diagnostic commands:

```bash

# Check LLM server model availability

curl -s http://localhost:8080/v1/models | jq '.data[].id'

# Verify speech-to-speech endpoint responsiveness

curl -s http://localhost:8765/v1/models

# Inspect container logs for errors

docker compose logs -f llama
docker compose logs -f pipeline

```

## Troubleshooting Common Issues

| Symptom | Cause | Resolution |
|---------|-------|------------|
| `nvidia-smi` fails inside container | NVIDIA runtime not configured | Install NVIDIA Container Toolkit and restart Docker |
| Port 8080 already bound | Conflicting service on host | Change `ports:` mapping to `8081:8080/tcp` |
| Model download hangs | Network or permission issue | Verify `./cache/` is writable and check connectivity |
| GPU memory exhausted | Model too large for VRAM | Select smaller GGUF quantization or reduce `-c` context size |
| Pipeline cannot reach LLM | DNS resolution failure | Ensure both containers on same Docker network: `docker network inspect speech-to-speech_default` |

## Summary

- The **huggingface/speech-to-speech** repository provides a complete [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml) for local LLM deployment
- Two services run: `llama` (LLM inference) and `pipeline` (speech-to-speech API)
- GPU acceleration requires NVIDIA Container Toolkit with matching driver versions
- Shared `./cache/` volume persists downloaded models across restarts
- The `pipeline` service routes LLM requests internally via `http://llama:8080/v1`
- Override `DOCKERFILE` environment variable to select alternative container builds

## Frequently Asked Questions

### What LLM models are compatible with this setup?

Any GGUF-format model hosted on Hugging Face works with the [`llama.cpp`](https://github.com/huggingface/speech-to-speech/blob/main/llama.cpp) backend. The default `gemma-4-E4B-it-GGUF` provides a balance of quality and speed, but you can substitute larger models like Llama 3 or Mistral by changing the `-hf` parameter. Ensure your GPU has sufficient VRAM for the selected model's activation weights.

### Do I need an OpenAI API key for local inference?

No. The `--llm_backend responses-api` configuration redirects all generation to the local `llama` container. The API compatibility layer mimics OpenAI's chat completion format, allowing tools designed for OpenAI to work with your local model without code changes.

### How do I monitor GPU utilization across both containers?

Run `docker exec` combined with `nvidia-smi` to check allocation per container, or use `nvidia-smi dmon` on the host for continuous monitoring. Since both services request the same GPU device, their processes appear together in host-level GPU statistics.

### Can I scale the LLM service to multiple replicas?

Docker Compose's default networking complicates multi-replica LLM deployments because the `pipeline` service expects a single `llama` hostname. For horizontal scaling, migrate to Kubernetes or implement a load balancer container that the pipeline targets instead of connecting directly to `llama`.