# How to Deploy the Hugging Face Speech‑to‑Speech Pipeline with Docker and Docker Compose

> Deploy the Hugging Face speech-to-speech pipeline easily with Docker and Docker Compose. Launch an LLM backend with GPU support and persistent caching using the provided docker-compose.yml.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-06

---

**Use the pre‑configured [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml) in the `huggingface/speech-to-speech` repository to launch an LLM backend and the speech pipeline together, with GPU support and persistent model caching via a shared volume.**

The `huggingface/speech-to-speech` repository provides a complete **Docker and Docker Compose deployment** for running a local, GPU‑accelerated speech conversation system. This setup orchestrates two containers—a [`llama.cpp`](https://github.com/huggingface/speech-to-speech/blob/main/llama.cpp) server for language modeling and the native speech pipeline—into a single stack that exposes socket endpoints for real‑time audio processing. Below is the authoritative guide based on the actual source files in the repository.

## Prerequisites and System Requirements

Before deploying, ensure your host meets these requirements:

- **NVIDIA GPU** with CUDA support (both containers require GPU access)
- **Docker Engine** 20.10+ and **Docker Compose** v2+
- **NVIDIA Container Toolkit** installed and configured

Install the NVIDIA Container Toolkit following the [official guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) to enable GPU passthrough to containers.

## Repository Structure and Key Files

The deployment relies on these source files:

| File | Purpose | Source Link |
|------|---------|-------------|
| `Dockerfile` | Builds the speech‑to‑speech image with CUDA runtime, Python dependencies, and `uv`-based package installation | [`Dockerfile`](https://github.com/huggingface/speech-to-speech/blob/main/Dockerfile) |
| `Dockerfile.arm64` | ARM64 variant for Apple Silicon or ARM servers | [`Dockerfile.arm64`](https://github.com/huggingface/speech-to-speech/blob/main/Dockerfile.arm64) |
| [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml) | Orchestrates the `llama` and `pipeline` services with GPU reservations, port mapping, and shared caching | [[`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml)](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml) |
| [`demo/server.py`](https://github.com/huggingface/speech-to-speech/blob/main/demo/server.py) | Optional HTTP server wrapper for custom integrations | [[`demo/server.py`](https://github.com/huggingface/speech-to-speech/blob/main/demo/server.py)](https://github.com/huggingface/speech-to-speech/blob/main/demo/server.py) |

## Architecture Overview

The Docker Compose configuration creates a two‑service architecture:

- **`llama` service** – runs a pre‑built [`llama.cpp`](https://github.com/huggingface/speech-to-speech/blob/main/llama.cpp) server image (`ghcr.io/ggml‑org/llama.cpp:server-cuda`) that exposes a REST API on port 8080 for text generation
- **`pipeline` service** – built from the repository's `Dockerfile`, runs `speech-to-speech` in **socket mode**, connects to the LLM backend, and exposes ports 12345 (voice‑to‑text) and 12346 (text‑to‑voice)

Both containers mount `./cache/` to `/root/.cache/` for persistent model storage and declare NVIDIA GPU reservations under `deploy.resources.reservations.devices`.

## Step‑by‑Step Deployment Guide

### 1. Clone the Repository

```bash
git clone https://github.com/huggingface/speech-to-speech.git
cd speech-to-speech

```

### 2. Select the Appropriate Dockerfile

| Platform | Command |
|----------|---------|
| x86_64 (default) | No action needed; uses `Dockerfile` |
| ARM64 | Export environment variable: `export DOCKERFILE=Dockerfile.arm64` |

### 3. Launch the Stack

For standard x86_64 systems:

```bash
docker compose up --build

```

For ARM64 systems:

```bash
DOCKERFILE=Dockerfile.arm64 docker compose up --build

```

The `--build` flag ensures the `pipeline` image compiles from the current source, installing dependencies via `uv` as defined in the `Dockerfile`.

### 4. Verify Endpoint Availability

| Service | URL | Purpose |
|---------|-----|---------|
| LLM backend | `http://localhost:8080/v1` | OpenAI‑compatible chat completions API |
| Speech pipeline | `tcp://localhost:12345` | Voice‑to‑text input socket |
| Speech pipeline | `tcp://localhost:12346` | Text‑to‑voice output socket |

Test the pipeline locally with matching CLI flags:

```bash
speech-to-speech --mode socket \
  --recv_host 0.0.0.0 \
  --send_host 0.0.0.0 \
  --llm_backend responses-api \
  --model_name ggml-org/gemma-4-E4B-it-GGUF \
  --responses_api_base_url http://localhost:8080/v1

```

These parameters mirror the `command` block in the [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml) `pipeline` service definition.

## Docker Compose Configuration Deep‑Dive

The [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml) file ([source](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml)) defines these critical elements:

**`llama` service excerpt:**

```yaml
services:
  llama:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda
    command: [
      -hf, ggml-org/gemma-4-E4B-it-GGUF,
      -np, "2",
      -c, "65536",
      -fa, "on",
      --swa-full,
      --host, 0.0.0.0,
      --port, "8080"
    ]
    ports: ["8080:8080/tcp"]
    volumes: ["./cache/:/root/.cache/"]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]

```

**`pipeline` service excerpt:**

```yaml
  pipeline:
    depends_on: [llama]
    build:
      context: .
      dockerfile: ${DOCKERFILE:-Dockerfile}
    command:
      - speech-to-speech
      - --mode, socket
      - --recv_host, 0.0.0.0
      - --send_host, 0.0.0.0
      - --llm_backend, responses-api
      - --model_name, ggml-org/gemma-4-E4B-it-GGUF
      - --responses_api_base_url, http://llama:8080/v1
      - --responses_api_api_key, ""
      - --init_chat_role, system
      - --init_chat_prompt, "You are a helpful assistant"
    expose: ["12345/tcp", "12346/tcp"]
    ports: ["12345:12345/tcp", "12346:12346/tcp"]
    volumes: ["./cache/:/root/.cache/"]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]

```

Key implementation details:

- **Service dependency** – `pipeline` waits for `llama` to start via `depends_on`
- **Inter‑service networking** – the pipeline references `http://llama:8080/v1` using Docker's internal DNS
- **GPU isolation** – both services request `device_ids: ['0']` (first GPU); modify for multi‑GPU hosts
- **Cache persistence** – the `./cache/` bind mount prevents re‑downloading models on container restart

## Dockerfile Build Process

The `Dockerfile` ([source](https://github.com/huggingface/speech-to-speech/blob/main/Dockerfile)) constructs the runtime environment:

```dockerfile
FROM nvidia/cuda:12.8.1-cudnn-runtime-ubuntu24.04
ENV DEBIAN_FRONTEND=noninteractive \
    PYTHONUNBUFFERED=1 \
    PATH="/usr/src/app/.venv/bin:${PATH}"

WORKDIR /usr/src/app

RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates git libportaudio2 libsndfile1 \
    python3 python3-pip python3-venv \
    && rm -rf /var/lib/apt/lists/*

RUN python3 -m pip install --no-cache-dir --break-system-packages uv

COPY pyproject.toml README.md LICENSE MANIFEST.in ./
RUN uv sync --python /usr/bin/python3 --no-install-project --no-dev

COPY . .
RUN uv sync --python /usr/bin/python3 --no-dev

RUN python -c "import nltk; nltk.download('punkt_tab'); nltk.download('averaged_perceptron_tagger_eng')"

```

The build uses **`uv`** for fast Python dependency resolution and installs required audio libraries (`libportaudio2`, `libsndfile1`) for real‑time audio I/O.

## Managing the Deployment

| Operation | Command |
|-----------|---------|
| Start (foreground) | `docker compose up` |
| Start (detached) | `docker compose up -d` |
| View logs | `docker compose logs -f` |
| Rebuild after code changes | `docker compose up --build` |
| Stop and remove containers | `docker compose down` |
| Stop, remove containers **and** delete cache | `docker compose down -v` (avoid unless intentional) |

Model files persist in `./cache/` across restarts. To force re‑download, remove `./cache/` before starting the stack.

## Customization Options

| Variable | Description | Default in [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml) |
|----------|-------------|--------------------------------|
| `DOCKERFILE` | Build target (`Dockerfile` or `Dockerfile.arm64`) | `Dockerfile` |
| `model_name` | GGUF model served by [`llama.cpp`](https://github.com/huggingface/speech-to-speech/blob/main/llama.cpp) | `ggml-org/gemma-4-E4B-it-GGUF` |
| `init_chat_prompt` | System prompt for the conversation | `"You are a helpful assistant"` |
| GPU device ID | `device_ids` array under `deploy.resources.reservations.devices` | `['0']` |

Modify the `llama` service `command` array to switch models or adjust context length (`-c` flag). Update the `pipeline` service environment or command to change LLM backend behavior.

## Summary

- **Clone** `huggingface/speech-to-speech` and **install the NVIDIA Container Toolkit** for GPU support
- **Choose** `Dockerfile` (x86_64) or `Dockerfile=Dockerfile.arm64` (ARM64) before building
- **Run** `docker compose up --build` to start the LLM backend and speech pipeline together
- **Connect** to `localhost:12345` (input) and `localhost:12346` (output) for real‑time audio processing
- **Persist** models in `./cache/` to avoid re‑download on restart

## Frequently Asked Questions

### Do I need a GPU to run the speech-to-speech Docker deployment?

Yes. Both the `llama` service (for LLM inference) and the `pipeline` service (for STT/TTS) declare `deploy.resources.reservations.devices` with NVIDIA driver requirements in [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml). CPU‑only execution is not supported in this configuration. Ensure your host has an NVIDIA GPU with CUDA capability and the NVIDIA Container Toolkit installed.

### How do I change the language model used by the deployment?

Edit the `command` array in the `llama` service within [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml). The `-hf` flag specifies the Hugging Face GGUF repository (default: `ggml-org/gemma-4-E4B-it-GGUF`). Update the matching `--model_name` in the `pipeline` service `command` to ensure consistency. Restart with `docker compose up --build` to apply changes.

### Can I deploy this on Apple Silicon Macs?

Yes, using the ARM64 Dockerfile. Set `DOCKERFILE=Dockerfile.arm64` before running `docker compose up --build`. Note that GPU acceleration on Apple Silicon requires specific Docker Desktop settings for virtualization—verify that your Docker installation supports GPU passthrough or expect CPU‑fallback performance for the LLM component.

### How do I integrate the pipeline with my own application?

Connect to the exposed TCP sockets on ports 12345 and 12346, or adapt [`demo/server.py`](https://github.com/huggingface/speech-to-speech/blob/main/demo/server.py) ([source](https://github.com/huggingface/speech-to-speech/blob/main/demo/server.py)) as an HTTP wrapper. The socket interface expects raw audio streams for voice-to-text input and returns synthesized audio on the output channel. For REST‑based integration, modify the `pipeline` service command to use `--mode http` and expose an appropriate port, though this requires custom container configuration beyond the default compose file.