How to Deploy the Hugging Face Speech‑to‑Speech Pipeline with Docker and Docker Compose

Use the pre‑configured docker-compose.yml in the huggingface/speech-to-speech repository to launch an LLM backend and the speech pipeline together, with GPU support and persistent model caching via a shared volume.

The huggingface/speech-to-speech repository provides a complete Docker and Docker Compose deployment for running a local, GPU‑accelerated speech conversation system. This setup orchestrates two containers—a llama.cpp server for language modeling and the native speech pipeline—into a single stack that exposes socket endpoints for real‑time audio processing. Below is the authoritative guide based on the actual source files in the repository.

Prerequisites and System Requirements

Before deploying, ensure your host meets these requirements:

• NVIDIA GPU with CUDA support (both containers require GPU access)
• Docker Engine 20.10+ and Docker Compose v2+
• NVIDIA Container Toolkit installed and configured

Install the NVIDIA Container Toolkit following the official guide to enable GPU passthrough to containers.

Repository Structure and Key Files

The deployment relies on these source files:

File	Purpose	Source Link
Dockerfile	Builds the speech‑to‑speech image with CUDA runtime, Python dependencies, and uv-based package installation	Dockerfile
Dockerfile.arm64	ARM64 variant for Apple Silicon or ARM servers	Dockerfile.arm64
docker-compose.yml	Orchestrates the llama and pipeline services with GPU reservations, port mapping, and shared caching	[docker-compose.yml](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml)
demo/server.py	Optional HTTP server wrapper for custom integrations	[demo/server.py](https://github.com/huggingface/speech-to-speech/blob/main/demo/server.py)

Architecture Overview

The Docker Compose configuration creates a two‑service architecture:

• llama service – runs a pre‑built llama.cpp server image (ghcr.io/ggml‑org/llama.cpp:server-cuda) that exposes a REST API on port 8080 for text generation
• pipeline service – built from the repository's Dockerfile, runs speech-to-speech in socket mode, connects to the LLM backend, and exposes ports 12345 (voice‑to‑text) and 12346 (text‑to‑voice)

Both containers mount ./cache/ to /root/.cache/ for persistent model storage and declare NVIDIA GPU reservations under deploy.resources.reservations.devices.

Step‑by‑Step Deployment Guide

1. Clone the Repository

git clone https://github.com/huggingface/speech-to-speech.git
cd speech-to-speech

2. Select the Appropriate Dockerfile

Platform	Command
x86_64 (default)	No action needed; uses Dockerfile
ARM64	Export environment variable: export DOCKERFILE=Dockerfile.arm64

3. Launch the Stack

For standard x86_64 systems:

docker compose up --build

For ARM64 systems:

DOCKERFILE=Dockerfile.arm64 docker compose up --build

The --build flag ensures the pipeline image compiles from the current source, installing dependencies via uv as defined in the Dockerfile.

4. Verify Endpoint Availability

Service	URL	Purpose
LLM backend	http://localhost:8080/v1	OpenAI‑compatible chat completions API
Speech pipeline	tcp://localhost:12345	Voice‑to‑text input socket
Speech pipeline	tcp://localhost:12346	Text‑to‑voice output socket

Test the pipeline locally with matching CLI flags:

speech-to-speech --mode socket \
  --recv_host 0.0.0.0 \
  --send_host 0.0.0.0 \
  --llm_backend responses-api \
  --model_name ggml-org/gemma-4-E4B-it-GGUF \
  --responses_api_base_url http://localhost:8080/v1

These parameters mirror the command block in the docker-compose.yml pipeline service definition.

Docker Compose Configuration Deep‑Dive

The docker-compose.yml file (source) defines these critical elements:

llama service excerpt:

services:
  llama:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda
    command: [
      -hf, ggml-org/gemma-4-E4B-it-GGUF,
      -np, "2",
      -c, "65536",
      -fa, "on",
      --swa-full,
      --host, 0.0.0.0,
      --port, "8080"
    ]
    ports: ["8080:8080/tcp"]
    volumes: ["./cache/:/root/.cache/"]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]

pipeline service excerpt:

  pipeline:
    depends_on: [llama]
    build:
      context: .
      dockerfile: ${DOCKERFILE:-Dockerfile}
    command:
      - speech-to-speech
      - --mode, socket
      - --recv_host, 0.0.0.0
      - --send_host, 0.0.0.0
      - --llm_backend, responses-api
      - --model_name, ggml-org/gemma-4-E4B-it-GGUF
      - --responses_api_base_url, http://llama:8080/v1
      - --responses_api_api_key, ""
      - --init_chat_role, system
      - --init_chat_prompt, "You are a helpful assistant"
    expose: ["12345/tcp", "12346/tcp"]
    ports: ["12345:12345/tcp", "12346:12346/tcp"]
    volumes: ["./cache/:/root/.cache/"]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]

Key implementation details:

• Service dependency – pipeline waits for llama to start via depends_on
• Inter‑service networking – the pipeline references http://llama:8080/v1 using Docker's internal DNS
• GPU isolation – both services request device_ids: ['0'] (first GPU); modify for multi‑GPU hosts
• Cache persistence – the ./cache/ bind mount prevents re‑downloading models on container restart

Dockerfile Build Process

The Dockerfile (source) constructs the runtime environment:

FROM nvidia/cuda:12.8.1-cudnn-runtime-ubuntu24.04
ENV DEBIAN_FRONTEND=noninteractive \
    PYTHONUNBUFFERED=1 \
    PATH="/usr/src/app/.venv/bin:${PATH}"

WORKDIR /usr/src/app

RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates git libportaudio2 libsndfile1 \
    python3 python3-pip python3-venv \
    && rm -rf /var/lib/apt/lists/*

RUN python3 -m pip install --no-cache-dir --break-system-packages uv

COPY pyproject.toml README.md LICENSE MANIFEST.in ./
RUN uv sync --python /usr/bin/python3 --no-install-project --no-dev

COPY . .
RUN uv sync --python /usr/bin/python3 --no-dev

RUN python -c "import nltk; nltk.download('punkt_tab'); nltk.download('averaged_perceptron_tagger_eng')"

The build uses uv for fast Python dependency resolution and installs required audio libraries (libportaudio2, libsndfile1) for real‑time audio I/O.

Managing the Deployment

Operation	Command
Start (foreground)	docker compose up
Start (detached)	docker compose up -d
View logs	docker compose logs -f
Rebuild after code changes	docker compose up --build
Stop and remove containers	docker compose down
Stop, remove containers and delete cache	docker compose down -v (avoid unless intentional)

Model files persist in ./cache/ across restarts. To force re‑download, remove ./cache/ before starting the stack.

Customization Options

Variable	Description	Default in docker-compose.yml
DOCKERFILE	Build target (Dockerfile or Dockerfile.arm64)	Dockerfile
model_name	GGUF model served by llama.cpp	ggml-org/gemma-4-E4B-it-GGUF
init_chat_prompt	System prompt for the conversation	"You are a helpful assistant"
GPU device ID	device_ids array under deploy.resources.reservations.devices	['0']

Modify the llama service command array to switch models or adjust context length (-c flag). Update the pipeline service environment or command to change LLM backend behavior.

Summary

• Clone huggingface/speech-to-speech and install the NVIDIA Container Toolkit for GPU support
• Choose Dockerfile (x86_64) or Dockerfile=Dockerfile.arm64 (ARM64) before building
• Run docker compose up --build to start the LLM backend and speech pipeline together
• Connect to localhost:12345 (input) and localhost:12346 (output) for real‑time audio processing
• Persist models in ./cache/ to avoid re‑download on restart

Frequently Asked Questions

Do I need a GPU to run the speech-to-speech Docker deployment?

Yes. Both the llama service (for LLM inference) and the pipeline service (for STT/TTS) declare deploy.resources.reservations.devices with NVIDIA driver requirements in docker-compose.yml. CPU‑only execution is not supported in this configuration. Ensure your host has an NVIDIA GPU with CUDA capability and the NVIDIA Container Toolkit installed.

How do I change the language model used by the deployment?

Edit the command array in the llama service within docker-compose.yml. The -hf flag specifies the Hugging Face GGUF repository (default: ggml-org/gemma-4-E4B-it-GGUF). Update the matching --model_name in the pipeline service command to ensure consistency. Restart with docker compose up --build to apply changes.

Can I deploy this on Apple Silicon Macs?

Yes, using the ARM64 Dockerfile. Set DOCKERFILE=Dockerfile.arm64 before running docker compose up --build. Note that GPU acceleration on Apple Silicon requires specific Docker Desktop settings for virtualization—verify that your Docker installation supports GPU passthrough or expect CPU‑fallback performance for the LLM component.

How do I integrate the pipeline with my own application?

Connect to the exposed TCP sockets on ports 12345 and 12346, or adapt demo/server.py (source) as an HTTP wrapper. The socket interface expects raw audio streams for voice-to-text input and returns synthesized audio on the output channel. For REST‑based integration, modify the pipeline service command to use --mode http and expose an appropriate port, though this requires custom container configuration beyond the default compose file.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}