How to Deploy Speech-to-Speech Server with Docker and Docker Compose

You can deploy the Hugging Face Speech-to-Speech server using the provided Docker Compose configuration, which orchestrates a GPU-accelerated LLM backend alongside the main server container.

The huggingface/speech-to-speech repository provides production-ready containerization that splits the architecture into two distinct services: a GGML-based LLM inference engine and the Speech-to-Speech pipeline server. This guide walks through the exact commands and configuration files—Dockerfile and docker-compose.yml—needed to launch a fully functional speech-to-speech API endpoint on port 8765.

Architecture Overview

The deployment architecture deliberately separates concerns between the language model inference and the speech processing pipeline.

• llama service: Runs the ghcr.io/ggml-org/llama.cpp:server-cuda image to serve the Gemma-4 GGUF model via CUDA acceleration on port 8080.
• pipeline service: Builds from the repository's Dockerfile and executes the speech-to-speech serve CLI, proxying LLM requests to the llama backend.

Both services mount a local ./cache/ directory to /root/.cache/ inside the containers, ensuring model weights persist across container restarts. The deploy.resources.reservations.devices configuration in docker-compose.yml requests NVIDIA GPU access for both containers.

Prerequisites

Before deploying, verify your environment meets these requirements:

1. Docker Engine (version 24 or newer) and Docker Compose (v2) installed.
2. NVIDIA GPU with proper drivers installed on the host.
3. NVIDIA Container Runtime configured as the default Docker runtime to enable GPU passthrough.

Step-by-Step Deployment Guide

1. Clone the Repository

Download the source code containing the Docker configuration files:

git clone https://github.com/huggingface/speech-to-speech.git
cd speech-to-speech

2. (Optional) Pre-Pull the LLM Image

While Docker Compose handles image retrieval automatically, you can expedite startup by pulling the llama.cpp server image beforehand:

docker pull ghcr.io/ggml-org/llama.cpp:server-cuda

3. Start the Stack

Build the pipeline image and launch both services in detached mode:

docker compose up -d --build

The --build flag ensures the pipeline service rebuilds if you have modified the source code. The DOCKERFILE environment variable can override the default build context if you need an alternative Dockerfile.

During startup, the pipeline service executes:

speech-to-speech serve \
  --host 0.0.0.0 \
  --port 8765 \
  --llm_backend responses-api \
  --model_name ggml-org/gemma-4-E4B-it-GGUF \
  --responses_api_base_url http://llama:8080/v1 \
  --responses_api_api_key "" \
  --init_chat_role system \
  --init_chat_prompt "You are a helpful assistant"

4. Verify the Deployment

Wait approximately 30-60 seconds for the LLM to load, then test the API endpoint:

curl http://localhost:8765/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

A successful response returns JSON containing the generated text from the model.

5. Stop and Clean Up

To halt the services:

docker compose down

To remove containers and delete the persistent cache volume:

docker compose down --volumes

Standalone Docker Deployment

If you prefer running only the Speech-to-Speech server without the bundled LLM backend—connecting instead to an external LLM endpoint—build and run the image manually:

# Build the image

docker build -t speech-to-speech:latest .

# Run with external LLM configuration

docker run -it --rm \
  -p 8765:8765 \
  -v $(pwd)/cache:/root/.cache \
  --gpus all \
  speech-to-speech:latest \
  speech-to-speech serve \
  --host 0.0.0.0 \
  --port 8765 \
  --llm_backend responses-api \
  --responses_api_base_url http://my-llm:8080/v1 \
  --responses_api_api_key "my-api-key"

Key Configuration Files

Understanding these source files helps customize your deployment:

• Dockerfile: Defines the Python environment, installs dependencies, and builds the speech-to-speech CLI binary. Located at the repository root.
• docker-compose.yml: Orchestrates the multi-container stack, defining service dependencies, GPU reservations, and the cache volume mount.
• demo/Dockerfile: Optional lightweight UI container that provides a web interface for interacting with the server.
• demo/server.py: FastAPI wrapper script used by the demo container to bridge browser clients with the Speech-to-Speech backend.

Summary

• The huggingface/speech-to-speech repository provides a complete Docker Compose stack combining a CUDA-enabled llama.cpp backend with the main pipeline server.
• The pipeline service exposes port 8765 and proxies LLM requests to the llama service on port 8080 using the responses-api backend.
• Model caches persist in ./cache/ mapped to /root/.cache/ to avoid re-downloading weights on restart.
• Both containers require NVIDIA GPU access configured through Docker's device reservations.

Frequently Asked Questions

What GPU requirements are needed for Docker deployment?

The deployment requires an NVIDIA GPU with CUDA support. The docker-compose.yml explicitly requests GPU resources via deploy.resources.reservations.devices, and the llama.cpp image uses CUDA-accelerated inference. Ensure your host has NVIDIA drivers and the nvidia-container-runtime installed.

Can I use a different LLM backend instead of llama.cpp?

Yes. While the default docker-compose.yml configures the responses-api backend pointing to the local llama.cpp container, you can modify the pipeline service command to use any OpenAI-compatible endpoint. Update --responses_api_base_url to point to your external LLM service and provide the appropriate API key via --responses_api_api_key.

How do I view logs when running in detached mode?

Use docker compose logs -f pipeline to stream logs from the Speech-to-Speech server, or docker compose logs -f llama to monitor the LLM backend. The -f flag follows log output in real-time, essential for debugging startup issues or monitoring inference requests.

Where are downloaded models stored?

Models download to the ./cache/ directory on your host machine, which mounts to /root/.cache/ inside both containers. This shared volume persists the GGUF weights and Hugging Face transformers cache across container restarts, preventing redundant downloads when you run docker compose up again.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}