# How to Deploy Speech-to-Speech Server with Docker and Docker Compose

> Easily deploy a speech-to-speech server with Docker and Docker Compose. This guide shows how to set up a GPU-accelerated LLM backend for seamless voice translation.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-10

---

**You can deploy the Hugging Face Speech-to-Speech server using the provided Docker Compose configuration, which orchestrates a GPU-accelerated LLM backend alongside the main server container.**

The `huggingface/speech-to-speech` repository provides production-ready containerization that splits the architecture into two distinct services: a GGML-based LLM inference engine and the Speech-to-Speech pipeline server. This guide walks through the exact commands and configuration files—`Dockerfile` and [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml)—needed to launch a fully functional speech-to-speech API endpoint on port 8765.

## Architecture Overview

The deployment architecture deliberately separates concerns between the language model inference and the speech processing pipeline.

- **llama service**: Runs the `ghcr.io/ggml-org/llama.cpp:server-cuda` image to serve the Gemma-4 GGUF model via CUDA acceleration on port 8080.
- **pipeline service**: Builds from the repository's `Dockerfile` and executes the `speech-to-speech serve` CLI, proxying LLM requests to the llama backend.

Both services mount a local `./cache/` directory to `/root/.cache/` inside the containers, ensuring model weights persist across container restarts. The `deploy.resources.reservations.devices` configuration in [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml) requests NVIDIA GPU access for both containers.

## Prerequisites

Before deploying, verify your environment meets these requirements:

1. **Docker Engine** (version 24 or newer) and **Docker Compose** (v2) installed.
2. **NVIDIA GPU** with proper drivers installed on the host.
3. **NVIDIA Container Runtime** configured as the default Docker runtime to enable GPU passthrough.

## Step-by-Step Deployment Guide

### 1. Clone the Repository

Download the source code containing the Docker configuration files:

```bash
git clone https://github.com/huggingface/speech-to-speech.git
cd speech-to-speech

```

### 2. (Optional) Pre-Pull the LLM Image

While Docker Compose handles image retrieval automatically, you can expedite startup by pulling the llama.cpp server image beforehand:

```bash
docker pull ghcr.io/ggml-org/llama.cpp:server-cuda

```

### 3. Start the Stack

Build the pipeline image and launch both services in detached mode:

```bash
docker compose up -d --build

```

The `--build` flag ensures the `pipeline` service rebuilds if you have modified the source code. The `DOCKERFILE` environment variable can override the default build context if you need an alternative Dockerfile.

During startup, the pipeline service executes:

```bash
speech-to-speech serve \
  --host 0.0.0.0 \
  --port 8765 \
  --llm_backend responses-api \
  --model_name ggml-org/gemma-4-E4B-it-GGUF \
  --responses_api_base_url http://llama:8080/v1 \
  --responses_api_api_key "" \
  --init_chat_role system \
  --init_chat_prompt "You are a helpful assistant"

```

### 4. Verify the Deployment

Wait approximately 30-60 seconds for the LLM to load, then test the API endpoint:

```bash
curl http://localhost:8765/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

```

A successful response returns JSON containing the generated text from the model.

### 5. Stop and Clean Up

To halt the services:

```bash
docker compose down

```

To remove containers and delete the persistent cache volume:

```bash
docker compose down --volumes

```

## Standalone Docker Deployment

If you prefer running only the Speech-to-Speech server without the bundled LLM backend—connecting instead to an external LLM endpoint—build and run the image manually:

```bash

# Build the image

docker build -t speech-to-speech:latest .

# Run with external LLM configuration

docker run -it --rm \
  -p 8765:8765 \
  -v $(pwd)/cache:/root/.cache \
  --gpus all \
  speech-to-speech:latest \
  speech-to-speech serve \
  --host 0.0.0.0 \
  --port 8765 \
  --llm_backend responses-api \
  --responses_api_base_url http://my-llm:8080/v1 \
  --responses_api_api_key "my-api-key"

```

## Key Configuration Files

Understanding these source files helps customize your deployment:

- **`Dockerfile`**: Defines the Python environment, installs dependencies, and builds the `speech-to-speech` CLI binary. Located at the repository root.
- **[`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml)**: Orchestrates the multi-container stack, defining service dependencies, GPU reservations, and the cache volume mount.
- **`demo/Dockerfile`**: Optional lightweight UI container that provides a web interface for interacting with the server.
- **[`demo/server.py`](https://github.com/huggingface/speech-to-speech/blob/main/demo/server.py)**: FastAPI wrapper script used by the demo container to bridge browser clients with the Speech-to-Speech backend.

## Summary

- The **huggingface/speech-to-speech** repository provides a complete Docker Compose stack combining a CUDA-enabled llama.cpp backend with the main pipeline server.
- The **pipeline service** exposes port 8765 and proxies LLM requests to the **llama service** on port 8080 using the responses-api backend.
- Model caches persist in `./cache/` mapped to `/root/.cache/` to avoid re-downloading weights on restart.
- Both containers require **NVIDIA GPU** access configured through Docker's device reservations.

## Frequently Asked Questions

### What GPU requirements are needed for Docker deployment?

The deployment requires an NVIDIA GPU with CUDA support. The [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml) explicitly requests GPU resources via `deploy.resources.reservations.devices`, and the llama.cpp image uses CUDA-accelerated inference. Ensure your host has NVIDIA drivers and the nvidia-container-runtime installed.

### Can I use a different LLM backend instead of llama.cpp?

Yes. While the default [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml) configures the `responses-api` backend pointing to the local llama.cpp container, you can modify the `pipeline` service command to use any OpenAI-compatible endpoint. Update `--responses_api_base_url` to point to your external LLM service and provide the appropriate API key via `--responses_api_api_key`.

### How do I view logs when running in detached mode?

Use `docker compose logs -f pipeline` to stream logs from the Speech-to-Speech server, or `docker compose logs -f llama` to monitor the LLM backend. The `-f` flag follows log output in real-time, essential for debugging startup issues or monitoring inference requests.

### Where are downloaded models stored?

Models download to the `./cache/` directory on your host machine, which mounts to `/root/.cache/` inside both containers. This shared volume persists the GGUF weights and Hugging Face transformers cache across container restarts, preventing redundant downloads when you run `docker compose up` again.