# Setting Up Docker Deployment with GPU Passthrough for Production

> Deploy Hugging Face Speech-to-Speech with GPU passthrough in production. Configure docker-compose.yml to enable CUDA access for LLM and pipeline containers.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-09

---

**To deploy the Hugging Face Speech-to-Speech pipeline with GPU passthrough in production, configure the [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml) to reserve NVIDIA devices via the `deploy.resources.reservations.devices` stanza, enabling both the LLM backend and pipeline containers to access CUDA drivers simultaneously.**

The `huggingface/speech-to-speech` repository ships a production-ready containerization strategy designed for NVIDIA GPU acceleration. Setting up Docker deployment with GPU passthrough for production requires configuring the provided `Dockerfile` and [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml) to ensure the NVIDIA Container Toolkit can inject driver libraries into the runtime environment.

## Prerequisites for NVIDIA GPU Passthrough

Before deploying containers, install the **NVIDIA Container Toolkit** on the host machine. This toolkit is essential because it intercepts Docker container creation requests and injects the necessary driver libraries, creating the `/dev/nvidia*` devices inside the container. Without this toolkit, the `deploy.resources.reservations.devices` configuration in Docker Compose will fail to allocate GPU resources.

## Core Architecture and Configuration Files

The deployment architecture centers on two files in the repository root that work together to provide CUDA support:

### Dockerfile CUDA Base Image

The `Dockerfile` builds upon `nvidia/cuda:12.8.1-cudnn-runtime-ubuntu24.04`, providing CUDA 12.8, cuDNN, and the runtime libraries required by deep learning frameworks like PyTorch and Transformers. The build process installs system dependencies, creates a virtual environment, and uses **uv** (a fast Python package resolver) to install the `speech-to-speech` package without development dependencies. This ensures all Python extensions are compiled against the specific CUDA version present in the image.

### Docker Compose Orchestration

The [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml) defines two interconnected services that both require GPU access:

| Component | Role | GPU Interaction |
|-----------|------|-----------------|
| **llama** | Runs the pre-built `ggml-org/llama.cpp:server-cuda` image | Executes a CUDA-accelerated binary for GGUF model inference |
| **pipeline** | Builds from the repository's `Dockerfile` | Runs `speech-to-speech serve` with access to the reserved GPU |

Both services use the `deploy.resources.reservations.devices` field to request GPU resources, ensuring the host's NVIDIA driver is accessible inside each container.

## Configuring GPU Resources in Docker Compose

The GPU passthrough mechanism relies on the `deploy` block within each service definition. The configuration specifies `driver: nvidia` and `capabilities: [gpu]`, which instructs Docker to expose the host's GPU to the container. As implemented in `huggingface/speech-to-speech`, this allows the `llama` service and the `pipeline` service to share GPU 0 without conflicting over device access.

## Step-by-Step Deployment Guide

### Building and Starting the Production Stack

To build the custom image and start both services with GPU support, run:

```bash
DOCKERFILE=Dockerfile docker compose up -d

```

This command builds the `pipeline` image using the specified Dockerfile and starts the orchestrated stack. The `-d` flag runs containers in detached mode suitable for production environments.

### Running a Standalone Container for Testing

For local testing without Docker Compose, build and run the container directly:

```bash
docker build -t s2s-demo -f Dockerfile .
docker run --gpus all -p 8765:8765 s2s-demo \
  speech-to-speech serve \
  --stt whisper \
  --tts pocket \
  --host 0.0.0.0 \
  --port 8765

```

The `--gpus all` flag tells Docker to pass through all available NVIDIA GPUs to the container.

## Verifying GPU Passthrough

Confirm that the containers can access the host GPU by executing `nvidia-smi` inside the running pipeline container:

```bash
docker exec -it $(docker ps -qf "name=pipeline") nvidia-smi

```

The output should display the host GPU with driver versions matching the host system, confirming that the NVIDIA Container Toolkit successfully injected the drivers.

## Low-Latency GPU Configuration

For production environments requiring minimal latency, the repository supports co-locating the TTS, STT, and LLM processing on the same GPU. According to [`src/speech_to_speech/TTS/README.md`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/README.md), the `--device` argument (defined in `src/speech_to_speech/arguments_classes/*_arguments.py`) defaults to `cuda`. When the container has GPU access via the reservation configuration, this default setting enables immediate GPU acceleration without additional flags.

## Summary

- The `Dockerfile` uses `nvidia/cuda:12.8.1-cudnn-runtime-ubuntu24.04` to ensure CUDA 12.8 compatibility and includes cuDNN runtime libraries
- [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml) orchestrates the `llama` and `pipeline` services with explicit GPU reservations using `deploy.resources.reservations.devices`
- The NVIDIA Container Toolkit must be installed on the host to enable driver injection and `/dev/nvidia*` device creation inside containers
- Verify successful GPU passthrough by running `nvidia-smi` inside the running container
- The `speech-to-speech serve` command uses `--device cuda` by default when GPU resources are available

## Frequently Asked Questions

### Do I need to install CUDA drivers inside the container?

No. The `Dockerfile` inherits from the official NVIDIA CUDA runtime base image, which includes the necessary CUDA libraries and cuDNN. The NVIDIA Container Toolkit handles binding the host's NVIDIA drivers into the container at runtime, so you do not need to install drivers manually.

### How do I specify which GPU to use if the host has multiple cards?

The default [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml) configuration reserves GPU 0. You can modify the `device_ids` field under `deploy.resources.reservations.devices` to specify alternative GPU indices (for example, `"1"` for the second GPU) or use `"all"` to pass through every available GPU to the container.

### Can I deploy the pipeline without the included LLM service?

Yes. While the default [`docker-compose.yml`](https://github.com/huggingface/speech-to-speech/blob/main/docker-compose.yml) includes the `llama` service for convenience, you can run the `pipeline` container standalone using `docker run --gpus all` as shown in the deployment steps. Alternatively, modify the compose file to remove the `llama` service dependency if you are using an external LLM backend API.

### Why does the build process use `uv` instead of `pip`?

The `Dockerfile` utilizes `uv` for faster Python package resolution and installation. This approach installs the `speech-to-speech` package and its dependencies more quickly than traditional pip, excludes development dependencies to reduce image size, and ensures the installed wheels are compiled specifically for the CUDA version present in the base image.