# How to Configure CUDA and Device Settings Across Components in Hugging Face Speech-to-Speech

> Master CUDA and device settings in Hugging Face speech-to-speech. Learn to assign device IDs and configure multi-GPU scheduling for optimal performance.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-06

---

**To configure CUDA and device settings in the Hugging Face `speech-to-speech` pipeline, you assign device IDs through component-specific argument classes like `LanguageModelHandlerArguments`, then ensure each handler initializes its model with `torch.device()` and handles multi-GPU scheduling via `device_map="auto"` or explicit CUDA ordinal strings.**

The `huggingface/speech-to-speech` repository implements a modular **speech-to-speech translation pipeline** where each component—language model, text-to-speech, and speech recognition—requires explicit device configuration for CUDA acceleration. Understanding the device cascade across these handlers ensures optimal GPU utilization and prevents cross-device tensor errors.

---

## Primary Configuration Entry Point: Argument Classes

Device configuration flows through **argument dataclasses** defined in `src/speech_to_speech/arguments_classes/`. These classes parse command-line inputs and propagate device settings to their respective handlers.

### Language Model Device Configuration

The `LanguageModelHandlerArguments` class in [`language_model_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/language_model_arguments.py) exposes the `lm_device` parameter:

```python

# From src/speech_to_speech/arguments_classes/language_model_arguments.py

@dataclass
class LanguageModelHandlerArguments:
    lm_device: str = field(
        default="cuda",
        metadata={"help": "The device to load the language model on. Defaults to 'cuda'."}
    )
    lm_device_map: str = field(
        default="auto",
        metadata={"help": "Device map for model parallelism. Use 'auto' for automatic placement."}
    )

```

**Key parameters:**
- `lm_device` — Accepts `"cuda"`, `"cuda:0"`, `"cuda:1"`, or `"cpu"`
- `lm_device_map` — Passed to `accelerate`'s `device_map` for model sharding across GPUs

### Text-to-Speech Device Configuration

Similarly, `MeloTTSHandlerArguments` in [`tts_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/tts_arguments.py) controls TTS device placement:

```python

# From src/speech_to_speech/arguments_classes/tts_arguments.py

@dataclass
class MeloTTSHandlerArguments:
    tts_device: str = field(
        default="cuda" if torch.cuda.is_available() else "cpu",
        metadata={"help": "Device for MeloTTS inference"}
    )

```

### Speech Recognition Device Configuration

The Whisper-based ASR component uses `WhisperSTTHandlerArguments`:

```python

# From src/speech_to_speech/arguments_classes/stt_arguments.py

@dataclass
class WhisperSTTHandlerArguments:
    stt_device: int = field(
        default=0,
        metadata={"help": "CUDA device ordinal for Whisper model (-1 for CPU)"}
    )

```

**Note the type difference:** `stt_device` uses `int` (ordinal) rather than `str`, requiring `-1` for CPU fallback.

---

## Handler-Level Device Initialization

Each handler converts argument values into `torch.device` objects during `__init__`. The `LanguageModelHandler` implementation in [`src/speech_to_speech/language_model_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/language_model_handler.py) demonstrates this pattern:

```python

# From src/speech_to_speech/language_model_handler.py

class LanguageModelHandler:
    def __init__(self, args: LanguageModelHandlerArguments):
        self.device = torch.device(args.lm_device)
        self.device_map = args.lm_device_map if self.device.type == "cuda" else None
        
        self.model = AutoModelForCausalLM.from_pretrained(
            args.lm_model_name,
            device_map=self.device_map,
            torch_dtype=torch.float16 if self.device.type == "cuda" else torch.float32,
        )
        
        # Explicit move when device_map is None (single GPU or CPU)

        if self.device_map is None:
            self.model = self.model.to(self.device)

```

**Critical implementation detail:** When `device_map="auto"` is set, `accelerate` handles placement and `.to(device)` must **not** be called—hence the conditional guard.

---

## Complete Configuration Example

Launch a distributed pipeline with explicit device assignment per component:

```bash
python -m speech_to_speech \
    --lm_device cuda:0 \
    --lm_device_map auto \
    --tts_device cuda:1 \
    --stt_device 0

```

For CPU-only inference:

```bash
python -m speech_to_speech \
    --lm_device cpu \
    --tts_device cpu \
    --stt_device -1

```

---

## Multi-GPU and Model Parallelism

The `lm_device_map` parameter enables **Hugging Face Accelerate** integration for large models exceeding single-GPU memory. Valid configurations from the source:

| Value | Behavior |
|-------|----------|
| `"auto"` | Automatic layer placement based on memory availability |
| `"balanced"` | Distribute layers evenly across visible GPUs |
| `"balanced_low_0"` | Reserve GPU 0 for generation, balance layers on others |
| Custom dict | Explicit layer-to-device mapping |

To combine pipeline parallel (different components on different GPUs) with model parallel (layers sharded within the LM):

```bash

# LM spans cuda:0 and cuda:1, TTS on cuda:2, STT on cuda:0

CUDA_VISIBLE_DEVICES=0,1,2 python -m speech_to_speech \
    --lm_device cuda \
    --lm_device_map auto \
    --tts_device cuda:2 \
    --stt_device 0

```

---

## Summary

- **Argument classes** in `arguments_classes/` define per-component device parameters with type-specific defaults (`str` for LM/TTS, `int` for STT)
- **Handlers** instantiate `torch.device` objects and conditionally apply `device_map` for Accelerate-compatible parallelism
- **Explicit device placement** prevents silent CPU fallback and enables multi-GPU pipeline architectures
- **Model parallelism** via `lm_device_map` operates orthogonally to component-level device assignment

---

## Frequently Asked Questions

### What happens if I specify conflicting device settings?

The handlers validate device availability but do not cross-check other components. Specifying `lm_device=cuda:0` and `tts_device=cuda:0` places both models on the same GPU, potentially causing out-of-memory errors. Use `nvidia-smi` monitoring and explicit ordinals (`cuda:0`, `cuda:1`) to distribute load.

### Why does `stt_device` use `int` while other devices use `str`?

The Whisper STT handler predates the unified device string convention in this codebase. As implemented in [`src/speech_to_speech/whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/whisper_stt_handler.py), it constructs `torch.device(f"cuda:{args.stt_device}")` internally, with `-1` mapping to CPU via conditional logic. This legacy pattern remains for backward compatibility.

### Can I use `device_map="auto"` with a specific GPU?

No—`device_map="auto"` delegates placement to Accelerate, which considers all visible CUDA devices. To restrict the LM to specific GPUs while using automatic sharding, set `CUDA_VISIBLE_DEVICES=0,1` before launching rather than specifying in `lm_device`.

### How do I verify my configuration loaded correctly?

Each handler logs device placement at `INFO` level on initialization. Enable verbose logging with `--logging_level info` to confirm tensor device alignment before the pipeline starts processing audio.