How to Configure CUDA and Device Settings Across Components in Hugging Face Speech-to-Speech

To configure CUDA and device settings in the Hugging Face speech-to-speech pipeline, you assign device IDs through component-specific argument classes like LanguageModelHandlerArguments, then ensure each handler initializes its model with torch.device() and handles multi-GPU scheduling via device_map="auto" or explicit CUDA ordinal strings.

The huggingface/speech-to-speech repository implements a modular speech-to-speech translation pipeline where each component—language model, text-to-speech, and speech recognition—requires explicit device configuration for CUDA acceleration. Understanding the device cascade across these handlers ensures optimal GPU utilization and prevents cross-device tensor errors.

---

Primary Configuration Entry Point: Argument Classes

Device configuration flows through argument dataclasses defined in src/speech_to_speech/arguments_classes/. These classes parse command-line inputs and propagate device settings to their respective handlers.

Language Model Device Configuration

The LanguageModelHandlerArguments class in language_model_arguments.py exposes the lm_device parameter:

# From src/speech_to_speech/arguments_classes/language_model_arguments.py

@dataclass
class LanguageModelHandlerArguments:
    lm_device: str = field(
        default="cuda",
        metadata={"help": "The device to load the language model on. Defaults to 'cuda'."}
    )
    lm_device_map: str = field(
        default="auto",
        metadata={"help": "Device map for model parallelism. Use 'auto' for automatic placement."}
    )

Key parameters:

• lm_device — Accepts "cuda", "cuda:0", "cuda:1", or "cpu"
• lm_device_map — Passed to accelerate's device_map for model sharding across GPUs

Text-to-Speech Device Configuration

Similarly, MeloTTSHandlerArguments in tts_arguments.py controls TTS device placement:

# From src/speech_to_speech/arguments_classes/tts_arguments.py

@dataclass
class MeloTTSHandlerArguments:
    tts_device: str = field(
        default="cuda" if torch.cuda.is_available() else "cpu",
        metadata={"help": "Device for MeloTTS inference"}
    )

Speech Recognition Device Configuration

The Whisper-based ASR component uses WhisperSTTHandlerArguments:

# From src/speech_to_speech/arguments_classes/stt_arguments.py

@dataclass
class WhisperSTTHandlerArguments:
    stt_device: int = field(
        default=0,
        metadata={"help": "CUDA device ordinal for Whisper model (-1 for CPU)"}
    )

Note the type difference: stt_device uses int (ordinal) rather than str, requiring -1 for CPU fallback.

---

Handler-Level Device Initialization

Each handler converts argument values into torch.device objects during __init__. The LanguageModelHandler implementation in src/speech_to_speech/language_model_handler.py demonstrates this pattern:

# From src/speech_to_speech/language_model_handler.py

class LanguageModelHandler:
    def __init__(self, args: LanguageModelHandlerArguments):
        self.device = torch.device(args.lm_device)
        self.device_map = args.lm_device_map if self.device.type == "cuda" else None
        
        self.model = AutoModelForCausalLM.from_pretrained(
            args.lm_model_name,
            device_map=self.device_map,
            torch_dtype=torch.float16 if self.device.type == "cuda" else torch.float32,
        )
        
        # Explicit move when device_map is None (single GPU or CPU)

        if self.device_map is None:
            self.model = self.model.to(self.device)

Critical implementation detail: When device_map="auto" is set, accelerate handles placement and .to(device) must not be called—hence the conditional guard.

---

Complete Configuration Example

Launch a distributed pipeline with explicit device assignment per component:

python -m speech_to_speech \
    --lm_device cuda:0 \
    --lm_device_map auto \
    --tts_device cuda:1 \
    --stt_device 0

For CPU-only inference:

python -m speech_to_speech \
    --lm_device cpu \
    --tts_device cpu \
    --stt_device -1

---

Multi-GPU and Model Parallelism

The lm_device_map parameter enables Hugging Face Accelerate integration for large models exceeding single-GPU memory. Valid configurations from the source:

Value	Behavior
"auto"	Automatic layer placement based on memory availability
"balanced"	Distribute layers evenly across visible GPUs
"balanced_low_0"	Reserve GPU 0 for generation, balance layers on others
Custom dict	Explicit layer-to-device mapping

To combine pipeline parallel (different components on different GPUs) with model parallel (layers sharded within the LM):

# LM spans cuda:0 and cuda:1, TTS on cuda:2, STT on cuda:0

CUDA_VISIBLE_DEVICES=0,1,2 python -m speech_to_speech \
    --lm_device cuda \
    --lm_device_map auto \
    --tts_device cuda:2 \
    --stt_device 0

---

Summary

• Argument classes in arguments_classes/ define per-component device parameters with type-specific defaults (str for LM/TTS, int for STT)
• Handlers instantiate torch.device objects and conditionally apply device_map for Accelerate-compatible parallelism
• Explicit device placement prevents silent CPU fallback and enables multi-GPU pipeline architectures
• Model parallelism via lm_device_map operates orthogonally to component-level device assignment

---

Frequently Asked Questions

What happens if I specify conflicting device settings?

The handlers validate device availability but do not cross-check other components. Specifying lm_device=cuda:0 and tts_device=cuda:0 places both models on the same GPU, potentially causing out-of-memory errors. Use nvidia-smi monitoring and explicit ordinals (cuda:0, cuda:1) to distribute load.

Why does stt_device use int while other devices use str?

The Whisper STT handler predates the unified device string convention in this codebase. As implemented in src/speech_to_speech/whisper_stt_handler.py, it constructs torch.device(f"cuda:{args.stt_device}") internally, with -1 mapping to CPU via conditional logic. This legacy pattern remains for backward compatibility.

Can I use device_map="auto" with a specific GPU?

No—device_map="auto" delegates placement to Accelerate, which considers all visible CUDA devices. To restrict the LM to specific GPUs while using automatic sharding, set CUDA_VISIBLE_DEVICES=0,1 before launching rather than specifying in lm_device.

How do I verify my configuration loaded correctly?

Each handler logs device placement at INFO level on initialization. Enable verbose logging with --logging_level info to confirm tensor device alignment before the pipeline starts processing audio.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}