How VoiceStudio Routes TTS and ASR Across CUDA, MPS, ROCm, and CPU: Engine Selection Explained

VoiceStudio routes text-to-speech and automatic speech recognition workloads by first detecting the best available compute device, then matching tasks to workers that advertise compatible engine capabilities.

VoiceStudio implements a two-layer routing strategy that separates hardware detection from engine selection. This design ensures that TTS and ASR jobs run on the most capable hardware with an appropriate inference backend. Below is a complete breakdown of how the system prioritizes devices, validates compatibility, and dispatches tasks to the right worker.

---

Device Detection Hierarchy: CUDA, ROCm, Intel XPU, DirectML, MPS, and CPU

The foundation of VoiceStudio's routing logic is services.model_manager.get_best_device(), which probes the host and returns a standardized device string.

Priority Order from Source

The detection logic follows a strict priority ranking (see [backend/services/model_manager.py](https://github.com/debpalash/VoiceStudio/blob/main/backend/services/model_manager.py#L54-L56)):

• CUDA or ROCm (NVIDIA/AMD discrete GPUs)
• Intel XPU (Intel GPU accelerators)
• DirectML (Windows machine learning)
• Apple MPS (Apple Silicon neural engine)
• CPU (fallback)

from services.model_manager import get_best_device

device = get_best_device()
print(device)  # "cuda", "mps", "cpu", etc.

The function queries core.device_caps.detect_host_caps() for raw capability detection, then filters through the priority list to return the first available option.

---

GPU Compatibility Validation and Fallback

VoiceStudio does not blindly trust GPU detection. The check_device_compatibility() helper validates that the detected GPU actually works with the installed PyTorch build.

Forced CPU Fallback for Incompatible GPUs

When a CUDA-capable GPU is present but PyTorch lacks support for its compute capability, the system warns the user and returns "cpu" (see [backend/services/model_manager.py](https://github.com/debpalash/VoiceStudio/blob/main/backend/services/model_manager.py#L70-L88)):

# Simplified excerpt from get_best_device()

if not check_device_compatibility("cuda"):
    import warnings
    warnings.warn("CUDA GPU detected but unsupported by PyTorch build; falling back to CPU")
    return "cpu"

Users can override this safety check by setting OMNIVOICE_FORCE_CUDA=1 before startup.

ROCm-Specific Environment Handling

For AMD ROCm hosts, get_best_device() automatically sets HSA_OVERRIDE_GFX_VERSION when required for compatibility with specific GPU generations (see [backend/services/model_manager.py](https://github.com/debpalash/VoiceStudio/blob/main/backend/services/model_manager.py#L14-L20)).

---

Engine Selection Through Worker Capability Matching

Hardware selection is only the first half. VoiceStudio must also ensure the requested engine (e.g., indextts, whisper, faster-whisper) is available on the chosen device.

Task Structure and Engine Field

Every task submitted to the scheduler carries an engine identifier:

from backend.worker.scheduler import Task

task = Task(
    operation="asr",
    engine="faster-whisper",   # Must match a worker's advertised capability

    model_id="large-v3",
    audio_data=b"...",
)

Worker Capability Advertisement

Workers register themselves with a list of pb.ModelCapability objects. Each capability specifies both the engine name and the backend (device type) it supports (see [tests/test_worker_registry.py](https://github.com/debpalash/VoiceStudio/blob/main/tests/test_worker_registry.py#L314-L319)):

# Example capability from test suite

capability = pb.ModelCapability(
    engine="indextts",
    backend="cuda",        # "cuda", "mps", "cpu", etc.

    model_id="index-tts-v1",
    batch_size=8,
)

The Scheduler's Selection Algorithm

Scheduler.select_worker() implements the matching logic in two phases (see [backend/worker/scheduler.py](https://github.com/debpalash/VoiceStudio/blob/main/backend/worker/scheduler.py#L562-L567)):

1. Filter eligible workers — Must be idle and meet GPU/CPU requirements from task constraints
2. Select from capable workers — Must support the task's engine and model_id on the current device

# Simplified logic from select_worker()

eligible = [w for w in workers if w.is_idle() and w.meets_gpu_requirement(task)]
if not eligible:
    capable = [w for w in workers if w.supports(task.engine, task.model_id, device)]
    if not capable:
        raise NoEligibleWorker(f"No worker for engine={task.engine} on {device}")
    worker = self._rank_select(capable)  # priority or least-loaded

If no capable worker exists, the scheduler raises NoEligibleWorker with a descriptive message (see [backend/worker/scheduler.py](https://github.com/debpalash/VoiceStudio/blob/main/backend/worker/scheduler.py#L68-L92)).

---

TTS and ASR Backend Integration

Both TTS and ASR services consume the same device detection and scheduling infrastructure.

Shared Device Resolution

services.tts_backend and services.asr_backend import get_best_device() to determine where to run inference. The resulting device string propagates through the task's deadline_policy and into the worker's execution_device() method (see [backend/worker/scheduler.py](https://github.com/debpalash/VoiceStudio/blob/main/backend/worker/scheduler.py#L52-L55)):

• "cuda" → CUDA tensors, torch.cuda
• "mps" → Metal Performance Shaders
• "cpu" → Standard CPU inference

Engine Context Enforcement

The TTS backend uses an engine_in_use context manager to ensure thread-safe engine selection during model loading and inference. This prevents cross-engine interference when multiple TTS engines are registered on the same worker.

---

Complete Routing Example

from services.model_manager import get_best_device
from backend.worker.scheduler import Scheduler, Task

# 1. Hardware detection (runs once at startup or per-request)

device = get_best_device()  # Returns "cuda" on NVIDIA, "mps" on Apple Silicon, etc.

# 2. Task creation with explicit engine

scheduler = Scheduler(worker_registry=registry, strategy="least-loaded")
tts_task = Task(
    operation="tts",
    engine="indextts",
    model_id="index-tts-v1",
    params={"text": "VoiceStudio routes efficiently across hardware targets"},
    gpu_preference="any",  # or "cuda", "mps", "none"

)

# 3. Scheduler matches to capable worker

try:
    assigned_worker = scheduler.select_worker(tts_task)
    result = assigned_worker.execute(tts_task, execution_device=device)
except NoEligibleWorker as e:
    # No worker registered with indextts engine on detected device

    print(f"Routing failed: {e}")

---

Summary

• get_best_device() determines hardware priority: CUDA/ROCm → Intel XPU → DirectML → MPS → CPU, with GPU compatibility validation and CPU fallback
• Worker capabilities pair engine names with backend device types, enabling fine-grained routing
• Scheduler.select_worker() filters by eligibility, then matches task engine to worker capability on the detected device
• TTS and ASR backends share the same routing infrastructure, calling get_best_device() and respecting scheduler assignments
• Fallback mechanisms include forced CPU when GPU support is missing and environment variable injection for ROCm compatibility

---

Frequently Asked Questions

What happens if I have both CUDA and MPS available?

VoiceStudio prioritizes CUDA/ROCm first, then Intel XPU, DirectML, and finally MPS. Apple Silicon users with an external GPU must explicitly configure device preference if they want MPS over CUDA.

Can I force a specific device even if VoiceStudio detects a better one?

Set environment variables before startup: CUDA_VISIBLE_DEVICES="" to hide CUDA, or OMNIVOICE_FORCE_CUDA=1 to override compatibility checks. The scheduler also respects gpu_preference="none" in task parameters to force CPU.

How does VoiceStudio handle engine versioning?

Workers advertise model_id alongside engine in their capabilities. The scheduler matches both fields, ensuring you don't route a whisper-v2 task to a worker only capable of faster-whisper-v3.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}