# How VoiceStudio Handles Different Hardware for Compute: GPU, MPS, and CPU Detection Explained

> VoiceStudio intelligently detects and utilizes NVIDIA GPU CUDA, Apple Silicon MPS, or CPU for optimal performance. Learn how our hardware abstraction layer maximizes your compute resources.

- Repository: [Palash Debnath/VoiceStudio](https://github.com/debpalash/VoiceStudio)
- Tags: internals
- Published: 2026-09-05

---

**VoiceStudio uses a hardware-abstraction layer that probes for NVIDIA CUDA, Apple Silicon MPS, and CPU fallback at runtime, then sizes a GPU-worker pool based on available VRAM or explicit environment overrides.**

The VoiceStudio codebase implements a **hardware-adaptive inference pipeline** that automatically detects the best available compute device and configures a safe, concurrent worker pool. According to the VoiceStudio source code, the system collapses the complexity of multi-backend PyTorch execution into a single `_pick_gpu_workers()` routine that balances performance with stability across disparate hardware configurations.

---

## Automatic Hardware Detection in model_manager.py

The heart of VoiceStudio's compute adaptation lives in [`backend/services/model_manager.py`](https://github.com/debpalash/VoiceStudio/blob/main/backend/services/model_manager.py). The `_pick_gpu_workers()` function performs a cascading detection sequence:

1. **Environment override** – `OMNIVOICE_GPU_WORKERS` takes precedence if set
2. **CUDA/ROCm probing** – `torch.cuda.is_available()` triggers VRAM-based sizing
3. **Apple MPS detection** – `torch.backends.mps.is_available()` yields a single worker
4. **CPU fallback** – any failure path results in one CPU worker

In [`model_manager.py`](https://github.com/debpalash/VoiceStudio/blob/main/model_manager.py) lines 85-97, the detection logic boots as follows:

```python
def _pick_gpu_workers() -> int:
    """Return number of GPU workers to spawn (1‑4) based on free VRAM."""
    # 1. explicit env override

    env = os.getenv("OMNIVOICE_GPU_WORKERS")
    if env:
        try:
            n = max(1, min(16, int(env)))
            return n
        except ValueError:
            pass

    # 2. CUDA / ROCm path

    if torch.cuda.is_available():
        free, total = torch.cuda.mem_get_info()
        free_gb = free / 1024**3
        # each job预留 ~5 GB

        workers = max(1, int(free_gb // _GPU_VRAM_PER_JOB_GB))
        return min(workers, 4)   # hard ceiling at 4

    # 3. Apple Silicon MPS

    if torch.backends.mps.is_available():
        return 1   # unified memory; single worker avoids thrash

    # 4. CPU fallback

    return 1

```

Source: [`model_manager.py#L85-L97`](https://github.com/debpalash/VoiceStudio/blob/main/backend/services/model_manager.py#L85-L97)

---

## VRAM Budgeting and Worker Pool Sizing

VoiceStudio prevents out-of-memory crashes through **conservative VRAM accounting**. The constant `_GPU_VRAM_PER_JOB_GB` (set to ~5 GB in lines 98-107) determines how many concurrent TTS jobs fit in free GPU memory:

| GPU VRAM | Calculated Workers | Hard Cap |
|----------|-------------------|----------|
| 8 GB | 1 | 1 |
| 12 GB | 2 | 2 |
| 16 GB | 3 | 3 |
| 24+ GB | 4+ → 4 | 4 |

This arithmetic occurs at lines 98-107:

```python
_GPU_VRAM_PER_JOB_GB: float = 5.0   # conservative headroom for XTTS etc.

def _pick_gpu_workers() -> int:
    ...
    if torch.cuda.is_available():
        free, total = torch.cuda.mem_get_info()
        free_gb = free / 1024**3
        workers = max(1, int(free_gb // _GPU_VRAM_PER_JOB_GB))
        return min(workers, 4)   # empirical: >4 workers rarely faster

```

Source: [`model_manager.py#L98-L107`](https://github.com/debpalash/VoiceStudio/blob/main/backend/services/model_manager.py#L98-L107)

---

## Environment Variable Overrides

Operators can **pin worker counts** regardless of auto-detection. The `OMNIVOICE_GPU_WORKERS` variable accepts integers 1-16, clamped to that range. This is essential for:

- **Containerized deployments** with fractional GPU visibility
- **Performance benchmarking** across fixed concurrency levels
- **Multi-tenant nodes** where VRAM reports are misleading

Implementation at lines 77-82:

```python
env = os.getenv("OMNIVOICE_GPU_WORKERS")
if env:
    try:
        n = max(1, min(16, int(env)))
        logger.info("OMNIVOICE_GPU_WORKERS override: %d", n)
        return n
    except ValueError:
        logger.warning("Invalid OMNIVOICE_GPU_WORKERS value: %r", env)

```

Source: [`model_manager.py#L77-L82`](https://github.com/debpalash/VoiceStudio/blob/main/backend/services/model_manager.py#L77-L82)

---

## Admission Control and Pool Protection

VoiceStudio implements **admission control** to shed load before GPU queue saturation. The `check_gpu_admission()` function (lines 110-126) compares pending queue depth against the computed worker count:

```python
class GpuPoolBusyError(Exception):
    """Raised when GPU workers are fully utilized."""
    retry_after: int = 30

def check_gpu_admission(*, what: str) -> None:
    """Raise if pool saturated; caller should HTTP 503 + Retry-After."""
    global _gpu_pool, _gpu_workers
    if _gpu_pool is None:
        _init_gpu_pool()
    qsize = _gpu_pool._work_queue.qsize()  # type: ignore

    if qsize >= _gpu_workers:
        raise GpuPoolBusyError(f"{what}: GPU pool busy ({qsize}/{_gpu_workers})")

```

Source: [`model_manager.py#L110-L126`](https://github.com/debpalash/VoiceStudio/blob/main/backend/services/model_manager.py#L110-L126)

---

## Phase‑Separated Timeout Model

Long-running TTS jobs are protected by **two timeout tiers**:

| Phase | Default | Purpose |
|-------|---------|---------|
| **Queue wait** (`GPU_QUEUE_TIMEOUT_S`) | 1800 s | Prevent infinite stall behind other jobs |
| **Execution** (`GPU_JOB_TIMEOUT_S`) | 300 s | Kill hung inference, trigger pool reset |

The guard wrapper at lines 66-84 and 124-140 implements this:

```python
async def run_on_gpu_pool_guarded(
    coro_factory: Callable[[], Awaitable[T]],
    *,
    what: str,
) -> T:
    """Run coroutine on GPU pool with separate queue+exec timeouts."""
    global _gpu_pool
    if _gpu_pool is None:
        _init_gpu_pool()

    # Phase 1: queue admission with long timeout

    try:
        fut = await asyncio.wait_for(
            _gpu_pool.submit(coro_factory),  # type: ignore

            timeout=GPU_QUEUE_TIMEOUT_S,
        )
    except asyncio.TimeoutError as e:
        raise GpuPoolBusyError(f"{what}: queue timeout after {GPU_QUEUE_TIMEOUT_S}s") from e

    # Phase 2: actual execution with tighter timeout

    try:
        return await asyncio.wait_for(fut, timeout=GPU_JOB_TIMEOUT_S)
    except asyncio.TimeoutError as e:
        logger.exception("GPU job hung; resetting pool: %s", what)
        _reset_gpu_pool()   # poisoned worker recovery

        raise GpuJobTimeoutError(f"{what}: execution timeout after {GPU_JOB_TIMEOUT_S}s") from e

```

Source: [`model_manager.py#L66-L84`](https://github.com/debpalash/VoiceStudio/blob/main/backend/services/model_manager.py#L66-L84) and [`model_manager.py#L124-L140`](https://github.com/debpalash/VoiceStudio/blob/main/backend/services/model_manager.py#L124-L140)

---

## CPU Fallback and Unified Timeouts

When no accelerator is detected, VoiceStudio degrades gracefully to **CPU inference**. The same pool abstraction applies, but timeout defaults shift to `OMNIVOICE_CPU_GENERATE_TIMEOUT_S` (default 600 s). Lines 49-57 handle this branch:

```python
CPU_TIMEOUT: int = int(os.getenv("OMNIVOICE_CPU_GENERATE_TIMEOUT_S", "600"))

def _init_gpu_pool() -> None:
    """Lazy‑initialize the process pool."""
    global _gpu_pool, _gpu_workers
    _gpu_workers = _pick_gpu_workers()
    mp_context = multiprocessing.get_context("spawn")
    _gpu_pool = ProcessPoolExecutor(
        max_workers=_gpu_workers,
        mp_context=mp_context,
        initializer=_init_worker_cuda,   # no‑op on CPU/MPS

    )
    logger.info("GPU pool initialized: %d workers", _gpu_workers)

```

Source: [`model_manager.py#L49-L57`](https://github.com/debpalash/VoiceStudio/blob/main/backend/services/model_manager.py#L49-L57)

---

## Practical Usage Examples

### Override Workers for Container Deployment

```python
import os
os.environ["OMNIVOICE_GPU_WORKERS"] = "2"  # force 2 workers on visible 16 GB GPU

from services.model_manager import _pick_gpu_workers, check_gpu_admission

workers = _pick_gpu_workers()
print(f"Pinned to {workers} GPU workers")  # → 2

```

### Adaptive Device Selection

```python
from services.model_manager import _pick_gpu_workers, run_on_gpu_pool_guarded

workers = _pick_gpu_workers()  # auto-detect: 1-4 GPU, 1 MPS, or 1 CPU

print(f"Hardware selected {workers} workers")

async def tts_infer(text: str) -> bytes:
    async def heavy():
        # XTTS or similar inference

        return b"<audio_blob>"
    return await run_on_gpu_pool_guarded(heavy, what=f"tts:{text[:20]}...")

```

### Handling Pool Saturation

```python
from services.model_manager import check_gpu_admission, GpuPoolBusyError

def request_handler(text: str):
    try:
        check_gpu_admission(what="batch_tts")
        # enqueue to pool...

    except GpuPoolBusyError as e:
        return {"error": "GPU saturated", "retry_after": e.retry_after}, 503

```

---

## Key Source Files

| File | Responsibility |
|------|---------------|
| [`backend/services/model_manager.py`](https://github.com/debpalash/VoiceStudio/blob/main/backend/services/model_manager.py) | Hardware detection, worker pool construction, admission control, timeout handling |
| [`backend/services/tts_backend.py`](https://github.com/debpalash/VoiceStudio/blob/main/backend/services/tts_backend.py) | Consumes the pool via `run_on_gpu_pool_guarded()` for TTS inference |
| [`core/device_caps.py`](https://github.com/debpalash/VoiceStudio/blob/main/core/device_caps.py) *(submodule)* | `detect_host_caps()` used for timeout heuristics across CPU/CUDA/MPS |
| [`tests/test_why_no_gpu_1274.py`](https://github.com/debpalash/VoiceStudio/blob/main/tests/test_why_no_gpu_1274.py) | Validates fallback behavior when GPU tools are absent |

---

## Summary

- **Automatic detection** probes CUDA → MPS → CPU in cascade, with single-call `torch` availability checks
- **VRAM budgeting** sizes worker pools conservatively at ~5 GB per job, capped at 4 workers
- **Environment override** via `OMNIVOICE_GPU_WORKERS` bypasses auto-detection for operator control
- **Admission control** rejects requests before queue overflow, enabling client-side retries
- **Phase-separated timeouts** isolate queue stalls from hung execution, with automatic pool reset on failure
- **CPU fallback** preserves identical semantics for accelerator-less hosts, extended timeout permitting

---

## Frequently Asked Questions

### How does VoiceStudio detect Apple Silicon GPUs?

VoiceStudio checks `torch.backends.mps.is_available()` in [`model_manager.py`](https://github.com/debpalash/VoiceStudio/blob/main/model_manager.py) lines 95-97. When true, it returns exactly one worker because MPS uses unified system memory rather than discrete VRAM, making multi-worker contention counterproductive.

### Can I force CPU mode even when a GPU is present?

There is no explicit "CPU-only" flag; however, setting `OMNIVOICE_GPU_WORKERS=1` and ensuring `CUDA_VISIBLE_DEVICES=""` or running without CUDA drivers will trigger the CPU fallback path. The detection logic treats unavailable CUDA as CPU in lines 85-97.

### Why is the worker cap set at 4?

The hard ceiling of 4 workers (line 107) reflects empirical observations that XTTS and similar models show diminishing returns beyond 4 concurrent streams on consumer GPUs, while increasing VRAM fragmentation risk. Data-center cards may override via environment variable if testing demonstrates higher efficiency.

### What happens when a GPU job hangs?

The `run_on_gpu_pool_guarded()` wrapper enforces `GPU_JOB_TIMEOUT_S` (default 300 s). Upon timeout, it logs the stack trace, calls `_reset_gpu_pool()` to terminate and recreate the process pool, and raises `GpuJobTimeoutError` for upstream handling. This prevents permanent pool poisoning from CUDA errors or driver stalls.