# Optimizations for Faster Inference on CPU vs GPU in Real-Time Voice Cloning

> Discover CPU vs GPU optimizations for faster real-time voice cloning inference. Learn how the Real-Time Voice Cloning repo uses CUDA batching and thread control for peak performance.

- Repository: [Corentin Jemine/Real-Time-Voice-Cloning](https://github.com/CorentinJ/Real-Time-Voice-Cloning)
- Tags: performance
- Published: 2026-03-05

---

**The Real-Time Voice Cloning repository automatically detects your hardware and applies platform-specific optimizations, including CUDA batching for GPUs and thread-controlled CPU execution, via runtime checks in [`vocoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/inference.py).**

The CorentinJ/Real-Time-Voice-Cloning toolkit implements a three-stage deep learning pipeline (encoder, synthesizer, and vocoder) that requires distinct optimizations for faster inference on CPU versus GPU. The codebase handles automatic device selection at runtime through `torch.cuda.is_available()` checks while exposing configuration flags like `--cpu` and `batched` that significantly impact performance characteristics. Understanding these implementation details ensures minimal latency whether running on CUDA-enabled GPUs or CPU-only servers.

## GPU-Oriented Optimizations

NVIDIA GPU acceleration leverages batch processing and CUDA-specific PyTorch builds to maximize throughput in the Wave-RNN vocoder and Tacotron synthesizer.

### Automatic CUDA Device Detection

All major components check for CUDA availability before loading weights. In [`vocoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/inference.py) (lines 28-32), the code automatically moves models to GPU when available:

```python
if torch.cuda.is_available():
    _model = _model.cuda()

```

This pattern ensures that the **encoder**, **synthesizer**, and **vocoder** all reside on CUDA memory when possible, eliminating CPU-GPU transfer bottlenecks during inference.

### Batched Generation for Parallel Processing

The `infer_waveform` function in [`vocoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/inference.py) (lines 45-64) accepts a `batched` parameter that enables parallel frame processing:

```python
def infer_waveform(..., batched=True, ...)

```

Setting `batched=True` processes multiple frames simultaneously, fully utilizing GPU parallelism and dramatically reducing inference time for the Wave-RNN vocoder.

### CUDA-Specific PyTorch Installation

The repository defines optional dependency groups in [`pyproject.toml`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/pyproject.toml) (lines 28-32). Installing with the `cuda` extra pulls a GPU-enabled PyTorch build:

```bash
uv run --extra cuda demo_toolbox.py

```

This ensures compatibility with NVIDIA drivers and enables the `.cuda()` calls throughout the inference pipeline.

## CPU-Oriented Optimizations

When CUDA is unavailable or disabled, the toolkit switches to CPU-optimized execution paths that minimize memory overhead and control thread allocation.

### CPU-Only PyTorch Wheels

The `cpu` extra in [`pyproject.toml`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/pyproject.toml) installs a lightweight CPU-only PyTorch build, removing GPU library overhead:

```bash
uv run --extra cpu demo_cli.py --cpu

```

This reduces installation size and eliminates CUDA initialization overhead on headless servers or machines without NVIDIA hardware.

### Explicit CPU Flag and Device Selection

The demo scripts [`demo_cli.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/demo_cli.py) and [`demo_toolbox.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/demo_toolbox.py) (lines 21-31) expose a `--cpu` flag that forces CPU execution even when GPUs are present. When activated, the code sets `_device = torch.device('cpu')` and bypasses all `.cuda()` calls, keeping tensors in system RAM.

### Disabling Batching for Low-Core Systems

For single-core or memory-constrained CPUs, set `batched=False` when calling `infer_waveform`:

```python
wav = infer_waveform(mel_spectrogram, batched=False)

```

As implemented in [`vocoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/inference.py) (lines 45-64), this eliminates the overhead of building large intermediate batches, reducing RAM usage and improving speed on limited hardware.

### Thread Control via Environment Variables

While not hard-coded in the repository, CPU builds respect standard PyTorch environment variables. Set `OMP_NUM_THREADS` or `MKL_NUM_THREADS` before launching to cap thread usage:

```bash
OMP_NUM_THREADS=4 uv run --extra cpu demo_cli.py --cpu

```

This prevents oversubscription on shared machines and ensures predictable performance.

## Practical Implementation Examples

Configure your environment based on available hardware using these device-specific patterns.

### GPU-Accelerated Inference

For maximum speed on NVIDIA hardware:

```python
from vocoder.inference import load_model, infer_waveform

# Automatically uses CUDA if available

load_model("pretrained/vocoder.pt")
wav = infer_waveform(mel_spectrogram, batched=True)

```

### CPU-Only Execution

For servers without GPUs:

```python
from vocoder.inference import load_model, infer_waveform

# Run with --cpu flag or when CUDA unavailable

load_model("pretrained/vocoder.pt")
wav = infer_waveform(mel_spectrogram, batched=False)

```

### Command-Line Usage

Launch the toolbox with hardware-specific extras:

```bash

# GPU (default if CUDA present)

uv run --extra cuda demo_toolbox.py

# CPU only

uv run --extra cpu demo_toolbox.py --cpu

```

## Summary

- **Automatic device detection** in [`vocoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/inference.py) selects GPU via `.cuda()` when `torch.cuda.is_available()` returns True, otherwise defaults to CPU.
- **Batched processing** (`batched=True`) maximizes GPU utilization for the Wave-RNN vocoder, while `batched=False` reduces memory overhead on CPU-only systems.
- **Installation optimization**: Use `--extra cuda` for GPU-enabled PyTorch wheels or `--extra cpu` for lightweight CPU-only builds.
- **Explicit CPU control**: The `--cpu` flag in [`demo_cli.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/demo_cli.py) and [`demo_toolbox.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/demo_toolbox.py) (lines 21-31) forces CPU execution and disables CUDA-specific logging.
- **Thread management**: Control CPU parallelism using `OMP_NUM_THREADS` environment variables to prevent core oversubscription on shared hardware.

## Frequently Asked Questions

### How does Real-Time Voice Cloning automatically select between CPU and GPU?

The code checks `torch.cuda.is_available()` at runtime in [`vocoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/inference.py) (lines 28-32). When returning True, models move to GPU memory via `.cuda()`; otherwise, the device remains on CPU. This happens automatically during `load_model()` calls without requiring manual intervention.

### Should I use `batched=True` when running inference on CPU?

Generally no. While `batched=True` improves GPU throughput by processing multiple frames in parallel, it adds memory overhead that can slow down CPU-only systems. For faster inference on CPU versus GPU configurations with limited cores or RAM, set `batched=False` in the `infer_waveform` call as implemented in lines 45-64 of [`vocoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/inference.py).

### What is the difference between the `cpu` and `cuda` installation extras?

The [`pyproject.toml`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/pyproject.toml) defines these extras (lines 28-32) to install different PyTorch wheels. The `cuda` extra includes GPU-enabled PyTorch builds supporting `.cuda()` calls and CUDA kernels, while the `cpu` extra installs a lightweight CPU-only version removing GPU library dependencies and reducing binary size.

### How do I force CPU mode even when a GPU is available?

Pass the `--cpu` flag when running [`demo_cli.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/demo_cli.py) or [`demo_toolbox.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/demo_toolbox.py) (lines 21-31). This sets the internal device to CPU and prevents the automatic `.cuda()` calls from executing, ensuring all tensor operations remain in system memory regardless of NVIDIA hardware presence.