# How to Use Text-to-Speech Models (VALL‑E‑X and GPT‑SoVITS) with ailia

> Explore VALL-E-X and GPT-SoVITS TTS models with ailia. Achieve offline speech synthesis and voice cloning using simple CLI or Python API calls for high-quality audio.

- Repository: [axinc-ai/ailia-models](https://github.com/axinc-ai/ailia-models)
- Tags: tutorial
- Published: 2026-02-25

---

**Both VALL‑E‑X and GPT‑SoVITS are multilingual text‑to‑speech models converted to ONNX and wrapped for the ailia SDK, enabling offline, high‑quality speech synthesis and voice cloning via simple CLI commands or direct Python API calls.**

The `axinc-ai/ailia-models` repository provides ready‑to‑run implementations for these state‑of‑the‑art TTS architectures. Each model ships with automated download logic, ailia `Net` wrappers, and command‑line interfaces that handle everything from text preprocessing to WAV file generation.

## VALL‑E‑X Architecture and Implementation

VALL‑E‑X follows a **dual‑stage** generation pipeline that separates coarse token prediction from autoregressive refinement. The ailia implementation in `audio_processing/vall-e-x/` orchestrates multiple ONNX subgraphs:

| Stage | Component | ONNX Model File |
|-------|-----------|-----------------|
| **NAR** (Non‑autoregressive) | Coarse audio token prediction | `nar_predict_layers.onnx` |
| **NAR** | Audio embedding for voice cloning | `nar_audio_embedding.onnx`, `nar_audio_embedding_layers.onnx` |
| **AR** (Autoregressive) | Text, language, and speaker encoding | `ar_text_embedding.onnx`, `ar_language_embedding.onnx`, `ar_audio_embedding.onnx` |
| **AR** | Token sequence decoding | `ar_decoder.onnx` (or `ar_decoder.opt.onnx` for optimized inference) |
| **Vocoder** | Mel‑spectrogram to discrete tokens | `encodec.onnx` |
| **Vocoder** | Final waveform reconstruction | `vocos.onnx` |
| **Utility** | Positional embeddings | `position_embedding.onnx` |

The high‑level logic resides in [`audio_processing/vall-e-x/vall-e-x.py`](https://github.com/axinc-ai/ailia-models/blob/main/audio_processing/vall-e-x/vall-e-x.py), which parses CLI arguments, downloads missing models from `https://storage.googleapis.com/ailia-models/`, and invokes `generate_audio()` from [`audio_processing/vall-e-x/utils/generation.py`](https://github.com/axinc-ai/ailia-models/blob/main/audio_processing/vall-e-x/utils/generation.py). The latter function builds ailia `Net` objects for each ONNX file, executes the forward passes, and writes the resulting waveform to `SAVE_WAV_PATH` (default `output.wav`).

## GPT‑SoVITS Architecture and Implementation

GPT‑SoVITS combines a **GPT‑style linguistic encoder** with a **VITS neural vocoder** to achieve zero‑shot voice cloning and cross‑lingual synthesis. The ailia port in `audio_processing/gpt-sovits/` splits the pipeline into five ONNX subgraphs:

| Sub‑module | Role | ONNX Model |
|------------|------|------------|
| **SSL** (Self‑Supervised Latent) | Extracts acoustic features from reference audio | `ssl.onnx` |
| **T2S Encoder** | Encodes text and optional speaker embeddings | `t2s_encoder.onnx` |
| **T2S First Decoder** | Generates initial coarse mel‑spectrogram | `t2s_first_decoder.onnx` |
| **T2S Stage Decoder** | Refines the mel‑spectrogram iteratively | `t2s_stage_decoder.onnx` |
| **VITS** | Final vocoder converting mel‑spectrogram to waveform | `vits.onnx` |

The entry point [`audio_processing/gpt-sovits/gpt-sovits.py`](https://github.com/axinc-ai/ailia-models/blob/main/audio_processing/gpt-sovits/gpt-sovits.py) handles argument parsing, model downloading, and the inference loop. When `--ailia_voice` is specified, the script uses `ailia_voice.G2P` for grapheme‑to‑phoneme conversion. The pipeline executes as SSL feature extraction → T2S encoding/decoding → VITS vocoding, ultimately producing a WAV file at the path specified by `--savepath`.

## Running Inference from the Command Line

Both models follow a common three‑step workflow: automatic model download, CLI argument parsing, and inference execution via ailia.

### Prerequisites

Install the ailia SDK and model‑specific dependencies:

```bash

# Install ailia

pip install ailia

# VALL-E-X dependencies (primarily for G2P)

pip install -r audio_processing/vall-e-x/requirements.txt

# GPT-SoVITS dependencies (audio processing stack)

pip install -r audio_processing/gpt-sovits/requirements.txt

```

### VALL‑E‑X Examples

Generate speech from plain text:

```bash
python3 audio_processing/vall-e-x/vall-e-x.py \
    --input "Hello world, this is a test of VALL‑E‑X." \
    --savepath my_output.wav

```

Perform zero‑shot voice cloning with a reference audio file:

```bash
python3 audio_processing/vall-e-x/vall-e-x.py \
    -i "音声合成のテストを行なっています。" \
    --audio BASIC5000_0001.wav \
    --transcript "水をマレーシアから買わなくてはならないのです" \
    -e 1

```

The `-e` flag enables emotion or speaker embedding extraction from the reference audio.

### GPT‑SoVITS Examples

Basic Japanese synthesis:

```bash
python3 audio_processing/gpt-sovits/gpt-sovits.py \
    -i "音声合成のテストです。" \
    --savepath gpt_output.wav

```

Cross‑lingual voice cloning (English text with Japanese reference speaker):

```bash
python3 audio_processing/gpt-sovits/gpt-sovits.py \
    -i "Hello world, testing GPT‑SoVITS." \
    --ref_audio reference_audio_captured_by_ax.wav \
    --ref_text "水をマレーシアから買わなくてはならない。" \
    --ref_language ja \
    --text_language en \
    --savepath en_output.wav

```

## Integrating with the ailia Python API

For advanced use cases—such as batch processing, custom preprocessing, or integration into larger applications—you can instantiate ailia `Net` objects directly and invoke the generation functions programmatically.

### Direct Model Loading Example

```python
import ailia
from audio_processing.vall_e_x.utils.generation import generate_audio

# Initialize ailia environment (0=CPU, 1=GPU)

env_id = 0  # Change to 1 for GPU acceleration

# Load a specific component (example: NAR decoder)

net = ailia.Net(
    weight='audio_processing/vall-e-x/nar_decoder.onnx',
    model='audio_processing/vall-e-x/nar_decoder.onnx.prototxt',
    env_id=env_id
)

# Generate audio using the high-level utility

wav_path = generate_audio(
    text="こんにちは、AIリアです。",
    prompt=None,          # Set to audio path for voice cloning

    language='ja',
    top_k=100
)
print(f"Generated waveform saved to: {wav_path}")

```

> **Note:** This snippet assumes the ONNX files have already been cached locally by running the CLI script at least once, or you must manually download them from the `REMOTE_PATH` defined in the source.

## Key Files and Repository Structure

Understanding the file layout helps when debugging, extending, or contributing to the models.

| File | Purpose |
|------|---------|
| [`audio_processing/vall-e-x/vall-e-x.py`](https://github.com/axinc-ai/ailia-models/blob/main/audio_processing/vall-e-x/vall-e-x.py) | CLI entry point for VALL‑E‑X; handles argument parsing, model download, and orchestrates inference. |
| [`audio_processing/vall-e-x/utils/generation.py`](https://github.com/axinc-ai/ailia-models/blob/main/audio_processing/vall-e-x/utils/generation.py) | Core synthesis logic; builds ailia `Net` objects for each ONNX subgraph and runs the dual‑stage pipeline. |
| [`audio_processing/vall-e-x/requirements.txt`](https://github.com/axinc-ai/ailia-models/blob/main/audio_processing/vall-e-x/requirements.txt) | Python dependencies (e.g., `pyopenjtalk` for Japanese G2P). |
| [`audio_processing/gpt-sovits/gpt-sovits.py`](https://github.com/axinc-ai/ailia-models/blob/main/audio_processing/gpt-sovits/gpt-sovits.py) | CLI entry point for GPT‑SoVITS; manages the SSL → T2S → VITS inference loop. |
| [`audio_processing/gpt-sovits/requirements.txt`](https://github.com/axinc-ai/ailia-models/blob/main/audio_processing/gpt-sovits/requirements.txt) | Audio processing stack including `librosa`, `pyopenjtalk`, and `tqdm`. |

All ONNX weights are automatically downloaded from `https://storage.googleapis.com/ailia-models/` on first run and cached locally.

## Summary

- **VALL‑E‑X** and **GPT‑SoVITS** are fully implemented in the `ailia-models` repository as ONNX graphs wrapped by the ailia SDK.
- Both models support **zero‑shot voice cloning** by providing reference audio and transcripts via CLI flags (`--audio`, `--ref_audio`, `--transcript`).
- The **dual‑stage pipeline** of VALL‑E‑X (NAR + AR) and the **hybrid GPT+VITS** architecture of GPT‑SoVITS are abstracted into simple CLI commands that handle model download and inference automatically.
- For production integration, instantiate `ailia.Net` directly and call the generation utilities in [`utils/generation.py`](https://github.com/axinc-ai/ailia-models/blob/main/utils/generation.py) (VALL‑E‑X) or replicate the SSL→T2S→VITS loop (GPT‑SoVITS).

## Frequently Asked Questions

### How do I enable GPU acceleration for VALL‑E‑X or GPT‑SoVITS?

Pass `env_id=1` when constructing the `ailia.Net` objects, or ensure your ailia runtime is configured to prefer GPU execution. The CLI scripts automatically detect and use GPU if available in the ailia environment.

### What audio format should I use for voice cloning reference files?

The models accept standard WAV files with mono, 16‑bit, 24 kHz or 16 kHz sampling rates (depending on the model). The scripts in `audio_processing/vall-e-x/` and `audio_processing/gpt-sovits/` use `librosa` or `soundfile` to resample inputs automatically, but providing pre‑normalized 24 kHz mono WAV yields the best cloning fidelity.

### Can I use these models for commercial applications?

The ailia‑models repository provides the inference code and ONNX conversion under the repository’s license (typically MIT), but the original model weights (VALL‑E‑X, GPT‑SoVITS) carry their own licenses. You must review the license terms of the original checkpoints (linked in each model’s [`README.md`](https://github.com/axinc-ai/ailia-models/blob/main/README.md)) to determine commercial usability.

### How do I handle out‑of‑memory errors during inference?

Reduce the `--max_len` or `--segment_size` parameters if exposed by the CLI, or modify the `top_k` sampling value in [`generation.py`](https://github.com/axinc-ai/ailia-models/blob/main/generation.py) to limit the autoregressive sequence length. For GPU runs, ensure your device has at least 4–6 GB VRAM; otherwise, force CPU execution by setting `env_id=0` in the script or using the `--env` flag if available.