How to Use Text-to-Speech Models (VALL‑E‑X and GPT‑SoVITS) with ailia

Both VALL‑E‑X and GPT‑SoVITS are multilingual text‑to‑speech models converted to ONNX and wrapped for the ailia SDK, enabling offline, high‑quality speech synthesis and voice cloning via simple CLI commands or direct Python API calls.

The axinc-ai/ailia-models repository provides ready‑to‑run implementations for these state‑of‑the‑art TTS architectures. Each model ships with automated download logic, ailia Net wrappers, and command‑line interfaces that handle everything from text preprocessing to WAV file generation.

VALL‑E‑X Architecture and Implementation

VALL‑E‑X follows a dual‑stage generation pipeline that separates coarse token prediction from autoregressive refinement. The ailia implementation in audio_processing/vall-e-x/ orchestrates multiple ONNX subgraphs:

Stage	Component	ONNX Model File
NAR (Non‑autoregressive)	Coarse audio token prediction	nar_predict_layers.onnx
NAR	Audio embedding for voice cloning	nar_audio_embedding.onnx, nar_audio_embedding_layers.onnx
AR (Autoregressive)	Text, language, and speaker encoding	ar_text_embedding.onnx, ar_language_embedding.onnx, ar_audio_embedding.onnx
AR	Token sequence decoding	ar_decoder.onnx (or ar_decoder.opt.onnx for optimized inference)
Vocoder	Mel‑spectrogram to discrete tokens	encodec.onnx
Vocoder	Final waveform reconstruction	vocos.onnx
Utility	Positional embeddings	position_embedding.onnx

The high‑level logic resides in audio_processing/vall-e-x/vall-e-x.py, which parses CLI arguments, downloads missing models from https://storage.googleapis.com/ailia-models/, and invokes generate_audio() from audio_processing/vall-e-x/utils/generation.py. The latter function builds ailia Net objects for each ONNX file, executes the forward passes, and writes the resulting waveform to SAVE_WAV_PATH (default output.wav).

GPT‑SoVITS Architecture and Implementation

GPT‑SoVITS combines a GPT‑style linguistic encoder with a VITS neural vocoder to achieve zero‑shot voice cloning and cross‑lingual synthesis. The ailia port in audio_processing/gpt-sovits/ splits the pipeline into five ONNX subgraphs:

Sub‑module	Role	ONNX Model
SSL (Self‑Supervised Latent)	Extracts acoustic features from reference audio	ssl.onnx
T2S Encoder	Encodes text and optional speaker embeddings	t2s_encoder.onnx
T2S First Decoder	Generates initial coarse mel‑spectrogram	t2s_first_decoder.onnx
T2S Stage Decoder	Refines the mel‑spectrogram iteratively	t2s_stage_decoder.onnx
VITS	Final vocoder converting mel‑spectrogram to waveform	vits.onnx

The entry point audio_processing/gpt-sovits/gpt-sovits.py handles argument parsing, model downloading, and the inference loop. When --ailia_voice is specified, the script uses ailia_voice.G2P for grapheme‑to‑phoneme conversion. The pipeline executes as SSL feature extraction → T2S encoding/decoding → VITS vocoding, ultimately producing a WAV file at the path specified by --savepath.

Running Inference from the Command Line

Both models follow a common three‑step workflow: automatic model download, CLI argument parsing, and inference execution via ailia.

Prerequisites

Install the ailia SDK and model‑specific dependencies:

# Install ailia

pip install ailia

# VALL-E-X dependencies (primarily for G2P)

pip install -r audio_processing/vall-e-x/requirements.txt

# GPT-SoVITS dependencies (audio processing stack)

pip install -r audio_processing/gpt-sovits/requirements.txt

VALL‑E‑X Examples

Generate speech from plain text:

python3 audio_processing/vall-e-x/vall-e-x.py \
    --input "Hello world, this is a test of VALL‑E‑X." \
    --savepath my_output.wav

Perform zero‑shot voice cloning with a reference audio file:

python3 audio_processing/vall-e-x/vall-e-x.py \
    -i "音声合成のテストを行なっています。" \
    --audio BASIC5000_0001.wav \
    --transcript "水をマレーシアから買わなくてはならないのです" \
    -e 1

The -e flag enables emotion or speaker embedding extraction from the reference audio.

GPT‑SoVITS Examples

Basic Japanese synthesis:

python3 audio_processing/gpt-sovits/gpt-sovits.py \
    -i "音声合成のテストです。" \
    --savepath gpt_output.wav

Cross‑lingual voice cloning (English text with Japanese reference speaker):

python3 audio_processing/gpt-sovits/gpt-sovits.py \
    -i "Hello world, testing GPT‑SoVITS." \
    --ref_audio reference_audio_captured_by_ax.wav \
    --ref_text "水をマレーシアから買わなくてはならない。" \
    --ref_language ja \
    --text_language en \
    --savepath en_output.wav

Integrating with the ailia Python API

For advanced use cases—such as batch processing, custom preprocessing, or integration into larger applications—you can instantiate ailia Net objects directly and invoke the generation functions programmatically.

Direct Model Loading Example

import ailia
from audio_processing.vall_e_x.utils.generation import generate_audio

# Initialize ailia environment (0=CPU, 1=GPU)

env_id = 0  # Change to 1 for GPU acceleration

# Load a specific component (example: NAR decoder)

net = ailia.Net(
    weight='audio_processing/vall-e-x/nar_decoder.onnx',
    model='audio_processing/vall-e-x/nar_decoder.onnx.prototxt',
    env_id=env_id
)

# Generate audio using the high-level utility

wav_path = generate_audio(
    text="こんにちは、AIリアです。",
    prompt=None,          # Set to audio path for voice cloning

    language='ja',
    top_k=100
)
print(f"Generated waveform saved to: {wav_path}")

Note: This snippet assumes the ONNX files have already been cached locally by running the CLI script at least once, or you must manually download them from the REMOTE_PATH defined in the source.

Key Files and Repository Structure

Understanding the file layout helps when debugging, extending, or contributing to the models.

File	Purpose
audio_processing/vall-e-x/vall-e-x.py	CLI entry point for VALL‑E‑X; handles argument parsing, model download, and orchestrates inference.
audio_processing/vall-e-x/utils/generation.py	Core synthesis logic; builds ailia Net objects for each ONNX subgraph and runs the dual‑stage pipeline.
audio_processing/vall-e-x/requirements.txt	Python dependencies (e.g., pyopenjtalk for Japanese G2P).
audio_processing/gpt-sovits/gpt-sovits.py	CLI entry point for GPT‑SoVITS; manages the SSL → T2S → VITS inference loop.
audio_processing/gpt-sovits/requirements.txt	Audio processing stack including librosa, pyopenjtalk, and tqdm.

All ONNX weights are automatically downloaded from https://storage.googleapis.com/ailia-models/ on first run and cached locally.

Summary

• VALL‑E‑X and GPT‑SoVITS are fully implemented in the ailia-models repository as ONNX graphs wrapped by the ailia SDK.
• Both models support zero‑shot voice cloning by providing reference audio and transcripts via CLI flags (--audio, --ref_audio, --transcript).
• The dual‑stage pipeline of VALL‑E‑X (NAR + AR) and the hybrid GPT+VITS architecture of GPT‑SoVITS are abstracted into simple CLI commands that handle model download and inference automatically.
• For production integration, instantiate ailia.Net directly and call the generation utilities in utils/generation.py (VALL‑E‑X) or replicate the SSL→T2S→VITS loop (GPT‑SoVITS).

Frequently Asked Questions

How do I enable GPU acceleration for VALL‑E‑X or GPT‑SoVITS?

Pass env_id=1 when constructing the ailia.Net objects, or ensure your ailia runtime is configured to prefer GPU execution. The CLI scripts automatically detect and use GPU if available in the ailia environment.

What audio format should I use for voice cloning reference files?

The models accept standard WAV files with mono, 16‑bit, 24 kHz or 16 kHz sampling rates (depending on the model). The scripts in audio_processing/vall-e-x/ and audio_processing/gpt-sovits/ use librosa or soundfile to resample inputs automatically, but providing pre‑normalized 24 kHz mono WAV yields the best cloning fidelity.

Can I use these models for commercial applications?

The ailia‑models repository provides the inference code and ONNX conversion under the repository’s license (typically MIT), but the original model weights (VALL‑E‑X, GPT‑SoVITS) carry their own licenses. You must review the license terms of the original checkpoints (linked in each model’s README.md) to determine commercial usability.

How do I handle out‑of‑memory errors during inference?

Reduce the --max_len or --segment_size parameters if exposed by the CLI, or modify the top_k sampling value in generation.py to limit the autoregressive sequence length. For GPU runs, ensure your device has at least 4–6 GB VRAM; otherwise, force CPU execution by setting env_id=0 in the script or using the --env flag if available.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}