How to Use Whisper for Speech-to-Text Transcription with ailia-models

You can transcribe audio files or stream microphone input using the Whisper implementation in ailia-models by executing the CLI with --input or -V flags, or by importing the Python API from audio_processing/whisper/whisper.py to run encoder-decoder inference with beam search or greedy decoding.

The ailia-models repository by AXINC provides a production-ready implementation of OpenAI's Whisper architecture for speech-to-text transcription. Located under audio_processing/whisper/, this implementation wraps ONNX encoder and decoder models with the ailia SDK (or ONNX Runtime), handling audio preprocessing, language detection, and beam-search decoding. Whether you need to transcribe pre-recorded files or stream from a microphone, the repository offers both command-line and programmatic interfaces.

Architecture and Core Components

The Whisper pipeline in ailia-models consists of distinct modules responsible for audio processing, tokenization, and neural inference.

Audio Preprocessing Pipeline

Audio files are processed in audio_processing/whisper/audio_utils.py. The load_audio function reads input files using librosa (or ffmpeg if enabled) and resamples to 16 kHz. The pad_or_trim function ensures the waveform matches the 30-second chunk size (N_SAMPLES = 480000), while log_mel_spectrogram computes the 80-bin Mel spectrogram required by the encoder.

Model Inference Engine

The core inference logic resides in audio_processing/whisper/whisper.py. The get_audio_features function runs the encoder ONNX model to produce audio embeddings. Decoding is handled by inference_logits and the decode function, which implements KV-cache management, logit filtering (timestamp suppression, blank token removal), and beam search or greedy sampling strategies. The decode_with_fallback function automatically retries with higher temperature settings when compression ratios indicate repetitive or low-probability outputs.

Tokenization and Language Detection

The tokenizer.py and ailia_simple_tokenizer.py files manage the 51,865-token vocabulary, language token mappings, and task-specific prefixes (transcribe vs. translate). Language detection occurs automatically during the initial decoding steps unless explicitly overridden via command-line arguments.

Installation and Model Setup

Install the required dependencies and allow the SDK to download model weights on first run.

pip install ailia librosa tqdm

# Optional: for ONNX Runtime backend instead of ailia SDK

pip install onnxruntime

The check_and_download_models function in whisper.py (lines 1199-1202) automatically fetches encoder and decoder ONNX files from https://storage.googleapis.com/ailia-models/whisper/ if they are not present locally. Available model sizes include tiny, base, small, medium, large, and turbo, with corresponding files named encoder_{size}.onnx and decoder_{size}.onnx.

Command-Line Interface for Speech-to-Text

The CLI entry point in whisper.py (lines 1263-1272) supports file-based and real-time microphone transcription.

Transcribe Audio Files

# Basic transcription with default small model

python -m audio_processing.whisper.whisper --input recording.wav

# High-accuracy transcription with large model

python -m audio_processing.whisper.whisper --model_type large --input meeting.wav

# Translate Japanese audio to English text

python -m audio_processing.whisper.whisper --task translate --language ja --input japanese_speech.wav

# Beam search for more stable results

python -m audio_processing.whisper.whisper --beam_size 5 --temperature 0 --input podcast.wav

Real-Time Microphone Input

Enable microphone mode with the -V flag. The recognize_from_microphone function captures audio streams and processes them through the same predict pipeline used for file input.

python -m audio_processing.whisper.whisper -V --model_type base

Python API Integration

Embed Whisper transcription directly into Python applications using the high-level and low-level APIs exposed in whisper.py.

High-Level Inference Example

from ailia import Net
from audio_processing.whisper.whisper import (
    check_and_download_models,
    load_audio,
    log_mel_spectrogram,
    recognize_from_audio,
    get_args_parser,
)

# Configure model parameters

model_type = "small"
remote_path = "https://storage.googleapis.com/ailia-models/whisper/"

# Download models if needed

enc_files = (f"encoder_{model_type}.onnx", f"encoder_{model_type}.onnx.prototxt")
dec_files = (f"decoder_{model_type}.onnx", f"decoder_{model_type}.onnx.prototxt")
check_and_download_models(enc_files[0], enc_files[1], remote_path)
check_and_download_models(dec_files[0], dec_files[1], remote_path)

# Initialize ailia networks (env_id=0 for GPU, 1 for CPU)

enc_net = Net(enc_files[1], enc_files[0], env_id=0)
dec_net = Net(dec_files[1], dec_files[0], env_id=0)

# Load and preprocess audio

wav = load_audio("interview.wav")
mel = log_mel_spectrogram(wav, n_mels=80)

# Run transcription

result = recognize_from_audio(enc_net, dec_net)
print("Transcription:", result["text"])

# Access segment-level details with timestamps

for segment in result["segments"]:
    print(f"[{segment['start']:.2f}s - {segment['end']:.2f}s] {segment['text']}")

Direct Prediction API

For lower-level control, use the predict function directly (lines 885-889 in whisper.py):

from audio_processing.whisper.whisper import predict

# wav is a numpy array of float32 audio at 16kHz

result = predict(wav, enc_net, dec_net, immediate=True, task="transcribe")

Key Source Files Reference

File	Purpose	Direct Link
audio_processing/whisper/whisper.py	Main inference script, CLI entry point, model download logic, beam search decoding	View on GitHub
audio_processing/whisper/audio_utils.py	Audio loading, resampling to 16kHz, padding/trimming to 30s chunks, log-Mel spectrogram computation	View on GitHub
audio_processing/whisper/tokenizer.py	Multilingual tokenizer, language token mappings, task prefixes (transcribe/translate)	View on GitHub
audio_processing/whisper/ailia_simple_tokenizer.py	Fallback tokenizer when ailia tokenizer is disabled	View on GitHub

Performance Optimization and Troubleshooting

Issue	Root Cause	Solution
Model download failures	Initial run requires fetching ONNX files from Google Cloud Storage (REMOTE_PATH)	Verify HTTPS access; rerun the command to trigger check_and_download_models retry logic
GPU out-of-memory errors	Large models (large, large-v3) require >10GB VRAM	Use --env_id 1 to force CPU execution, or select smaller models (small, base)
Slow preprocessing	load_audio resamples arbitrary sample rates to 16kHz	Pre-resample audio to 16kHz using ffmpeg -ar 16000 to skip librosa resampling
Repetitive or hallucinated output	Low compression ratio indicates repetitive token generation	Enable fallback logic with higher --temperature values (0.2-0.8) or use --beam_size 5
Timestamp misalignment	Long silences cause segment boundary drift	Adjust --no_speech_threshold or inspect boundaries with --debug flag

Summary

• The ailia-models Whisper implementation provides a complete speech-to-text pipeline with ONNX runtime support and ailia SDK acceleration.
• Audio preprocessing occurs in audio_utils.py, converting input to 16kHz log-Mel spectrograms before encoder inference.
• Model files (encoder/decoder ONNX pairs) auto-download from Google Cloud Storage when first invoked via check_and_download_models.
• Inference strategies include greedy decoding, beam search, and temperature-based fallback logic to handle repetition and silence.
• Integration options range from CLI usage (python -m audio_processing.whisper.whisper) to direct Python API calls using recognize_from_audio() or predict().

Frequently Asked Questions

How do I switch between different Whisper model sizes in ailia-models?

Modify the --model_type argument when running the CLI or set the model_type variable in Python before calling check_and_download_models. Valid options are tiny, base, small, medium, large, and turbo. The corresponding encoder and decoder files (e.g., encoder_small.onnx and decoder_small.onnx) are automatically fetched from the remote storage path.

Can I use ONNX Runtime instead of the ailia SDK for inference?

Yes. Pass the --onnx flag when using the CLI to force ONNX Runtime execution. In Python, replace the ailia.Net initialization with onnxruntime.InferenceSession, ensuring you handle the input/output tensor binding manually. The whisper.py script contains compatibility shims (lines 255-270) that abstract the backend selection.

Why does my transcription show repetitive text or hallucinations?

Repetitive output typically indicates a low compression ratio during decoding, triggering the fallback mechanism in decode_with_fallback. To mitigate this, increase the --temperature parameter (try 0.2 to 0.8) to introduce sampling randomness, or use --beam_size 5 to enable beam search decoding instead of greedy decoding. These parameters are processed in the decode function (lines 177-224 of whisper.py).

How do I enable real-time microphone transcription?

Use the -V flag when running whisper.py from the command line. This activates the recognize_from_microphone function, which spawns an audio capture thread and processes chunks through the same predict pipeline used for file input. Ensure your system has microphone permissions and that librosa or sounddevice is installed for audio I/O handling.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}