# How to Use Whisper for Speech-to-Text Transcription with ailia-models

> Transcribe audio with Whisper using ailia-models CLI or Python API. Easily perform speech-to-text from files or microphone input for fast and accurate results.

- Repository: [axinc-ai/ailia-models](https://github.com/axinc-ai/ailia-models)
- Tags: how-to-guide
- Published: 2026-02-26

---

**You can transcribe audio files or stream microphone input using the Whisper implementation in ailia-models by executing the CLI with `--input` or `-V` flags, or by importing the Python API from [`audio_processing/whisper/whisper.py`](https://github.com/axinc-ai/ailia-models/blob/main/audio_processing/whisper/whisper.py) to run encoder-decoder inference with beam search or greedy decoding.**

The **ailia-models** repository by AXINC provides a production-ready implementation of OpenAI's Whisper architecture for **speech-to-text transcription**. Located under `audio_processing/whisper/`, this implementation wraps ONNX encoder and decoder models with the ailia SDK (or ONNX Runtime), handling audio preprocessing, language detection, and beam-search decoding. Whether you need to transcribe pre-recorded files or stream from a microphone, the repository offers both command-line and programmatic interfaces.

## Architecture and Core Components

The Whisper pipeline in ailia-models consists of distinct modules responsible for audio processing, tokenization, and neural inference.

### Audio Preprocessing Pipeline

Audio files are processed in [`audio_processing/whisper/audio_utils.py`](https://github.com/axinc-ai/ailia-models/blob/main/audio_processing/whisper/audio_utils.py). The `load_audio` function reads input files using `librosa` (or ffmpeg if enabled) and resamples to 16 kHz. The `pad_or_trim` function ensures the waveform matches the 30-second chunk size (`N_SAMPLES = 480000`), while `log_mel_spectrogram` computes the 80-bin Mel spectrogram required by the encoder.

### Model Inference Engine

The core inference logic resides in [`audio_processing/whisper/whisper.py`](https://github.com/axinc-ai/ailia-models/blob/main/audio_processing/whisper/whisper.py). The `get_audio_features` function runs the encoder ONNX model to produce audio embeddings. Decoding is handled by `inference_logits` and the `decode` function, which implements KV-cache management, logit filtering (timestamp suppression, blank token removal), and beam search or greedy sampling strategies. The `decode_with_fallback` function automatically retries with higher temperature settings when compression ratios indicate repetitive or low-probability outputs.

### Tokenization and Language Detection

The [`tokenizer.py`](https://github.com/axinc-ai/ailia-models/blob/main/tokenizer.py) and [`ailia_simple_tokenizer.py`](https://github.com/axinc-ai/ailia-models/blob/main/ailia_simple_tokenizer.py) files manage the 51,865-token vocabulary, language token mappings, and task-specific prefixes (transcribe vs. translate). Language detection occurs automatically during the initial decoding steps unless explicitly overridden via command-line arguments.

## Installation and Model Setup

Install the required dependencies and allow the SDK to download model weights on first run.

```bash
pip install ailia librosa tqdm

# Optional: for ONNX Runtime backend instead of ailia SDK

pip install onnxruntime

```

The `check_and_download_models` function in [`whisper.py`](https://github.com/axinc-ai/ailia-models/blob/main/whisper.py) (lines 1199-1202) automatically fetches encoder and decoder ONNX files from `https://storage.googleapis.com/ailia-models/whisper/` if they are not present locally. Available model sizes include `tiny`, `base`, `small`, `medium`, `large`, and `turbo`, with corresponding files named `encoder_{size}.onnx` and `decoder_{size}.onnx`.

## Command-Line Interface for Speech-to-Text

The CLI entry point in [`whisper.py`](https://github.com/axinc-ai/ailia-models/blob/main/whisper.py) (lines 1263-1272) supports file-based and real-time microphone transcription.

### Transcribe Audio Files

```bash

# Basic transcription with default small model

python -m audio_processing.whisper.whisper --input recording.wav

# High-accuracy transcription with large model

python -m audio_processing.whisper.whisper --model_type large --input meeting.wav

# Translate Japanese audio to English text

python -m audio_processing.whisper.whisper --task translate --language ja --input japanese_speech.wav

# Beam search for more stable results

python -m audio_processing.whisper.whisper --beam_size 5 --temperature 0 --input podcast.wav

```

### Real-Time Microphone Input

Enable microphone mode with the `-V` flag. The `recognize_from_microphone` function captures audio streams and processes them through the same `predict` pipeline used for file input.

```bash
python -m audio_processing.whisper.whisper -V --model_type base

```

## Python API Integration

Embed Whisper transcription directly into Python applications using the high-level and low-level APIs exposed in [`whisper.py`](https://github.com/axinc-ai/ailia-models/blob/main/whisper.py).

### High-Level Inference Example

```python
from ailia import Net
from audio_processing.whisper.whisper import (
    check_and_download_models,
    load_audio,
    log_mel_spectrogram,
    recognize_from_audio,
    get_args_parser,
)

# Configure model parameters

model_type = "small"
remote_path = "https://storage.googleapis.com/ailia-models/whisper/"

# Download models if needed

enc_files = (f"encoder_{model_type}.onnx", f"encoder_{model_type}.onnx.prototxt")
dec_files = (f"decoder_{model_type}.onnx", f"decoder_{model_type}.onnx.prototxt")
check_and_download_models(enc_files[0], enc_files[1], remote_path)
check_and_download_models(dec_files[0], dec_files[1], remote_path)

# Initialize ailia networks (env_id=0 for GPU, 1 for CPU)

enc_net = Net(enc_files[1], enc_files[0], env_id=0)
dec_net = Net(dec_files[1], dec_files[0], env_id=0)

# Load and preprocess audio

wav = load_audio("interview.wav")
mel = log_mel_spectrogram(wav, n_mels=80)

# Run transcription

result = recognize_from_audio(enc_net, dec_net)
print("Transcription:", result["text"])

# Access segment-level details with timestamps

for segment in result["segments"]:
    print(f"[{segment['start']:.2f}s - {segment['end']:.2f}s] {segment['text']}")

```

### Direct Prediction API

For lower-level control, use the `predict` function directly (lines 885-889 in [`whisper.py`](https://github.com/axinc-ai/ailia-models/blob/main/whisper.py)):

```python
from audio_processing.whisper.whisper import predict

# wav is a numpy array of float32 audio at 16kHz

result = predict(wav, enc_net, dec_net, immediate=True, task="transcribe")

```

## Key Source Files Reference

| File | Purpose | Direct Link |
|------|---------|-------------|
| [`audio_processing/whisper/whisper.py`](https://github.com/axinc-ai/ailia-models/blob/main/audio_processing/whisper/whisper.py) | Main inference script, CLI entry point, model download logic, beam search decoding | [View on GitHub](https://github.com/axinc-ai/ailia-models/blob/master/audio_processing/whisper/whisper.py) |
| [`audio_processing/whisper/audio_utils.py`](https://github.com/axinc-ai/ailia-models/blob/main/audio_processing/whisper/audio_utils.py) | Audio loading, resampling to 16kHz, padding/trimming to 30s chunks, log-Mel spectrogram computation | [View on GitHub](https://github.com/axinc-ai/ailia-models/blob/master/audio_processing/whisper/audio_utils.py) |
| [`audio_processing/whisper/tokenizer.py`](https://github.com/axinc-ai/ailia-models/blob/main/audio_processing/whisper/tokenizer.py) | Multilingual tokenizer, language token mappings, task prefixes (transcribe/translate) | [View on GitHub](https://github.com/axinc-ai/ailia-models/blob/master/audio_processing/whisper/tokenizer.py) |
| [`audio_processing/whisper/ailia_simple_tokenizer.py`](https://github.com/axinc-ai/ailia-models/blob/main/audio_processing/whisper/ailia_simple_tokenizer.py) | Fallback tokenizer when ailia tokenizer is disabled | [View on GitHub](https://github.com/axinc-ai/ailia-models/blob/master/audio_processing/whisper/ailia_simple_tokenizer.py) |

## Performance Optimization and Troubleshooting

| Issue | Root Cause | Solution |
|-------|------------|----------|
| **Model download failures** | Initial run requires fetching ONNX files from Google Cloud Storage (`REMOTE_PATH`) | Verify HTTPS access; rerun the command to trigger `check_and_download_models` retry logic |
| **GPU out-of-memory errors** | Large models (large, large-v3) require >10GB VRAM | Use `--env_id 1` to force CPU execution, or select smaller models (`small`, `base`) |
| **Slow preprocessing** | `load_audio` resamples arbitrary sample rates to 16kHz | Pre-resample audio to 16kHz using `ffmpeg -ar 16000` to skip librosa resampling |
| **Repetitive or hallucinated output** | Low compression ratio indicates repetitive token generation | Enable fallback logic with higher `--temperature` values (0.2-0.8) or use `--beam_size 5` |
| **Timestamp misalignment** | Long silences cause segment boundary drift | Adjust `--no_speech_threshold` or inspect boundaries with `--debug` flag |

## Summary

- The **ailia-models** Whisper implementation provides a complete speech-to-text pipeline with ONNX runtime support and ailia SDK acceleration.
- **Audio preprocessing** occurs in [`audio_utils.py`](https://github.com/axinc-ai/ailia-models/blob/main/audio_utils.py), converting input to 16kHz log-Mel spectrograms before encoder inference.
- **Model files** (encoder/decoder ONNX pairs) auto-download from Google Cloud Storage when first invoked via `check_and_download_models`.
- **Inference strategies** include greedy decoding, beam search, and temperature-based fallback logic to handle repetition and silence.
- **Integration options** range from CLI usage (`python -m audio_processing.whisper.whisper`) to direct Python API calls using `recognize_from_audio()` or `predict()`.

## Frequently Asked Questions

### How do I switch between different Whisper model sizes in ailia-models?

Modify the `--model_type` argument when running the CLI or set the `model_type` variable in Python before calling `check_and_download_models`. Valid options are `tiny`, `base`, `small`, `medium`, `large`, and `turbo`. The corresponding encoder and decoder files (e.g., `encoder_small.onnx` and `decoder_small.onnx`) are automatically fetched from the remote storage path.

### Can I use ONNX Runtime instead of the ailia SDK for inference?

Yes. Pass the `--onnx` flag when using the CLI to force ONNX Runtime execution. In Python, replace the `ailia.Net` initialization with `onnxruntime.InferenceSession`, ensuring you handle the input/output tensor binding manually. The [`whisper.py`](https://github.com/axinc-ai/ailia-models/blob/main/whisper.py) script contains compatibility shims (lines 255-270) that abstract the backend selection.

### Why does my transcription show repetitive text or hallucinations?

Repetitive output typically indicates a low compression ratio during decoding, triggering the fallback mechanism in `decode_with_fallback`. To mitigate this, increase the `--temperature` parameter (try 0.2 to 0.8) to introduce sampling randomness, or use `--beam_size 5` to enable beam search decoding instead of greedy decoding. These parameters are processed in the `decode` function (lines 177-224 of [`whisper.py`](https://github.com/axinc-ai/ailia-models/blob/main/whisper.py)).

### How do I enable real-time microphone transcription?

Use the `-V` flag when running [`whisper.py`](https://github.com/axinc-ai/ailia-models/blob/main/whisper.py) from the command line. This activates the `recognize_from_microphone` function, which spawns an audio capture thread and processes chunks through the same `predict` pipeline used for file input. Ensure your system has microphone permissions and that `librosa` or `sounddevice` is installed for audio I/O handling.