# How to Benchmark AI Model Inference Performance on Different Hardware with Ailia Models

> Easily benchmark AI model inference performance on CPU, GPU, and NPU hardware using ailia models. Learn to measure latency across different hardware without code changes.

- Repository: [axinc-ai/ailia-models](https://github.com/axinc-ai/ailia-models)
- Tags: how-to-guide
- Published: 2026-02-25

---

**The Ailia Models repository provides built-in benchmarking flags (`-b` and `-bc`) that measure inference latency across CPU, GPU, and NPU hardware without modifying any code.**

To benchmark AI model inference performance on different hardware, the ailia-models repository offers a unified command-line interface that works across every demo script. This open-source collection from axinc-ai includes standardized timing utilities that capture wall-clock latency for model loading, encoding, and generation stages across multiple backends including Ailia, PyTorch, and vendor-specific accelerators.

## Built-in Benchmark Architecture

### Command-Line Flags (`-b` and `-bc`)

All demo scripts inherit a common argument parser defined in [[`util/arg_utils.py`](https://github.com/axinc-ai/ailia-models/blob/main/util/arg_utils.py)](https://github.com/axinc-ai/ailia-models/blob/master/util/arg_utils.py). This utility automatically injects two critical flags into every model script:

- **`-b` / `--benchmark`** – Enables benchmark mode, which processes the same input repeatedly to measure pure inference time
- **`-bc` / `--benchmark_count <N>`** – Sets the number of repetitions (defaults to 5)

When benchmark mode is active, scripts wrap the critical inference block with millisecond-precision timers, execute the block `benchmark_count` times, and log both per-iteration and average execution times.

### Timing Implementation in Demo Scripts

The timing logic lives directly within each model script, ensuring accurate measurement of model-specific operations. For example, in [[`vision_language_model/qwen2_vl/qwen2_vl.py`](https://github.com/axinc-ai/ailia-models/blob/main/vision_language_model/qwen2_vl/qwen2_vl.py)](https://github.com/axinc-ai/ailia-models/blob/master/vision_language_model/qwen2_vl/qwen2_vl.py), the benchmark pattern captures both encoding and generation phases:

```python
if args.benchmark:
    start = int(round(time.time() * 1000))

# ... inference (model.predict / model.run) ...

if args.benchmark:
    end = int(round(time.time() * 1000))
    estimation_time = end - start
    logger.info(f"\tencode time {estimation_time} ms")

```

This pattern repeats for the decode/generation loop, providing granular visibility into where time is spent during inference.

## Selecting Hardware Backends for Benchmarking

### Ailia Backend (CPU/GPU/NPU)

When using the Ailia runtime, hardware selection is controlled via the `--env_id` argument added by [`arg_utils.py`](https://github.com/axinc-ai/ailia-models/blob/main/arg_utils.py). This flag selects from pre-registered Ailia environments:

```bash

# List available environments

python some_demo.py --env_list

# Run on GPU (env_id 0 on most machines)

python some_demo.py -b -bc 10 -e 0

# Force CPU execution

python some_demo.py -b -bc 10 -e 1

```

If `--env_id` is omitted, the script calls `ailia.get_gpu_environment_id()` and automatically falls back to CPU when the requested accelerator is unavailable.

### PyTorch Backend

For models running on PyTorch or Transformers, hardware selection follows standard PyTorch conventions. Scripts typically set a `device_map` based on the host OS (`cpu` for macOS, otherwise `cuda:0`), but you can override this via environment variables or script modification.

The dedicated benchmark script [[`vision_language_model/qwen2_vl/benchmark_torch.py`](https://github.com/axinc-ai/ailia-models/blob/main/vision_language_model/qwen2_vl/benchmark_torch.py)](https://github.com/axinc-ai/ailia-models/blob/master/vision_language_model/qwen2_vl/benchmark_torch.py) demonstrates this pattern:

```bash

# GPU benchmark (default on Linux)

python vision_language_model/qwen2_vl/benchmark_torch.py -b -bc 30

# CPU benchmark

CUDA_VISIBLE_DEVICES= python vision_language_model/qwen2_vl/benchmark_torch.py -b -bc 30

```

## What Gets Measured During Benchmarking

The benchmark timers capture **wall-clock milliseconds** for three distinct stages:

1. **Model loading** – One-time initialization measured before the benchmark loop begins
2. **Encode / forward pass** – Heavy visual-token processing or initial model evaluation
3. **Generate / decode** – Autoregressive token generation or iterative refinement

Because the same input tensor is reused across all iterations, the reported times reflect **pure inference latency** without file I/O or preprocessing overhead.

## Practical Benchmark Examples

### Vision-Language Model on Ailia

```bash
python vision_language_model/qwen2_vl/qwen2_vl.py \
    -i demo.jpeg \
    -s output_res \
    -b \
    -bc 10 \
    -e 0

```

**Sample output:**

```

[INFO] env_id: 0
[INFO] encode time 45 ms
[INFO] generate time 112 ms
[INFO] average time estimation 78 ms

```

### Text Recognition with CPU Fallback

```bash
python text_recognition/paddleocr/paddleocr.py \
    -i sample.png \
    -b \
    -bc 5 \
    -e 1

```

### Rotation Prediction on GPU

```bash
python rotation_prediction/rotnet/rotnet.py \
    -i test.jpg \
    -b \
    -bc 5 \
    -e 0

```

## Summary

- **Unified interface:** Every demo script in ailia-models supports `-b` and `-bc` flags for instant benchmarking without code changes
- **Hardware flexibility:** Use `-e` or `--env_id` to switch between CPU, GPU, and NPU execution environments when using the Ailia backend
- **Granular timing:** Benchmarks measure model loading, encoding, and generation phases separately to identify bottlenecks
- **Cross-backend support:** The same flags work for both Ailia-native models and PyTorch/Transformers implementations, though hardware selection differs between backends

## Frequently Asked Questions

### How do I know which environment IDs are available on my system?

Run any demo script with the `--env_list` flag to display all registered Ailia environments and their corresponding IDs. The list typically includes CPU (often ID 1), GPU (often ID 0), and any available NPUs or remote inference servers.

### Can I benchmark models that use PyTorch instead of Ailia?

Yes. PyTorch-based models in the repository respect the same `-b` and `-bc` flags. For hardware selection, PyTorch scripts use `device_map` logic rather than `--env_id`. You can force CPU execution by setting `CUDA_VISIBLE_DEVICES=` or modifying the `device_map` variable in the script.

### What is the difference between encode time and generate time in the benchmark output?

**Encode time** measures the initial forward pass that processes input data (such as image encoding in vision-language models). **Generate time** measures the autoregressive decoding loop that produces output tokens. Separating these helps identify whether latency comes from heavy input processing or from generation length.

### Does the benchmark warm up the model before timing?

The benchmark runs the inference block exactly `benchmark_count` times as specified by `-bc`, with the first iteration included in the average. For the most stable results on GPU, use a larger `-bc` value (20-30) to ensure the GPU reaches steady-state temperature and clock speeds.