How to Benchmark AI Model Inference Performance on Different Hardware with Ailia Models

The Ailia Models repository provides built-in benchmarking flags (-b and -bc) that measure inference latency across CPU, GPU, and NPU hardware without modifying any code.

To benchmark AI model inference performance on different hardware, the ailia-models repository offers a unified command-line interface that works across every demo script. This open-source collection from axinc-ai includes standardized timing utilities that capture wall-clock latency for model loading, encoding, and generation stages across multiple backends including Ailia, PyTorch, and vendor-specific accelerators.

Built-in Benchmark Architecture

Command-Line Flags (-b and -bc)

All demo scripts inherit a common argument parser defined in [util/arg_utils.py](https://github.com/axinc-ai/ailia-models/blob/master/util/arg_utils.py). This utility automatically injects two critical flags into every model script:

• -b / --benchmark – Enables benchmark mode, which processes the same input repeatedly to measure pure inference time
• -bc / --benchmark_count <N> – Sets the number of repetitions (defaults to 5)

When benchmark mode is active, scripts wrap the critical inference block with millisecond-precision timers, execute the block benchmark_count times, and log both per-iteration and average execution times.

Timing Implementation in Demo Scripts

The timing logic lives directly within each model script, ensuring accurate measurement of model-specific operations. For example, in [vision_language_model/qwen2_vl/qwen2_vl.py](https://github.com/axinc-ai/ailia-models/blob/master/vision_language_model/qwen2_vl/qwen2_vl.py), the benchmark pattern captures both encoding and generation phases:

if args.benchmark:
    start = int(round(time.time() * 1000))

# ... inference (model.predict / model.run) ...

if args.benchmark:
    end = int(round(time.time() * 1000))
    estimation_time = end - start
    logger.info(f"\tencode time {estimation_time} ms")

This pattern repeats for the decode/generation loop, providing granular visibility into where time is spent during inference.

Selecting Hardware Backends for Benchmarking

Ailia Backend (CPU/GPU/NPU)

When using the Ailia runtime, hardware selection is controlled via the --env_id argument added by arg_utils.py. This flag selects from pre-registered Ailia environments:

# List available environments

python some_demo.py --env_list

# Run on GPU (env_id 0 on most machines)

python some_demo.py -b -bc 10 -e 0

# Force CPU execution

python some_demo.py -b -bc 10 -e 1

If --env_id is omitted, the script calls ailia.get_gpu_environment_id() and automatically falls back to CPU when the requested accelerator is unavailable.

PyTorch Backend

For models running on PyTorch or Transformers, hardware selection follows standard PyTorch conventions. Scripts typically set a device_map based on the host OS (cpu for macOS, otherwise cuda:0), but you can override this via environment variables or script modification.

The dedicated benchmark script [vision_language_model/qwen2_vl/benchmark_torch.py](https://github.com/axinc-ai/ailia-models/blob/master/vision_language_model/qwen2_vl/benchmark_torch.py) demonstrates this pattern:

# GPU benchmark (default on Linux)

python vision_language_model/qwen2_vl/benchmark_torch.py -b -bc 30

# CPU benchmark

CUDA_VISIBLE_DEVICES= python vision_language_model/qwen2_vl/benchmark_torch.py -b -bc 30

What Gets Measured During Benchmarking

The benchmark timers capture wall-clock milliseconds for three distinct stages:

1. Model loading – One-time initialization measured before the benchmark loop begins
2. Encode / forward pass – Heavy visual-token processing or initial model evaluation
3. Generate / decode – Autoregressive token generation or iterative refinement

Because the same input tensor is reused across all iterations, the reported times reflect pure inference latency without file I/O or preprocessing overhead.

Practical Benchmark Examples

Vision-Language Model on Ailia

python vision_language_model/qwen2_vl/qwen2_vl.py \
    -i demo.jpeg \
    -s output_res \
    -b \
    -bc 10 \
    -e 0

Sample output:

[INFO] env_id: 0
[INFO] encode time 45 ms
[INFO] generate time 112 ms
[INFO] average time estimation 78 ms

Text Recognition with CPU Fallback

python text_recognition/paddleocr/paddleocr.py \
    -i sample.png \
    -b \
    -bc 5 \
    -e 1

Rotation Prediction on GPU

python rotation_prediction/rotnet/rotnet.py \
    -i test.jpg \
    -b \
    -bc 5 \
    -e 0

Summary

• Unified interface: Every demo script in ailia-models supports -b and -bc flags for instant benchmarking without code changes
• Hardware flexibility: Use -e or --env_id to switch between CPU, GPU, and NPU execution environments when using the Ailia backend
• Granular timing: Benchmarks measure model loading, encoding, and generation phases separately to identify bottlenecks
• Cross-backend support: The same flags work for both Ailia-native models and PyTorch/Transformers implementations, though hardware selection differs between backends

Frequently Asked Questions

How do I know which environment IDs are available on my system?

Run any demo script with the --env_list flag to display all registered Ailia environments and their corresponding IDs. The list typically includes CPU (often ID 1), GPU (often ID 0), and any available NPUs or remote inference servers.

Can I benchmark models that use PyTorch instead of Ailia?

Yes. PyTorch-based models in the repository respect the same -b and -bc flags. For hardware selection, PyTorch scripts use device_map logic rather than --env_id. You can force CPU execution by setting CUDA_VISIBLE_DEVICES= or modifying the device_map variable in the script.

What is the difference between encode time and generate time in the benchmark output?

Encode time measures the initial forward pass that processes input data (such as image encoding in vision-language models). Generate time measures the autoregressive decoding loop that produces output tokens. Separating these helps identify whether latency comes from heavy input processing or from generation length.

Does the benchmark warm up the model before timing?

The benchmark runs the inference block exactly benchmark_count times as specified by -bc, with the first iteration included in the average. For the most stable results on GPU, use a larger -bc value (20-30) to ensure the GPU reaches steady-state temperature and clock speeds.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}