# How to Handle Video Inference and Real-Time Camera Processing with AILIA Models

> Learn how to perform video inference and real-time camera processing with AILIA Models. Discover our unified three-stage pipeline for seamless AI application development.

- Repository: [axinc-ai/ailia-models](https://github.com/axinc-ai/ailia-models)
- Tags: how-to-guide
- Published: 2026-02-26

---

**AILIA Models provides a unified three-stage pipeline for processing both pre-recorded video files and live camera streams through acquisition, pre-processing, and inference stages.**

The `axinc-ai/ailia-models` repository offers production-ready utilities for **video inference and real-time camera processing** across diverse hardware setups. Whether analyzing archived footage with `fetch_video()` or capturing live streams from generic webcams or Basler industrial cameras, the framework standardizes frame acquisition, tensor preparation, and model execution.

## The Three-Stage Pipeline for Video Inference and Real-Time Camera Processing

### Stage 1: Acquisition

The acquisition layer abstracts video sources into a consistent NumPy frame interface. For video files, the `fetch_video()` function in **[`vision_language_model/qwen2_vl/qwen2_vl.py`](https://github.com/axinc-ai/ailia-models/blob/main/vision_language_model/qwen2_vl/qwen2_vl.py)** (lines 91-100) handles `cv2.VideoCapture` initialization, frame sampling, and BGRA-to-RGB conversion. For live feeds, **[`util/webcamera_utils.py`](https://github.com/axinc-ai/ailia-models/blob/main/util/webcamera_utils.py)** provides `BaslerCameraCapture` (lines 78-101) for Pylon SDK integration and standard webcam capture.

### Stage 2: Pre-processing

Raw frames require normalization and tensor layout conversion before inference. The `preprocess_frame()` function in **[`util/webcamera_utils.py`](https://github.com/axinc-ai/ailia-models/blob/main/util/webcamera_utils.py)** (lines 82-126) executes aspect-ratio-preserving resizing via `adjust_frame_size()`, optional BGR-to-RGB conversion, and ImageNet normalization. It finally transposes dimensions from HWC to CHW format and adds a batch dimension, producing arrays compatible with AILIA's input expectations.

### Stage 3: Inference

Pre-processed frames enter the model through the `preprocess()` and `forward()` functions defined in **[`vision_language_model/qwen2_vl/qwen2_vl.py`](https://github.com/axinc-ai/ailia-models/blob/main/vision_language_model/qwen2_vl/qwen2_vl.py)** (lines 49-115 and 18-50 respectively). The `preprocess()` function patchifies image tensors, while `forward()` executes the ONNX model inference via `ailia.Net`, returning logits and updated key-value caches for autoregressive tasks.

## Processing Pre-Recorded Video Files

For archived video analysis, `fetch_video()` implements intelligent frame sampling to balance temporal coverage with computational constraints. The function performs the following operations as implemented in lines 91-141 of **[`vision_language_model/qwen2_vl/qwen2_vl.py`](https://github.com/axinc-ai/ailia-models/blob/main/vision_language_model/qwen2_vl/qwen2_vl.py)**:

1. **Video Decoding**: Opens the file via `cv2.VideoCapture` and extracts metadata (FPS, total frames)
2. **Temporal Sampling**: Calculates target frame count based on video duration and a default 2 FPS inference rate, uniformly sampling frames to capture the full temporal range
3. **Pixel Budget Management**: Resizes selected frames using Pillow's `Image.Resampling.BICUBIC` to respect `min_pixels` and `max_pixels` constraints (lines 124-138)
4. **Format Standardization**: Returns a `List[np.ndarray]` of RGB images ready for model preprocessing

## Real-Time Camera Processing with Webcams and Basler Cameras

### Generic Webcam Capture

For standard USB cameras, the repository leverages OpenCV's `cv2.VideoCapture` interface directly. Frames captured through `cap.read()` undergo processing via `preprocess_frame()` in **[`util/webcamera_utils.py`](https://github.com/axinc-ai/ailia-models/blob/main/util/webcamera_utils.py)**, which handles:

- **Aspect Ratio Preservation**: `adjust_frame_size()` (lines 22-56) creates a padded canvas to prevent distortion
- **Colour Space Conversion**: Optional `cv2.cvtColor(..., cv2.COLOR_BGR2RGB)` for models expecting RGB input
- **Tensor Layout**: Transposition from HWC to CHW and batch dimension addition

### Basler Industrial Camera Integration

For high-performance industrial applications, `BaslerCameraCapture` in **[`util/webcamera_utils.py`](https://github.com/axinc-ai/ailia-models/blob/main/util/webcamera_utils.py)** (lines 78-101) abstracts the Pylon SDK. The class provides:

- **SDK Initialization**: Automatic camera detection and buffer configuration
- **Frame Acquisition**: `read()` method yielding NumPy BGR arrays synchronized with the camera's frame rate
- **Resource Management**: Context manager support for proper camera disconnection

## Writing Inference Output to Video Files

The `get_writer()` function in **[`util/webcamera_utils.py`](https://github.com/axinc-ai/ailia-models/blob/main/util/webcamera_utils.py)** (lines 29-75) creates `cv2.VideoWriter` instances supporting both local file output and GStreamer TCP streaming. This utility enables recording annotated inference results (e.g., bounding box overlays, classification labels) at specified FPS and resolution while handling RGB-to-BGR conversion for OpenCV compatibility.

## Complete Code Examples

### Example A: Inference on Pre-Recorded Video

```python
from vision_language_model.qwen2_vl.qwen2_vl import fetch_video, preprocess, forward
from tokenizer import Tokenizer   # placeholder – use the repo’s tokenizer

import ailia

# 1️⃣ Load the ONNX model

net = ailia.Net()
net.load('qwen2_vl.onnx')

# 2️⃣ Acquire frames

frames = fetch_video('demo.mp4')          # returns List[np.ndarray]

# 3️⃣ Pre‑process frames for the model

patches, shape = preprocess(frames)       # patches: (N, C, H, W)

# 4️⃣ Prepare tokenisation (example assumes a VQA task)

tokenizer = Tokenizer()
input_ids = tokenizer.encode("Describe the video")
attention_mask = np.ones_like(input_ids)

# 5️⃣ Run inference

logits, _ = forward(net,
                    input_ids,
                    patches,
                    np.arange(patches.shape[0]),   # position_ids

                    attention_mask,
                    past_key_values=[],
                    first_run=True)

# 6️⃣ Decode result (implementation‑specific)

answer = tokenizer.decode(np.argmax(logits, axis=-1))
print("Model answer:", answer)

```

*Key source links*  
- `fetch_video` implementation in [`vision_language_model/qwen2_vl/qwen2_vl.py`](https://github.com/axinc-ai/ailia-models/blob/main/vision_language_model/qwen2_vl/qwen2_vl.py) (lines 91-141)  
- `preprocess` and `forward` logic (lines 49-115 and 18-50)

### Example B: Real-Time Webcam Processing

```python
import cv2
from util.webcamera_utils import preprocess_frame, get_writer

# Model loading (same as above)

net = ailia.Net()
net.load('qwen2_vl.onnx')

# Capture from default webcam (device 0)

cap = cv2.VideoCapture(0)
writer = get_writer('output.mp4', 224, 224, fps=20)

while True:
    ret, frame = cap.read()
    if not ret:
        break

    # 1️⃣ Pre‑process frame for the model

    _, data = preprocess_frame(frame, input_height=224, input_width=224)

    # 2️⃣ Run inference (single‑frame case)

    patches, _ = preprocess([data.squeeze(0)])   # data: (1, C, H, W)

    logits, _ = forward(net,
                        input_ids=np.array([101]),   # dummy token

                        patches,
                        np.arange(patches.shape[0]),
                        attention_mask=np.ones(1),
                        past_key_values=[],
                        first_run=True)

    # 3️⃣ Visualise (simple example)

    label = np.argmax(logits)
    cv2.putText(frame, f"Label: {label}", (10,30),
                cv2.FONT_HERSHEY_SIMPLEX, 1, (0,255,0), 2)

    # 4️⃣ Write annotated frame to output video

    writer.write(frame)

    cv2.imshow('Live', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
writer.release()
cv2.destroyAllWindows()

```

*Key source links*  
- `preprocess_frame` – colour conversion & normalisation in [`util/webcamera_utils.py`](https://github.com/axinc-ai/ailia-models/blob/main/util/webcamera_utils.py) (lines 82-126)  
- `get_writer` – video writer setup (lines 29-75)

### Example C: Basler Camera Processing

```python
from util.webcamera_utils import BaslerCameraCapture, preprocess_frame, get_writer

# Initialise Basler camera

cam = BaslerCameraCapture()
cam.start_capture()

writer = get_writer('basler_out.mp4', 224, 224)

while True:
    frame = cam.read()          # NumPy BGR frame from Basler

    _, data = preprocess_frame(frame, 224, 224)

    # Same inference path as the generic webcam example …

    patches, _ = preprocess([data.squeeze(0)])
    logits, _ = forward(net, input_ids, patches, ...)

    # Visualise & save

    cv2.putText(frame, f"Score: {logits.max():.2f}", (10,30),
                cv2.FONT_HERSHEY_SIMPLEX, 1, (255,0,0), 2)
    writer.write(frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

```

*Key source links* – `BaslerCameraCapture` class definition in [`util/webcamera_utils.py`](https://github.com/axinc-ai/ailia-models/blob/main/util/webcamera_utils.py) (lines 78-101).

## Key Implementation Files

| File | Purpose | Direct link |
|------|---------|-------------|
| [`vision_language_model/qwen2_vl/qwen2_vl.py`](https://github.com/axinc-ai/ailia-models/blob/main/vision_language_model/qwen2_vl/qwen2_vl.py) | Core video loader (`fetch_video`), preprocessing & model forward pass | <https://github.com/axinc-ai/ailia-models/blob/master/vision_language_model/qwen2_vl/qwen2_vl.py> |
| [`util/webcamera_utils.py`](https://github.com/axinc-ai/ailia-models/blob/main/util/webcamera_utils.py) | Utilities for webcam/Basler capture, frame padding, colour conversion, normalisation, and video writer | <https://github.com/axinc-ai/ailia-models/blob/master/util/webcamera_utils.py> |
| [`hand_detection/hand_detection_pytorch/hand_detection_pytorch.py`](https://github.com/axinc-ai/ailia-models/blob/main/hand_detection/hand_detection_pytorch/hand_detection_pytorch.py) (example of live‑camera loop) | Demonstrates real‑time inference with a webcam using the same preprocessing utilities | <https://github.com/axinc-ai/ailia-models/blob/master/hand_detection/hand_detection_pytorch/hand_detection_pytorch.py> |
| [`vehicle_recognition/vehicle-license-plate-detection-barrier/vehicle-license-plate-detection-barrier.py`](https://github.com/axinc-ai/ailia-models/blob/main/vehicle_recognition/vehicle-license-plate-detection-barrier/vehicle-license-plate-detection-barrier.py) | Shows video capture, frame‑by‑frame processing, and visual overlay – useful reference for integrating model outputs | <https://github.com/axinc-ai/ailia-models/blob/master/vehicle_recognition/vehicle-license-plate-detection-barrier/vehicle-license-plate-detection-barrier.py> |

These files together illustrate the full end‑to‑end flow for both **offline video inference** and **live camera processing** using the AILIA model ecosystem.

## Summary

- **Unified Pipeline**: AILIA Models standardizes **video inference and real-time camera processing** through three stages—Acquisition, Pre-processing, and Inference—regardless of input source.
- **Video File Handling**: The `fetch_video()` function in [`vision_language_model/qwen2_vl/qwen2_vl.py`](https://github.com/axinc-ai/ailia-models/blob/main/vision_language_model/qwen2_vl/qwen2_vl.py) intelligently samples frames, manages pixel budgets, and standardizes color formats for batch processing.
- **Live Camera Support**: [`util/webcamera_utils.py`](https://github.com/axinc-ai/ailia-models/blob/main/util/webcamera_utils.py) provides `BaslerCameraCapture` for industrial Pylon SDK integration and `preprocess_frame()` for aspect-ratio-preserving normalization of webcam feeds.
- **Output Recording**: The `get_writer()` utility supports both local MP4 encoding and GStreamer TCP streaming for saving annotated inference results.

## Frequently Asked Questions

### How does `fetch_video()` handle long video files without exhausting memory?

`fetch_video()` in [`vision_language_model/qwen2_vl/qwen2_vl.py`](https://github.com/axinc-ai/ailia-models/blob/main/vision_language_model/qwen2_vl/qwen2_vl.py) (lines 91-141) implements intelligent temporal sampling that limits the total frame count to 4-30 frames regardless of video length. It calculates a target frame count based on the video's native FPS and a default inference rate of 2 FPS, then uniformly samples frames across the timeline. This ensures the model receives representative temporal information while maintaining constant memory usage.

### What is the difference between `preprocess_frame()` and the `preprocess()` function used for video files?

`preprocess_frame()` in [`util/webcamera_utils.py`](https://github.com/axinc-ai/ailia-models/blob/main/util/webcamera_utils.py) (lines 82-126) is designed for single-frame camera inputs and handles aspect-ratio-preserving padding via `adjust_frame_size()`, color conversion, and HWC-to-CHW transposition. In contrast, `preprocess()` in [`vision_language_model/qwen2_vl/qwen2_vl.py`](https://github.com/axinc-ai/ailia-models/blob/main/vision_language_model/qwen2_vl/qwen2_vl.py) (lines 49-115) accepts lists of already-resized frames and performs patchification—converting images into flattened patch tensors suitable for vision-language models like Qwen2-VL.

### Can the pipeline record inference results while processing live camera feeds?

Yes, the `get_writer()` function in [`util/webcamera_utils.py`](https://github.com/axinc-ai/ailia-models/blob/main/util/webcamera_utils.py) (lines 29-75) creates a `cv2.VideoWriter` instance that supports both local file output (MP4/AVI) and GStreamer TCP streaming. You can initialize the writer with target dimensions and FPS, then write annotated frames inside the inference loop. This allows real-time recording of bounding box overlays, classification labels, or segmentation masks alongside the live processing pipeline.

### How does the Basler camera integration differ from standard webcam capture?

`BaslerCameraCapture` in [`util/webcamera_utils.py`](https://github.com/axinc-ai/ailia-models/blob/main/util/webcamera_utils.py) (lines 78-101) abstracts the Pylon SDK specifically for Basler industrial cameras, handling camera initialization, buffer configuration, and hardware-triggered acquisition. Unlike standard `cv2.VideoCapture` which uses DirectShow/V4L2 backends, the Basler class yields NumPy BGR frames synchronized to the camera's native frame rate and provides proper resource cleanup through context manager protocols.