How to Handle Video Inference and Real-Time Camera Processing with AILIA Models

AILIA Models provides a unified three-stage pipeline for processing both pre-recorded video files and live camera streams through acquisition, pre-processing, and inference stages.

The axinc-ai/ailia-models repository offers production-ready utilities for video inference and real-time camera processing across diverse hardware setups. Whether analyzing archived footage with fetch_video() or capturing live streams from generic webcams or Basler industrial cameras, the framework standardizes frame acquisition, tensor preparation, and model execution.

The Three-Stage Pipeline for Video Inference and Real-Time Camera Processing

Stage 1: Acquisition

The acquisition layer abstracts video sources into a consistent NumPy frame interface. For video files, the fetch_video() function in vision_language_model/qwen2_vl/qwen2_vl.py (lines 91-100) handles cv2.VideoCapture initialization, frame sampling, and BGRA-to-RGB conversion. For live feeds, util/webcamera_utils.py provides BaslerCameraCapture (lines 78-101) for Pylon SDK integration and standard webcam capture.

Stage 2: Pre-processing

Raw frames require normalization and tensor layout conversion before inference. The preprocess_frame() function in util/webcamera_utils.py (lines 82-126) executes aspect-ratio-preserving resizing via adjust_frame_size(), optional BGR-to-RGB conversion, and ImageNet normalization. It finally transposes dimensions from HWC to CHW format and adds a batch dimension, producing arrays compatible with AILIA's input expectations.

Stage 3: Inference

Pre-processed frames enter the model through the preprocess() and forward() functions defined in vision_language_model/qwen2_vl/qwen2_vl.py (lines 49-115 and 18-50 respectively). The preprocess() function patchifies image tensors, while forward() executes the ONNX model inference via ailia.Net, returning logits and updated key-value caches for autoregressive tasks.

Processing Pre-Recorded Video Files

For archived video analysis, fetch_video() implements intelligent frame sampling to balance temporal coverage with computational constraints. The function performs the following operations as implemented in lines 91-141 of vision_language_model/qwen2_vl/qwen2_vl.py:

1. Video Decoding: Opens the file via cv2.VideoCapture and extracts metadata (FPS, total frames)
2. Temporal Sampling: Calculates target frame count based on video duration and a default 2 FPS inference rate, uniformly sampling frames to capture the full temporal range
3. Pixel Budget Management: Resizes selected frames using Pillow's Image.Resampling.BICUBIC to respect min_pixels and max_pixels constraints (lines 124-138)
4. Format Standardization: Returns a List[np.ndarray] of RGB images ready for model preprocessing

Real-Time Camera Processing with Webcams and Basler Cameras

Generic Webcam Capture

For standard USB cameras, the repository leverages OpenCV's cv2.VideoCapture interface directly. Frames captured through cap.read() undergo processing via preprocess_frame() in util/webcamera_utils.py, which handles:

• Aspect Ratio Preservation: adjust_frame_size() (lines 22-56) creates a padded canvas to prevent distortion
• Colour Space Conversion: Optional cv2.cvtColor(..., cv2.COLOR_BGR2RGB) for models expecting RGB input
• Tensor Layout: Transposition from HWC to CHW and batch dimension addition

Basler Industrial Camera Integration

For high-performance industrial applications, BaslerCameraCapture in util/webcamera_utils.py (lines 78-101) abstracts the Pylon SDK. The class provides:

• SDK Initialization: Automatic camera detection and buffer configuration
• Frame Acquisition: read() method yielding NumPy BGR arrays synchronized with the camera's frame rate
• Resource Management: Context manager support for proper camera disconnection

Writing Inference Output to Video Files

The get_writer() function in util/webcamera_utils.py (lines 29-75) creates cv2.VideoWriter instances supporting both local file output and GStreamer TCP streaming. This utility enables recording annotated inference results (e.g., bounding box overlays, classification labels) at specified FPS and resolution while handling RGB-to-BGR conversion for OpenCV compatibility.

Complete Code Examples

Example A: Inference on Pre-Recorded Video

from vision_language_model.qwen2_vl.qwen2_vl import fetch_video, preprocess, forward
from tokenizer import Tokenizer   # placeholder – use the repo’s tokenizer

import ailia

# 1️⃣ Load the ONNX model

net = ailia.Net()
net.load('qwen2_vl.onnx')

# 2️⃣ Acquire frames

frames = fetch_video('demo.mp4')          # returns List[np.ndarray]

# 3️⃣ Pre‑process frames for the model

patches, shape = preprocess(frames)       # patches: (N, C, H, W)

# 4️⃣ Prepare tokenisation (example assumes a VQA task)

tokenizer = Tokenizer()
input_ids = tokenizer.encode("Describe the video")
attention_mask = np.ones_like(input_ids)

# 5️⃣ Run inference

logits, _ = forward(net,
                    input_ids,
                    patches,
                    np.arange(patches.shape[0]),   # position_ids

                    attention_mask,
                    past_key_values=[],
                    first_run=True)

# 6️⃣ Decode result (implementation‑specific)

answer = tokenizer.decode(np.argmax(logits, axis=-1))
print("Model answer:", answer)

Key source links

• fetch_video implementation in vision_language_model/qwen2_vl/qwen2_vl.py (lines 91-141)
• preprocess and forward logic (lines 49-115 and 18-50)

Example B: Real-Time Webcam Processing

import cv2
from util.webcamera_utils import preprocess_frame, get_writer

# Model loading (same as above)

net = ailia.Net()
net.load('qwen2_vl.onnx')

# Capture from default webcam (device 0)

cap = cv2.VideoCapture(0)
writer = get_writer('output.mp4', 224, 224, fps=20)

while True:
    ret, frame = cap.read()
    if not ret:
        break

    # 1️⃣ Pre‑process frame for the model

    _, data = preprocess_frame(frame, input_height=224, input_width=224)

    # 2️⃣ Run inference (single‑frame case)

    patches, _ = preprocess([data.squeeze(0)])   # data: (1, C, H, W)

    logits, _ = forward(net,
                        input_ids=np.array([101]),   # dummy token

                        patches,
                        np.arange(patches.shape[0]),
                        attention_mask=np.ones(1),
                        past_key_values=[],
                        first_run=True)

    # 3️⃣ Visualise (simple example)

    label = np.argmax(logits)
    cv2.putText(frame, f"Label: {label}", (10,30),
                cv2.FONT_HERSHEY_SIMPLEX, 1, (0,255,0), 2)

    # 4️⃣ Write annotated frame to output video

    writer.write(frame)

    cv2.imshow('Live', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
writer.release()
cv2.destroyAllWindows()

Key source links

• preprocess_frame – colour conversion & normalisation in util/webcamera_utils.py (lines 82-126)
• get_writer – video writer setup (lines 29-75)

Example C: Basler Camera Processing

from util.webcamera_utils import BaslerCameraCapture, preprocess_frame, get_writer

# Initialise Basler camera

cam = BaslerCameraCapture()
cam.start_capture()

writer = get_writer('basler_out.mp4', 224, 224)

while True:
    frame = cam.read()          # NumPy BGR frame from Basler

    _, data = preprocess_frame(frame, 224, 224)

    # Same inference path as the generic webcam example …

    patches, _ = preprocess([data.squeeze(0)])
    logits, _ = forward(net, input_ids, patches, ...)

    # Visualise & save

    cv2.putText(frame, f"Score: {logits.max():.2f}", (10,30),
                cv2.FONT_HERSHEY_SIMPLEX, 1, (255,0,0), 2)
    writer.write(frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

Key source links – BaslerCameraCapture class definition in util/webcamera_utils.py (lines 78-101).

Key Implementation Files

File	Purpose	Direct link
vision_language_model/qwen2_vl/qwen2_vl.py	Core video loader (fetch_video), preprocessing & model forward pass	https://github.com/axinc-ai/ailia-models/blob/master/vision_language_model/qwen2_vl/qwen2_vl.py
util/webcamera_utils.py	Utilities for webcam/Basler capture, frame padding, colour conversion, normalisation, and video writer	https://github.com/axinc-ai/ailia-models/blob/master/util/webcamera_utils.py
hand_detection/hand_detection_pytorch/hand_detection_pytorch.py (example of live‑camera loop)	Demonstrates real‑time inference with a webcam using the same preprocessing utilities	https://github.com/axinc-ai/ailia-models/blob/master/hand_detection/hand_detection_pytorch/hand_detection_pytorch.py
vehicle_recognition/vehicle-license-plate-detection-barrier/vehicle-license-plate-detection-barrier.py	Shows video capture, frame‑by‑frame processing, and visual overlay – useful reference for integrating model outputs	https://github.com/axinc-ai/ailia-models/blob/master/vehicle_recognition/vehicle-license-plate-detection-barrier/vehicle-license-plate-detection-barrier.py

These files together illustrate the full end‑to‑end flow for both offline video inference and live camera processing using the AILIA model ecosystem.

Summary

• Unified Pipeline: AILIA Models standardizes video inference and real-time camera processing through three stages—Acquisition, Pre-processing, and Inference—regardless of input source.
• Video File Handling: The fetch_video() function in vision_language_model/qwen2_vl/qwen2_vl.py intelligently samples frames, manages pixel budgets, and standardizes color formats for batch processing.
• Live Camera Support: util/webcamera_utils.py provides BaslerCameraCapture for industrial Pylon SDK integration and preprocess_frame() for aspect-ratio-preserving normalization of webcam feeds.
• Output Recording: The get_writer() utility supports both local MP4 encoding and GStreamer TCP streaming for saving annotated inference results.

Frequently Asked Questions

How does fetch_video() handle long video files without exhausting memory?

fetch_video() in vision_language_model/qwen2_vl/qwen2_vl.py (lines 91-141) implements intelligent temporal sampling that limits the total frame count to 4-30 frames regardless of video length. It calculates a target frame count based on the video's native FPS and a default inference rate of 2 FPS, then uniformly samples frames across the timeline. This ensures the model receives representative temporal information while maintaining constant memory usage.

What is the difference between preprocess_frame() and the preprocess() function used for video files?

preprocess_frame() in util/webcamera_utils.py (lines 82-126) is designed for single-frame camera inputs and handles aspect-ratio-preserving padding via adjust_frame_size(), color conversion, and HWC-to-CHW transposition. In contrast, preprocess() in vision_language_model/qwen2_vl/qwen2_vl.py (lines 49-115) accepts lists of already-resized frames and performs patchification—converting images into flattened patch tensors suitable for vision-language models like Qwen2-VL.

Can the pipeline record inference results while processing live camera feeds?

Yes, the get_writer() function in util/webcamera_utils.py (lines 29-75) creates a cv2.VideoWriter instance that supports both local file output (MP4/AVI) and GStreamer TCP streaming. You can initialize the writer with target dimensions and FPS, then write annotated frames inside the inference loop. This allows real-time recording of bounding box overlays, classification labels, or segmentation masks alongside the live processing pipeline.

How does the Basler camera integration differ from standard webcam capture?

BaslerCameraCapture in util/webcamera_utils.py (lines 78-101) abstracts the Pylon SDK specifically for Basler industrial cameras, handling camera initialization, buffer configuration, and hardware-triggered acquisition. Unlike standard cv2.VideoCapture which uses DirectShow/V4L2 backends, the Basler class yields NumPy BGR frames synchronized to the camera's native frame rate and provides proper resource cleanup through context manager protocols.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}