How to Implement Gesture and Hand Detection Models with Ailia‑Models

The Ailia‑Models repository provides ready‑to‑run ONNX pipelines that combine hand detection, 21‑point landmark estimation, and gesture classification into a single inference stack.

You can implement gesture and hand detection models by leveraging the modular architecture in the axinc-ai/ailia-models repository, which separates detection, pose estimation, and classification into discrete, swappable components. Each model ships as an ONNX file paired with a .prototxt descriptor, allowing you to run inference via the Ailia‑SDK either through convenience scripts or direct API calls. This guide walks through the three‑layer pipeline, provides runnable code snippets, and identifies the critical source files you need to customize.

Three‑Layer Pipeline Architecture

The repository organizes hand‑related functionality into three logical layers. Understanding this separation lets you swap individual components—such as replacing the detector for higher accuracy or injecting a custom classifier—without rewriting the entire stack.

Layer 1: Hand Detection

Hand detection locates one or more hands in an image or video frame and returns bounding box coordinates. The repository offers multiple backbone options:

• YOLO‑v3 (hand_detection/yolov3-hand): General‑purpose object detector tuned for hands.
• PyTorch hand detector (hand_detection/hand_detection_pytorch): Lightweight CNN trained on hand datasets.
• BlazePalm (hand_detection/blazepalm): MediaPipe‑style palm detector optimized for mobile inference.

In hand_detection/hand_detection_pytorch/hand_detection_pytorch.py, the detector loads an ONNX model expecting a 320 × 320 input tensor normalized to [0, 1]. The inference returns an N×4 array of bounding boxes (x1, y1, x2, y2) and confidence scores.

Layer 2: Hand Pose Estimation

Once a hand is localized, pose estimation extracts 21 3D landmarks (finger joints and palm keypoints) from the cropped region. Key implementations include:

• BlazeHand (hand_recognition/blazehand): Outputs 21 landmarks plus a handedness flag.
• Minimal‑hand (hand_recognition/minimal-hand): Direct regression of landmarks with built‑in gesture labels.
• Hand‑3D (hand_recognition/hand3d): Depth‑aware pose recovery.

The BlazeHand model in hand_recognition/blazehand/blazehand.py accepts a 224 × 224 RGB crop and returns a tensor of shape (1, 21, 3) representing normalized (x, y, z) coordinates.

Layer 3: Gesture Classification

Gesture classification maps the 21‑point skeleton to human‑readable labels such as "thumbs‑up", "peace", or "fist". The BlazeHand script embeds a lightweight rule‑based classifier that inspects landmark geometric relationships (lines 139‑146 in blazehand.py). Alternatively, you can replace this logic with a trained neural network that consumes the landmark array as input features.

End‑to‑End Data Flow

When you run the integrated pipeline, data flows through five distinct stages:

1. Pre‑processing: Input images are resized, padded, and normalized to the detector’s input resolution (e.g., 320 × 320).
2. Detection inference: The detector returns bounding boxes and confidence scores.
3. ROI extraction: High‑confidence boxes are cropped, optionally rotated to canonical orientation, and resized for the pose model.
4. Pose inference: The pose estimator outputs heatmaps or direct landmark coordinates.
5. Post‑processing: Landmarks are transformed back to the original image coordinate system, and the gesture classifier produces the final label.

Running Pre‑Built Models via CLI

For rapid prototyping, each model folder contains a driver script that wraps the entire pipeline. To detect hands, estimate pose, and classify gestures in a single command:

python3 hand_recognition/blazehand/blazehand.py \
    --input sample.jpg \
    --savepath output.png \
    --hands 2

The --hands argument limits the maximum number of simultaneous detections. For real‑time webcam input, substitute --input with --video 0 to stream from the default camera device.

Implementing Custom Pipelines with the Ailia‑SDK

When you need tighter integration into a larger application, instantiate AiliaModel directly instead of using the CLI wrappers.

Hand Detection (PyTorch Backend)

This snippet demonstrates loading the PyTorch‑based detector and filtering results by confidence threshold:

from ailia import AiliaModel
import cv2
import numpy as np

# Load detector

detector = AiliaModel(
    model_path='hand_detection/hand_detection_pytorch/hand_detection_pytorch.onnx',
    proto_path='hand_detection/hand_detection_pytorch/hand_detection_pytorch.onnx.prototxt',
    weight_path='hand_detection/hand_detection_pytorch/hand_detection_pytorch.onnx'
)

# Prepare image

img = cv2.imread('sample.jpg')
blob = cv2.resize(img, (320, 320))
blob = blob.astype(np.float32) / 255.0

# Run inference

detector.set_input(blob)
boxes, scores = detector.run()

# Filter by confidence

conf_thr = 0.5
keep = scores[:, 0] > conf_thr
boxes = boxes[keep]

Source file: hand_detection/hand_detection_pytorch/hand_detection_pytorch.py

Pose Estimation with BlazeHand

After obtaining bounding boxes, extract ROIs and run the pose estimator:

# Load pose model

pose = AiliaModel(
    model_path='hand_recognition/blazehand/blazehand.onnx',
    proto_path='hand_recognition/blazehand/blazehand.onnx.prototxt',
    weight_path='hand_recognition/blazehand/blazehand.onnx'
)

# Process each detection

for (x1, y1, x2, y2) in boxes:
    roi = img[int(y1):int(y2), int(x1):int(x2)]
    roi = cv2.resize(roi, (224, 224)).astype(np.float32) / 255.0
    
    pose.set_input(roi)
    flags, handedness, landmarks = pose.run()
    # landmarks shape: (1, 21, 3)

    
    # handedness > 0.5 indicates right hand

    is_right = handedness[0] > 0.5

The landmarks array contains 21 normalized 3D coordinates ready for gesture analysis or custom classifiers.

Adding a Custom Gesture Classifier

To recognize domain‑specific gestures, train a small fully‑connected network that consumes the 21 × 3 landmark tensor:

import torch
import torch.nn as nn

class GestureNet(nn.Module):
    def __init__(self, n_classes=10):
        super().__init__()
        self.fc = nn.Sequential(
            nn.Linear(21 * 3, 128),
            nn.ReLU(inplace=True),
            nn.Linear(128, n_classes)
        )
    
    def forward(self, x):
        x = x.view(x.size(0), -1)  # Flatten (B, 21, 3) → (B, 63)

        return self.fc(x)

# Inference example

model = GestureNet(n_classes=5)
landmarks = ...  # From BlazeHand, shape (1, 21, 3)

logits = model(torch.from_numpy(landmarks).float())
gesture_id = logits.argmax(dim=1).item()

Replace the built‑in classification block in blazehand.py (around lines 139‑146) with your trained GestureNet to deploy custom gestures.

Key Source Files and Repository Structure

Understanding the file layout helps you locate specific logic for modification:

• hand_detection/hand_detection_pytorch/hand_detection_pytorch.py: End‑to‑end detector driver with CLI argument parsing and NMS post‑processing.
• hand_detection/blazepalm/blazepalm.py: Real‑time palm detection optimized for edge devices.
• hand_recognition/blazehand/blazehand.py: Integrated detection → pose → gesture pipeline; contains the built‑in gesture mapping logic.
• hand_recognition/minimal-hand/minimal-hand.py: Lightweight alternative that outputs landmarks and gesture labels in a single forward pass.
• util/load_model.py: Helper utility that abstracts AiliaModel initialization and handles .onnx + .prototxt pairing.

All weights can be downloaded in bulk using scripts/download_all_models.sh, which pulls the latest ONNX files from the repository’s release artifacts.

Summary

• Ailia‑Models structures hand analysis into three layers: detection, pose estimation, and gesture classification.
• CLI scripts such as blazehand.py provide immediate inference capabilities without writing boilerplate code.
• Direct SDK usage via AiliaModel lets you integrate the pipeline into production applications and control preprocessing explicitly.
• Landmark-based classification allows you to swap the default gesture recognizer with custom neural networks trained on the 21‑point skeleton output.

Frequently Asked Questions

What input resolution do the hand detection models require?

The PyTorch hand detector expects 320 × 320 pixel inputs normalized to the range [0, 1], while BlazeHand requires 224 × 224 crops for pose estimation. The convenience scripts automatically handle resizing and padding, but when using the SDK directly you must perform these transformations manually before calling set_input().

Can I run these models on a webcam stream in real time?

Yes. Pass the --video argument followed by the device index (typically 0) to any driver script. For example: python3 hand_recognition/blazehand/blazehand.py --video 0 --hands 2. The scripts use OpenCV to capture frames and maintain a detection‑to‑pose pipeline that runs at interactive frame rates on modern GPUs.

How do I add custom gestures beyond the built-in labels?

Extract the 21 × 3 landmark array from BlazeHand or Minimal‑hand, then train a small classifier (e.g., a two‑layer MLP) on your labeled landmark data. Replace the gesture mapping logic in blazehand.py lines 139‑146 with your model’s inference code. Since the classifier operates on skeletal coordinates rather than raw pixels, it requires minimal training data and runs inference in microseconds.

Which detector should I choose for high-accuracy batch processing versus real-time mobile inference?

For batch processing on desktop GPUs, use the YOLO‑v3 detector (hand_detection/yolov3-hand) or the PyTorch detector for higher mAP. For real‑time mobile inference, select BlazePalm (hand_detection/blazepalm), which uses a lighter backbone designed for ARM processors and achieves 30+ FPS on mid‑range mobile devices.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}