# How to Implement Gesture and Hand Detection Models with Ailia‑Models

> Implement gesture and hand detection models with ailia-models. Discover ready-to-run ONNX pipelines for hand detection, landmark estimation, and gesture classification in one inference stack.

- Repository: [axinc-ai/ailia-models](https://github.com/axinc-ai/ailia-models)
- Tags: how-to-guide
- Published: 2026-02-26

---

**The Ailia‑Models repository provides ready‑to‑run ONNX pipelines that combine hand detection, 21‑point landmark estimation, and gesture classification into a single inference stack.**

You can implement gesture and hand detection models by leveraging the modular architecture in the `axinc-ai/ailia-models` repository, which separates detection, pose estimation, and classification into discrete, swappable components. Each model ships as an ONNX file paired with a `.prototxt` descriptor, allowing you to run inference via the Ailia‑SDK either through convenience scripts or direct API calls. This guide walks through the three‑layer pipeline, provides runnable code snippets, and identifies the critical source files you need to customize.

## Three‑Layer Pipeline Architecture

The repository organizes hand‑related functionality into three logical layers. Understanding this separation lets you swap individual components—such as replacing the detector for higher accuracy or injecting a custom classifier—without rewriting the entire stack.

### Layer 1: Hand Detection

**Hand detection** locates one or more hands in an image or video frame and returns bounding box coordinates. The repository offers multiple backbone options:

- **YOLO‑v3** (`hand_detection/yolov3-hand`): General‑purpose object detector tuned for hands.
- **PyTorch hand detector** (`hand_detection/hand_detection_pytorch`): Lightweight CNN trained on hand datasets.
- **BlazePalm** (`hand_detection/blazepalm`): MediaPipe‑style palm detector optimized for mobile inference.

In [`hand_detection/hand_detection_pytorch/hand_detection_pytorch.py`](https://github.com/axinc-ai/ailia-models/blob/main/hand_detection/hand_detection_pytorch/hand_detection_pytorch.py), the detector loads an ONNX model expecting a **320 × 320** input tensor normalized to `[0, 1]`. The inference returns an `N×4` array of bounding boxes `(x1, y1, x2, y2)` and confidence scores.

### Layer 2: Hand Pose Estimation

Once a hand is localized, **pose estimation** extracts 21 3D landmarks (finger joints and palm keypoints) from the cropped region. Key implementations include:

- **BlazeHand** (`hand_recognition/blazehand`): Outputs 21 landmarks plus a handedness flag.
- **Minimal‑hand** (`hand_recognition/minimal-hand`): Direct regression of landmarks with built‑in gesture labels.
- **Hand‑3D** (`hand_recognition/hand3d`): Depth‑aware pose recovery.

The BlazeHand model in [`hand_recognition/blazehand/blazehand.py`](https://github.com/axinc-ai/ailia-models/blob/main/hand_recognition/blazehand/blazehand.py) accepts a **224 × 224** RGB crop and returns a tensor of shape `(1, 21, 3)` representing normalized `(x, y, z)` coordinates.

### Layer 3: Gesture Classification

**Gesture classification** maps the 21‑point skeleton to human‑readable labels such as "thumbs‑up", "peace", or "fist". The BlazeHand script embeds a lightweight rule‑based classifier that inspects landmark geometric relationships (lines 139‑146 in [`blazehand.py`](https://github.com/axinc-ai/ailia-models/blob/main/blazehand.py)). Alternatively, you can replace this logic with a trained neural network that consumes the landmark array as input features.

## End‑to‑End Data Flow

When you run the integrated pipeline, data flows through five distinct stages:

1. **Pre‑processing**: Input images are resized, padded, and normalized to the detector’s input resolution (e.g., 320 × 320).
2. **Detection inference**: The detector returns bounding boxes and confidence scores.
3. **ROI extraction**: High‑confidence boxes are cropped, optionally rotated to canonical orientation, and resized for the pose model.
4. **Pose inference**: The pose estimator outputs heatmaps or direct landmark coordinates.
5. **Post‑processing**: Landmarks are transformed back to the original image coordinate system, and the gesture classifier produces the final label.

## Running Pre‑Built Models via CLI

For rapid prototyping, each model folder contains a driver script that wraps the entire pipeline. To detect hands, estimate pose, and classify gestures in a single command:

```bash
python3 hand_recognition/blazehand/blazehand.py \
    --input sample.jpg \
    --savepath output.png \
    --hands 2

```

The `--hands` argument limits the maximum number of simultaneous detections. For real‑time webcam input, substitute `--input` with `--video 0` to stream from the default camera device.

## Implementing Custom Pipelines with the Ailia‑SDK

When you need tighter integration into a larger application, instantiate `AiliaModel` directly instead of using the CLI wrappers.

### Hand Detection (PyTorch Backend)

This snippet demonstrates loading the PyTorch‑based detector and filtering results by confidence threshold:

```python
from ailia import AiliaModel
import cv2
import numpy as np

# Load detector

detector = AiliaModel(
    model_path='hand_detection/hand_detection_pytorch/hand_detection_pytorch.onnx',
    proto_path='hand_detection/hand_detection_pytorch/hand_detection_pytorch.onnx.prototxt',
    weight_path='hand_detection/hand_detection_pytorch/hand_detection_pytorch.onnx'
)

# Prepare image

img = cv2.imread('sample.jpg')
blob = cv2.resize(img, (320, 320))
blob = blob.astype(np.float32) / 255.0

# Run inference

detector.set_input(blob)
boxes, scores = detector.run()

# Filter by confidence

conf_thr = 0.5
keep = scores[:, 0] > conf_thr
boxes = boxes[keep]

```

*Source file:* [`hand_detection/hand_detection_pytorch/hand_detection_pytorch.py`](https://github.com/axinc-ai/ailia-models/blob/main/hand_detection/hand_detection_pytorch/hand_detection_pytorch.py)

### Pose Estimation with BlazeHand

After obtaining bounding boxes, extract ROIs and run the pose estimator:

```python

# Load pose model

pose = AiliaModel(
    model_path='hand_recognition/blazehand/blazehand.onnx',
    proto_path='hand_recognition/blazehand/blazehand.onnx.prototxt',
    weight_path='hand_recognition/blazehand/blazehand.onnx'
)

# Process each detection

for (x1, y1, x2, y2) in boxes:
    roi = img[int(y1):int(y2), int(x1):int(x2)]
    roi = cv2.resize(roi, (224, 224)).astype(np.float32) / 255.0
    
    pose.set_input(roi)
    flags, handedness, landmarks = pose.run()
    # landmarks shape: (1, 21, 3)

    
    # handedness > 0.5 indicates right hand

    is_right = handedness[0] > 0.5

```

The `landmarks` array contains 21 normalized 3D coordinates ready for gesture analysis or custom classifiers.

### Adding a Custom Gesture Classifier

To recognize domain‑specific gestures, train a small fully‑connected network that consumes the 21 × 3 landmark tensor:

```python
import torch
import torch.nn as nn

class GestureNet(nn.Module):
    def __init__(self, n_classes=10):
        super().__init__()
        self.fc = nn.Sequential(
            nn.Linear(21 * 3, 128),
            nn.ReLU(inplace=True),
            nn.Linear(128, n_classes)
        )
    
    def forward(self, x):
        x = x.view(x.size(0), -1)  # Flatten (B, 21, 3) → (B, 63)

        return self.fc(x)

# Inference example

model = GestureNet(n_classes=5)
landmarks = ...  # From BlazeHand, shape (1, 21, 3)

logits = model(torch.from_numpy(landmarks).float())
gesture_id = logits.argmax(dim=1).item()

```

Replace the built‑in classification block in [`blazehand.py`](https://github.com/axinc-ai/ailia-models/blob/main/blazehand.py) (around lines 139‑146) with your trained `GestureNet` to deploy custom gestures.

## Key Source Files and Repository Structure

Understanding the file layout helps you locate specific logic for modification:

- **[`hand_detection/hand_detection_pytorch/hand_detection_pytorch.py`](https://github.com/axinc-ai/ailia-models/blob/main/hand_detection/hand_detection_pytorch/hand_detection_pytorch.py)**: End‑to‑end detector driver with CLI argument parsing and NMS post‑processing.
- **[`hand_detection/blazepalm/blazepalm.py`](https://github.com/axinc-ai/ailia-models/blob/main/hand_detection/blazepalm/blazepalm.py)**: Real‑time palm detection optimized for edge devices.
- **[`hand_recognition/blazehand/blazehand.py`](https://github.com/axinc-ai/ailia-models/blob/main/hand_recognition/blazehand/blazehand.py)**: Integrated detection → pose → gesture pipeline; contains the built‑in gesture mapping logic.
- **[`hand_recognition/minimal-hand/minimal-hand.py`](https://github.com/axinc-ai/ailia-models/blob/main/hand_recognition/minimal-hand/minimal-hand.py)**: Lightweight alternative that outputs landmarks and gesture labels in a single forward pass.
- **[`util/load_model.py`](https://github.com/axinc-ai/ailia-models/blob/main/util/load_model.py)**: Helper utility that abstracts `AiliaModel` initialization and handles `.onnx` + `.prototxt` pairing.

All weights can be downloaded in bulk using [`scripts/download_all_models.sh`](https://github.com/axinc-ai/ailia-models/blob/main/scripts/download_all_models.sh), which pulls the latest ONNX files from the repository’s release artifacts.

## Summary

- **Ailia‑Models** structures hand analysis into three layers: detection, pose estimation, and gesture classification.
- **CLI scripts** such as [`blazehand.py`](https://github.com/axinc-ai/ailia-models/blob/main/blazehand.py) provide immediate inference capabilities without writing boilerplate code.
- **Direct SDK usage** via `AiliaModel` lets you integrate the pipeline into production applications and control preprocessing explicitly.
- **Landmark-based classification** allows you to swap the default gesture recognizer with custom neural networks trained on the 21‑point skeleton output.

## Frequently Asked Questions

### What input resolution do the hand detection models require?

The PyTorch hand detector expects **320 × 320** pixel inputs normalized to the range `[0, 1]`, while BlazeHand requires **224 × 224** crops for pose estimation. The convenience scripts automatically handle resizing and padding, but when using the SDK directly you must perform these transformations manually before calling `set_input()`.

### Can I run these models on a webcam stream in real time?

Yes. Pass the `--video` argument followed by the device index (typically `0`) to any driver script. For example: `python3 hand_recognition/blazehand/blazehand.py --video 0 --hands 2`. The scripts use OpenCV to capture frames and maintain a detection‑to‑pose pipeline that runs at interactive frame rates on modern GPUs.

### How do I add custom gestures beyond the built-in labels?

Extract the **21 × 3 landmark array** from BlazeHand or Minimal‑hand, then train a small classifier (e.g., a two‑layer MLP) on your labeled landmark data. Replace the gesture mapping logic in [`blazehand.py`](https://github.com/axinc-ai/ailia-models/blob/main/blazehand.py) lines 139‑146 with your model’s inference code. Since the classifier operates on skeletal coordinates rather than raw pixels, it requires minimal training data and runs inference in microseconds.

### Which detector should I choose for high-accuracy batch processing versus real-time mobile inference?

For **batch processing** on desktop GPUs, use the **YOLO‑v3** detector (`hand_detection/yolov3-hand`) or the PyTorch detector for higher mAP. For **real‑time mobile inference**, select **BlazePalm** (`hand_detection/blazepalm`), which uses a lighter backbone designed for ARM processors and achieves 30+ FPS on mid‑range mobile devices.