How to Implement Pose Estimation Using ST-GCN or MoveNet in Ailia Models

You can implement pose estimation using ST-GCN for action recognition from skeleton sequences or MoveNet for real-time 2D keypoint detection by running the respective Python scripts in the ailia-models repository, which handle model downloading, preprocessing, and visualization automatically.

The ailia-models repository by axinc-ai provides production-ready implementations for two distinct pose estimation approaches. Whether you need to classify human actions from video sequences using spatial-temporal graph convolutional networks or detect body keypoints in real-time using single-shot detectors, the repository offers complete pipelines with automatic model management and hardware acceleration via the Ailia SDK.

Understanding the Two Pose Estimation Pipelines

ST-GCN for Action Recognition

ST-GCN (Spatial-Temporal Graph Convolutional Network) tracks 2D skeletons over time and classifies actions from the resulting joint sequences. This pipeline combines a pose estimator backend with a graph neural network to recognize human activities like walking, dancing, or sports movements from video input.

MoveNet for Real-Time Keypoint Detection

MoveNet is a single-shot 2D keypoint detector that identifies 17 body parts in still images or video streams. Available in Thunder (256×256 input, higher accuracy) and Lightning (192×192 input, faster inference) variants, MoveNet provides frame-by-frame pose estimation without temporal tracking.

Implementing ST-GCN Pose Estimation

Architecture Overview

The ST-GCN pipeline in action_recognition/st_gcn/st_gcn.py processes video through several distinct stages:

1. Input handling – Video frames are read using OpenCV.
2. 2D joint detection – The system loads a pose estimator via ailia.PoseEstimator, supporting backends including OpenPose, PyOpenPose, or LW-Human-Pose ([st_gcn.py lines 70-78]).
3. Coordinate normalization – Detected keypoints undergo pose_postprocess to normalize coordinates to the range [0, 1] ([st_gcn.py lines 104-108]).
4. Temporal tracking – The naive_pose_tracker function in st_gcn_util.py stitches frames into a fixed-length skeleton tensor with shape 3×T×V×M (channels × time × vertices × persons).
5. Graph convolution – The ST-GCN ONNX model (st_gcn.onnx) runs via ailia.Net ([st_gcn.py lines 62-64]), producing action classification logits.
6. Label generation – The postprocess function ([st_gcn.py lines 111-138]) converts raw outputs into voting labels and per-frame label sequences using the KINETICS_LABEL mapping from st_gcn_labels.py.
7. Visualization – The stgcn_visualize function in st_gcn_util.py renders skeletons, heatmaps, and action labels onto the output video.

Key Files and Functions

File	Role
action_recognition/st_gcn/st_gcn.py	CLI driver containing recognize_offline() and recognize_realtime() functions, argument parsing, and model initialization.
action_recognition/st_gcn/st_gcn_util.py	Contains naive_pose_tracker for temporal skeleton assembly and stgcn_visualize for rendering results.
action_recognition/st_gcn/st_gcn_labels.py	Defines KINETICS_LABEL dictionary mapping class indices to human-readable action names from the Kinetics-400 dataset.
pose_estimation/openpose or pose_estimation/lw_human_pose	Pose estimation backends used by ST-GCN for 2D joint detection.

Running ST-GCN

To run ST-GCN action recognition on a video file using the default OpenPose backend:

python3 action_recognition/st_gcn/st_gcn.py \
    --video skateboarding.mp4 \
    --arch openpose

Available architecture options for the pose estimator include openpose, pyopenpose, and lw_human_pose.

To process a video offline and save the rendered output instead of displaying it in a window:

python3 action_recognition/st_gcn/st_gcn.py \
    --input skateboarding.mp4 \
    --savepath result.mp4

Important CLI arguments:

Flag	Description
--fps	Target frames per second for realtime processing mode.
--arch	Pose estimation backend selection (openpose, pyopenpose, lw_human_pose).
--img-save	Save individual frames as PNG files instead of video output.
--env_id	Ailia runtime environment selector for GPU/CPU acceleration.

Implementing MoveNet Pose Estimation

Architecture Overview

The MoveNet implementation in pose_estimation/movenet/movenet.py follows a streamlined single-shot detection pipeline:

1. Model acquisition – The check_and_download_models function automatically fetches movenet_thunder.onnx or movenet_lightning.onnx along with their prototxt files from Google Cloud Storage ([movenet.py lines 88-90]).
2. Input preprocessing – The crop_and_padding function in movenet_utils.py pads input images to square aspect ratio, resizes them to the model's expected resolution (256×256 for Thunder, 192×192 for Lightning), and returns normalized tensors.
3. Inference – The ONNX model executes via ailia.Net or onnxruntime, outputting a heatmap tensor of shape (1, 1, 17, 3) representing 17 body keypoints with x, y coordinates and confidence scores.
4. Coordinate restoration – Post-processing converts normalized coordinates back to the original image space, accounting for padding offsets applied during preprocessing.
5. Visualization – The draw_prediction_on_image function in movenet_utils.py ([lines 71-115]) overlays circles and skeletal lines for all 17 keypoints onto the original image.

For video processing, MoveNet maintains a crop region that follows the person across frames using init_crop_region and determine_crop_region. This tracking reduces jitter and allows consistent input sizing without processing the full frame each time.

Key Files and Functions

File	Role
pose_estimation/movenet/movenet.py	CLI driver handling argument parsing, model initialization, and main inference loops for images and video.
pose_estimation/movenet/movenet_utils.py	Contains crop_and_padding, crop_and_resize for preprocessing, and draw_prediction_on_image for visualization.
util/model_utils.py	Shared helper functions for automatic model downloading and verification.
util/image_utils.py	Wrapper functions for image loading used across both pipelines.

Running MoveNet

To detect pose keypoints in a single image using the default Thunder model:

python3 pose_estimation/movenet/movenet.py \
    --input input.jpg \
    --savepath result.png

To process a video using the lightweight Lightning variant for faster inference:

python3 pose_estimation/movenet/movenet.py \
    --video input.mp4 \
    --model_variant lightning \
    --savepath output.mp4

Essential CLI arguments:

Flag	Description
-i/--input	Path to input image file(s). Multiple files can be specified.
-v/--video	Path to video file or 0 to use webcam input.
-m/--model_variant	Model selection: thunder (default, 256×256) or lightning (192×192).
-o/--onnx	Force usage of onnxruntime instead of the Ailia SDK.
--benchmark	Run warm-up iterations and report average inference time.

Comparing ST-GCN and MoveNet

While both pipelines perform pose estimation, they serve different use cases:

• ST-GCN requires a temporal sequence and classifies actions (e.g., "walking," "waving") using graph convolutions on skeleton data. It depends on an external pose estimator (OpenPose or LW-Human-Pose) to generate the initial 2D joints.

• MoveNet performs single-frame keypoint detection optimized for real-time applications. It outputs 17 body part locations directly without requiring a separate pose estimator, making it ideal for immediate coordinate extraction or low-latency tracking.

Choose ST-GCN when you need to understand what action a person is performing over time, and MoveNet when you need where body parts are located in individual frames.

Summary

• ST-GCN combines 2D pose estimation with temporal tracking via naive_pose_tracker to create skeleton tensors (3×T×V×M) for action classification using graph convolutional networks.
• MoveNet provides single-shot 2D keypoint detection for 17 body parts through two model variants: Thunder (256×256, higher accuracy) and Lightning (192×192, faster inference).
• Both implementations in the ailia-models repository handle automatic model downloading via check_and_download_models, support CPU and GPU acceleration through the Ailia SDK, and include OpenCV-based visualization utilities.
• ST-GCN requires specifying a pose estimation backend (--arch openpose or lw_human_pose), while MoveNet operates as a standalone detector without external pose dependencies.

Frequently Asked Questions

What is the difference between ST-GCN and MoveNet in the ailia-models repository?

ST-GCN is an action recognition pipeline that takes sequences of 2D skeletons over time and classifies the performed action using graph convolutions, while MoveNet is a pose estimation model that detects 17 body keypoints in individual images or video frames without temporal analysis. ST-GCN requires a separate pose estimator backend to generate skeleton data, whereas MoveNet performs detection directly.

How do I choose between the Thunder and Lightning variants of MoveNet?

Choose Thunder (256×256 input resolution) when accuracy is critical and computational resources are available, as it provides more precise keypoint localization. Choose Lightning (192×192 input resolution) for applications requiring higher frame rates or running on edge devices with limited processing power, as it offers significantly faster inference with slightly reduced accuracy.

What pose estimation backends can I use with ST-GCN?

The ST-GCN implementation in action_recognition/st_gcn/st_gcn.py supports three pose estimation backends specified via the --arch argument: openpose (standard OpenPose implementation), pyopenpose (Python OpenPose bindings), and lw_human_pose (lightweight human pose estimator). These backends generate the 2D joint coordinates required by the naive_pose_tracker to construct the skeleton tensor for action classification.

Do I need to manually download the ONNX models before running the scripts?

No, both ST-GCN and MoveNet implementations automatically handle model downloading through the check_and_download_models function found in util/model_utils.py. On first run, the scripts download the required ONNX files and prototxt configurations from Google Cloud Storage to a local models directory, then load them using ailia.Net for inference.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}