How to Improve PFLD Accuracy on Specific Facial Regions or Landmarks

You can improve PFLD accuracy on specific facial regions by adjusting the Gaussian sigma in heatmap generation, applying region-specific loss weights, and fusing high-resolution early features from the MobileNet-V2 backbone.

The PFLD (Pose-Invariant Face Landmark Detection) model in the guoqiangqi/pfld repository implements a lightweight MobileNet-V2-derived backbone that extracts multi-scale features and predicts a 196-dimensional landmark vector (98 × 2 coordinates). While the default configuration treats all facial regions equally, you can significantly boost precision for critical areas like eyes, nose, and mouth by intervening at three specific architectural points in model2.py and utils.py.

Adjust Gaussian Sigma for Sharper Regional Supervision

The ground-truth heatmaps are generated in utils.py by the LandmarkImage and LandmarkImage_98 functions (lines 31‑70). The sigma parameter controls the Gaussian spread, determining how much spatial context each landmark contributes to the loss. A smaller sigma yields sharper peaks, forcing the model to focus on precise locations, while a larger sigma encourages smoother gradients.

To prioritize specific regions, modify the heatmap generation to accept a per-landmark sigma multiplier:

# utils.py – custom sigma map implementation

import tensorflow as tf
import numpy as np

def LandmarkImage_RegionWeighted(Landmarks, image_size, region_weights):
    """
    region_weights: 98-length tensor with a sigma multiplier per landmark.
    Larger sigma → smoother heatmap; smaller sigma → sharper heatmap.
    """
    sigma_base = tf.to_float(tf.reduce_max(image_size[1:3]))/4
    sigma = sigma_base * region_weights          # shape (98,)

    # Expand sigma to per-point operation inside the heatmap loop

    # (same logic as LandmarkImage_98, but replace constant sigma with per-point sigma)

    # …

    return heatmaps

Implementation tip: Eyes and mouth typically require sub-pixel precision. Setting region_weights to approximately 0.5 for the 24 eye landmarks and 20 mouth landmarks narrows the Gaussian, providing a sharper gradient signal exactly where you need higher accuracy.

Apply Region-Specific Loss Weighting

The default training loss in model2.py (lines 1004‑1006) computes an unweighted L2 distance over all 196 outputs. You can penalize errors on critical regions more heavily by multiplying the per-landmark error with a weight map before reduction:

# model2.py – modify create_model function

def create_model(input, landmark, phase_train, args):
    ...
    features, landmarks_pre = pfld_inference(input, args.weight_decay, batch_norm_params)

    # ---------- region-specific weighting ----------

    # weight vector: 1.0 for most points, 2.0 for eyes & mouth

    region_weight = tf.constant(
        [2.0]*24 + [1.0]*30 + [2.0]*20 + [1.0]*22, dtype=tf.float32)  # length 98*2 = 196

    region_weight = tf.reshape(region_weight, [1, -1])               # (1,196)

    # L2 error per coordinate, then apply weights

    diff = tf.square(landmarks_pre - landmark)               # (batch,196)

    weighted_diff = diff * region_weight                     # broadcast

    landmarks_loss = tf.reduce_mean(tf.reduce_sum(weighted_diff, axis=1))
    # -------------------------------------------------

    ...
    return landmarks_pre, landmarks_loss, euler_angles_pre

Why this works: Errors on the weighted landmarks contribute proportionally more to the total loss, steering the optimizer to reduce prediction errors on your target regions preferentially without changing the model architecture.

Fuse High-Resolution Features for Fine Detail

The backbone in model2.py (lines 12‑58) outputs intermediate feature maps (feature2 through feature6) at different resolutions. Early layers retain more spatial detail that gets lost in later downsampling. You can add a side-branch that processes a higher-resolution early feature (e.g., features['feature2']) and fuses it back before the final fully-connected layer:

# model2.py – add after features['feature2'] extraction in pfld_inference

def pfld_inference(...):
    ...
    # existing multi-scale branch continues ...

    # -------------------------------------------------

    # New side-branch: up-sample feature2 and concatenate

    high_res = slim.conv2d(features['feature2'], 64, [1, 1], scope='high_res_reduce')
    high_res = tf.image.resize_images(high_res, [28, 28])    # match later feature map size

    
    # concatenate with the deepest feature before the final FC

    deep = conv7_4   # from the original pipeline (shape 7×7×320)

    deep_up = tf.image.resize_images(deep, [28, 28])
    fused = tf.concat([high_res, deep_up], axis=-1)          # richer representation

    
    # continue with avg-pool, flatten, FC as before

    # -------------------------------------------------

    return landmarks_pre, euler_angles_pre

Performance impact: This fusion gives the final predictor access to both high-level semantic features (from deep layers) and fine-grained spatial details (from early layers), significantly improving localization accuracy for small or finely-structured regions like eye corners and lip boundaries.

Data-Centric Optimization Strategies

Beyond architectural changes, adjust your data pipeline in generate_data.py to support regional accuracy improvements:

• Balanced pose distribution: Oversample under-represented yaw and pitch angles in your training list to prevent bias. Modify the DateSet class to apply weighted sampling when parsing list.txt.

• Region-aware augmentations: Implement random cropping and scaling focused specifically on eye or mouth areas before tf.image.resize_images in the _parse_data function. This forces the network to learn invariance to local occlusions.

• Higher input resolution: Increase args.image_size from 112 to 224 in your training arguments. This improves the granularity of feature maps throughout the network, directly benefiting fine-grained landmark detection, though it requires additional GPU memory.

Training Configuration Considerations

When implementing region-specific modifications, adjust your optimization strategy accordingly:

• Learning rate schedule: The piecewise-constant schedule in utils.py (lines 5‑12) works well for standard training, but region-specific loss functions may benefit from a slower decay. Add an extra epoch boundary to maintain higher learning rates longer for the weighted loss components.

• Batch size: Larger batches stabilize the gradients when using weighted loss terms. If GPU memory permits, increase args.batch_size beyond the default to reduce variance in the region-specific gradient updates.

Summary

• Sharpen supervision: Reduce Gaussian sigma in LandmarkImage_98 for critical landmarks to provide tighter spatial targets.
• Emphasize errors: Apply a weight vector in create_model to multiply the L2 loss of priority regions (eyes, mouth) by factors of 2× or higher.
• Preserve detail: Fuse early high-resolution features (feature2) with deep semantic features before the final prediction layer.
• Balance data: Use pose-balanced sampling and region-focused augmentations in generate_data.py to ensure robust generalization.

Frequently Asked Questions

How do I identify which landmarks correspond to eyes versus mouth in the 98-point format?

The PFLD 98-point annotation follows a standard ordering where landmarks 0‑23 typically represent the left and right eyes (12 points per eye), 24‑53 cover the nose and cheeks, 54‑73 define the mouth (20 points), and 74‑97 mark the face contour. You should verify the exact indices in your specific dataset's documentation, but this ordering allows you to construct the region_weight vector with [2.0]*24 for eyes and [2.0]*20 for the mouth region.

Will increasing input resolution from 112 to 224 significantly improve accuracy?

Yes, doubling the input resolution improves the granularity of all feature maps in the MobileNet-V2 backbone, which directly benefits the localization of fine details like eye corners. However, this increases GPU memory consumption quadratically for feature maps and requires adjusting the heatmap generation dimensions in LandmarkImage. The trade-off is worthwhile when targeting sub-pixel accuracy on high-resolution facial regions.

Can I combine all three methods (sigma tuning, loss weighting, and feature fusion) simultaneously?

Absolutely. These interventions operate at different stages—data preparation (sigma), loss computation (weighting), and architecture (fusion)—so they complement each other. Start with loss weighting for immediate impact, then add sigma tuning for sharper supervision, and finally implement feature fusion if you need the last mile of precision on specific regions.

Does modifying the Gaussian sigma require regenerating my entire training dataset?

Yes, because LandmarkImage_98 generates heatmaps during the data loading phase in generate_data.py. You must replace the default heatmap generation call with your custom LandmarkImage_RegionWeighted function and reprocess your training data. Alternatively, generate heatmaps on-the-fly during training if storage is limited, though this increases CPU load during the training loop.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}