How to Integrate PFLD with Other Face Analysis Tasks: A Complete Pipeline Guide

PFLD provides a compact TensorFlow model that outputs 98 facial landmarks and 3D head pose angles, enabling plug-and-play integration with downstream face analysis tasks such as emotion recognition, face alignment, and attribute classification.

The guoqiangqi/pfld repository implements a lightweight Progressive Face Localization and Detection (PFLD) architecture that separates landmark detection from pose estimation. This dual-output design produces geometrically consistent facial features that you can feed directly into specialized analyzers without retraining the backbone network.

Understanding PFLD's Dual-Output Architecture

The model is designed specifically for integration, exposing two distinct tensors through a shared MobileNet-V2 backbone defined in model2.py.

Facial Landmark Heat-Maps (196-Dimensional Vector)

The pfld_inference() function in model2.py (lines 11-90) extracts intermediate feature maps and feeds them into a multi-scale fully-connected head. This produces landmarks_pre, a 196-dimensional vector that reshapes to 98 (x, y) coordinate pairs representing facial keypoints.

Head Pose Euler Angles (3-Dimensional Vector)

An auxiliary branch sharing early feature maps regresses head orientation via euler_angles_pre. Implemented in model2.py (lines 118-138), this sub-network outputs yaw, pitch, and roll angles without requiring a separate pose estimation model.

Step-by-Step Integration Pipeline

Integrating PFLD into a broader face analysis system follows a five-stage workflow using only TensorFlow, NumPy, and OpenCV operations.

1. Face Detection with MTCNN

The repository includes a standalone MTCNN implementation in mtcnn/detect_face.py. Use this to generate bounding boxes before cropping:

from mtcnn import detect_face

detectors = detect_face.create_mtcnn(sess, None)
bounding_boxes, _ = detect_face.detect_face(
    frame, minsize=20,
    thresholds=[0.6, 0.7, 0.7],
    factor=0.709,
    pnet=detectors[0],
    rnet=detectors[1],
    onet=detectors[2]
)

2. Preprocessing for PFLD Input

Extract each face region and resize to the model's expected input dimensions. PFLD requires 112×112 RGB images normalized to [0, 1]:

face = frame[y1:y2, x1:x2]
face_rgb = cv2.cvtColor(cv2.resize(face, (112, 112)), cv2.COLOR_BGR2RGB)
face_rgb = face_rgb.astype(np.float32) / 256.0
face_rgb = np.expand_dims(face_rgb, 0)  # Add batch dimension

3. Model Inference

Load the pretrained checkpoint and retrieve output tensors by their fixed graph names. The landmark output is fc:0 and the pose vector is pfld_fc2:0:

# Load graph

saver = tf.train.import_meta_graph('./models2/model0/model.meta')
saver.restore(sess, './models2/model0/model.ckpt-0')

# Get tensors

img_ph = graph.get_tensor_by_name('image_batch:0')
phase_ph = graph.get_tensor_by_name('phase_train:0')
landmarks_tensor = graph.get_tensor_by_name('fc:0')      # 98 points

pose_tensor = graph.get_tensor_by_name('pfld_fc2:0')   # Euler angles

# Run inference

landmarks_vec, pose_vec = sess.run(
    [landmarks_tensor, pose_tensor],
    feed_dict={img_ph: face_rgb, phase_ph: False}
)

4. Post-Processing Landmarks

Reshape the 196-dimensional vector into coordinate pairs and denormalize to the original image scale:

landmarks = landmarks_vec.reshape(-1, 2) * np.array([face.shape[1], face.shape[0]])

5. Feeding Downstream Modules

Pass the processed landmarks and pose data to specialized analyzers. The separation of concerns allows you to swap detectors or classifiers while keeping PFLD as a fixed geometric extractor.

Connecting PFLD to Specific Downstream Tasks

The 98-point landmark set and pose vector support multiple analysis pipelines:

• Face Alignment: Convert landmarks into a similarity transform using key points (e.g., eye centers and nose tip) to warp faces to a canonical view before recognition.
• Head-Pose Estimation: Use euler_angles_pre directly for gaze tracking or driver monitoring, or fuse it with geometric solvers for higher precision.
• Facial Expression Recognition: Feed aligned face crops (generated via landmark-based affine transforms) into emotion classification CNNs.
• Face Recognition: Normalize face orientation using the predicted geometry before extracting embeddings from deep recognition networks.
• Attribute Analysis: Define region-of-interest masks (mouth, eyes, cheeks) using specific landmark indices to focus attribute classifiers on relevant facial zones.

Complete Integration Code Example

The following snippet demonstrates a full detect-align-analyze pipeline combining MTCNN detection, PFLD inference, and downstream emotion classification:

import tensorflow as tf
import numpy as np
import cv2
from mtcnn import detect_face

# Initialize session and load PFLD

graph = tf.Graph()
with graph.as_default():
    sess = tf.Session()
    saver = tf.train.import_meta_graph('./models2/model0/model.meta')
    saver.restore(sess, './models2/model0/model.ckpt-0')
    
    img_ph = graph.get_tensor_by_name('image_batch:0')
    phase_ph = graph.get_tensor_by_name('phase_train:0')
    landmarks_tensor = graph.get_tensor_by_name('fc:0')
    pose_tensor = graph.get_tensor_by_name('pfld_fc2:0')

detectors = detect_face.create_mtcnn(sess, None)

def align_face(img, landmarks):
    """Affine transform using left eye (36), right eye (45), nose tip (30)"""
    pts_src = np.float32([landmarks[36], landmarks[45], landmarks[30]])
    pts_dst = np.float32([[30, 30], [82, 30], [56, 70]])
    M = cv2.getAffineTransform(pts_src, pts_dst)
    return cv2.warpAffine(img, M, (112, 112))

def process_frame(frame):
    # Detect faces

    boxes, _ = detect_face.detect_face(frame, 20, [0.6, 0.7, 0.7], 0.709, 
                                      detectors[0], detectors[1], detectors[2])
    
    for bbox in boxes:
        x1, y1, x2, y2 = map(int, bbox[:4])
        face_crop = frame[y1:y2, x1:x2]
        
        # Preprocess

        face_rgb = cv2.cvtColor(cv2.resize(face_crop, (112, 112)), cv2.COLOR_BGR2RGB)
        face_rgb = face_rgb.astype(np.float32) / 256.0
        face_rgb = np.expand_dims(face_rgb, 0)
        
        # Inference

        lm_vec, pose = sess.run([landmarks_tensor, pose_tensor],
                               feed_dict={img_ph: face_rgb, phase_ph: False})
        
        # Post-process

        landmarks = lm_vec.reshape(-1, 2) * np.array([face_crop.shape[1], face_crop.shape[0]])
        
        # Downstream: Align and classify emotion

        aligned = align_face(face_crop, landmarks)
        emotion = emotion_classifier.predict(aligned)  # Your model here

        
        print(f"Pose (yaw, pitch, roll): {pose}")
        print(f"Detected emotion: {emotion}")

# Run on video

cap = cv2.VideoCapture(0)
while True:
    ret, frame = cap.read()
    if not ret:
        break
    process_frame(frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break
cap.release()

Key Source Files for Integration

File	Role
model2.py	Defines the MobileNet-V2 backbone, multi-scale landmark head (pfld_inference), and auxiliary pose branch.
mtcnn/detect_face.py	Stand-alone MTCNN face detector for obtaining bounding boxes before PFLD processing.
utils.py	Implements LandmarkImage and LandmarkImage_98 for converting coordinates to Gaussian heat-maps.
test_model.py	Reference inference script demonstrating checkpoint loading and visualization.
train_model.py	Training loop implementation showing how the auxiliary loss is computed alongside landmark regression.

Summary

• PFLD outputs dual tensors: 196-dimensional landmarks (98 points) and 3-dimensional Euler angles via the architecture in model2.py.
• Input requirements: 112×112 RGB images normalized to [0, 1], typically cropped using the bundled MTCNN detector.
• Graph tensor names: Retrieve landmarks via fc:0 and pose via pfld_fc2:0 when loading checkpoints.
• Plug-and-play design: Use landmarks for face alignment before feeding recognition or emotion networks, or consume pose angles directly for head tracking.
• No external dependencies: The pipeline relies solely on TensorFlow, NumPy, and OpenCV operations defined within the repository.

Frequently Asked Questions

What input image size does PFLD require?

PFLD expects 112×112 pixel RGB images with pixel values normalized to the range [0, 1]. The repository uses this fixed input dimension to maintain consistent landmark localization accuracy across different face scales.

How do I extract usable coordinate pairs from the model output?

The fc:0 tensor outputs a 196-dimensional vector. Reshape this to (-1, 2) to obtain 98 (x, y) coordinate pairs. Multiply these by the original face crop dimensions (width, height) to denormalize coordinates to the source image scale.

Can I disable the pose estimation branch during inference?

Yes. While model2.py defines the auxiliary branch for euler_angles_pre, you can choose not to fetch pfld_fc2:0 during sess.run(). The landmark regression operates independently, so downstream tasks requiring only facial geometry can ignore the pose output entirely.

Which face detector should I use with PFLD?

The repository provides MTCNN in mtcnn/detect_face.py, which is tested and ready to use. However, any detector producing bounding boxes (x1, y1, x2, y2) works provided you resize crops to 112×112 before feeding them to PFLD.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}