Deep-Live-Cam Mouth Mask Feature: How Lower Lip Landmarks Drive Face Swap Isolation

The mouth mask feature isolates the lower lip using landmarks 52-63 from the 106-point facial landmark set, expands the polygon via configurable scaling factors, and blends swapped mouths back into frames using GPU-accelerated Gaussian blur and feathered masking.

The Deep-Live-Cam repository implements real-time face swapping with surgical precision by targeting specific facial regions. The mouth mask feature zeroes in on the lower lip area using precise lower lip landmarks to ensure seamless mouth replacement without affecting surrounding facial features. This implementation relies on the 106-point landmark model and configurable expansion parameters defined in the codebase.

Landmark-Based Isolation in face_masking.py

The core algorithm lives in modules/processors/frame/face_masking.py, where the create_lower_mouth_mask function processes detected faces to generate binary masks. The system extracts the 106-point facial landmark array from the face analyser and immediately filters for the outer mouth contour.

Selecting Lower Lip Landmarks 52-63

The algorithm targets indices 52 through 63 (inclusive) from the landmark array, which correspond exclusively to the outer lip contour. This selection excludes chin and jaw points, ensuring the mask covers only the mouth region.

landmarks = face.landmark_2d_106  # face_masking.py#L83-L86

lower_lip_order = list(range(52, 64))  # face_masking.py#L85-L87

lower_lip_landmarks = landmarks[lower_lip_order].astype(np.float32)  # face_masking.py#L91-L92

Geometric Center and Expansion Logic

After isolating the twelve lower lip points, the code calculates the arithmetic mean to find the polygon center. It then scales each point outward using the formula 1 + mask_down_size * mouth_mask_size, creating a safety margin around the lip.

center = np.mean(lower_lip_landmarks, axis=0)  # face_masking.py#L94-L95

expansion_factor = (
    1 + modules.globals.mask_down_size * modules.globals.mouth_mask_size
)  # face_masking.py#L98-L100

expanded_landmarks = (lower_lip_landmarks - center) * expansion_factor + center
expanded_landmarks = expanded_landmarks.astype(np.int32)  # face_masking.py#L101-L106

Building the Binary Mask and ROI

With the expanded polygon defined, the system constructs a binary mask limited to the mouth region. This process involves calculating padded bounding boxes and drawing filled polygons relative to the region of interest.

Padded Bounding Box Calculation

The code derives the tight bounding box from the expanded landmarks and adds 10% padding relative to the mask width. This prevents edge clipping during the blending phase.

min_x, min_y = np.min(expanded_landmarks, axis=0)  # face_masking.py#L108-L109

max_x, max_y = np.max(expanded_landmarks, axis=0)  # face_masking.py#L110-L111

padding = int((max_x - min_x) * 0.1)  # face_masking.py#L112-L114

min_x = max(0, min_x - padding)  # face_masking.py#L115-L116

min_y = max(0, min_y - padding)
max_x = min(frame.shape[1], max_x + padding)  # face_masking.py#L117-L118

max_y = min(frame.shape[0], max_y + padding)

Polygon Drawing and ROI Mask Generation

The algorithm creates a blank ROI mask sized to the padded bounding box, translates the expanded landmarks to ROI-relative coordinates, and fills the polygon with white (255).

mask_roi = np.zeros((max_y - min_y, max_x - min_x), dtype=np.uint8)  # face_masking.py#L126-L127

polygon_relative_to_roi = expanded_landmarks - [min_x, min_y]  # face_masking.py#L129-L130

cv2.fillPoly(mask_roi, [polygon_relative_to_roi], 255)  # face_masking.py#L131-L132

GPU-Accelerated Edge Feathering

To eliminate hard edges during blending, the mouth mask feature applies GPU-accelerated Gaussian blur to the binary mask. The implementation uses a 15-pixel kernel with sigma 5 to create a soft alpha gradient.

mask_roi = gpu_gaussian_blur(mask_roi, (15, 15), 5)  # face_masking.py#L132-L134

mask[min_y:max_y, min_x:max_x] = mask_roi  # face_masking.py#L135-L136

This feathering ensures the swapped mouth integrates naturally with the original skin texture and lighting conditions.

Extracting and Reinserting the Mouth Cutout

The completed mask enables surgical extraction and replacement of mouth regions without disturbing the broader face structure.

Original Mouth Patch Extraction

Before swapping occurs, the system preserves the original mouth pixels using the calculated bounding box.

mouth_cutout = frame[min_y:max_y, min_x:max_x].copy()  # face_masking.py#L138-L139

Blending with apply_mask_area

During the face swap pipeline, the apply_mask_area function (defined at face_masking.py#L21-L50) resizes the swapped mouth to fit the ROI, performs color transfer, and blends it back using the feathered polygon mask combined with the global face mask.

The function returns four critical artifacts:

• mask: Full-frame binary mask
• mouth_cutout: Original pixel data for the mouth region
• box: Tuple of (xmin, ymin, xmax, ymax) coordinates
• expanded_landmarks: Polygon vertices in original frame coordinates

Configuration Parameters in globals.py

The expansion behavior is controlled by two global parameters defined in modules/globals.py:

• mask_down_size: Base expansion factor (default 0.1)
• mouth_mask_size: User-tunable multiplier (default 1.0)

These values multiply to determine the final polygon expansion, allowing fine-tuning for different face shapes and mouth sizes.

Code Example: Implementing the Mouth Mask Pipeline

The following snippet demonstrates direct usage of the lower lip mask API outside the automatic face swap processor:

import cv2
from modules.processors.frame.face_masking import create_lower_mouth_mask
from modules.processors.frame.face_masking import apply_mask_area
import modules.globals

# Configure expansion (optional)

modules.globals.mask_down_size = 0.1
modules.globals.mouth_mask_size = 1.2

# Generate mask from detected face

mask, cutout, box, polygon = create_lower_mouth_mask(detected_face, frame)

# Assume new_mouth contains the swapped texture

result = apply_mask_area(
    frame=frame,
    cutout=new_mouth,
    box=box,
    face_mask=mask,
    polygon=polygon
)

cv2.imwrite("swapped_output.png", result)

Summary

• The mouth mask feature targets landmarks 52-63 from the 106-point facial landmark set to isolate the lower lip exclusively.
• Polygon expansion uses the formula 1 + mask_down_size * mouth_mask_size to create configurable safety margins around the mouth.
• 10% padding is added to the bounding box to prevent edge clipping during extraction.
• GPU-accelerated Gaussian blur with a 15x15 kernel and sigma 5 feathers the mask edges for seamless blending.
• The primary implementation resides in modules/processors/frame/face_masking.py, with duplicate logic in face_swapper.py.
• The apply_mask_area function handles color transfer and final compositing of the swapped mouth region.

Frequently Asked Questions

Which landmark indices define the lower lip mask in Deep-Live-Cam?

The lower lip mask exclusively uses landmarks 52 through 63 from the 106-point facial landmark array. These indices correspond to the outer lip contour, deliberately excluding chin and jaw points to restrict the mask strictly to the mouth region.

How does the mouth_mask_size parameter affect the mask?

The mouth_mask_size parameter acts as a user-tunable multiplier that scales the base expansion factor. Combined with mask_down_size (default 0.1) via the formula 1 + mask_down_size * mouth_mask_size, it controls how far the polygon expands outward from the geometric center of the lower lip landmarks.

Why is GPU Gaussian blur used instead of CPU blur?

The implementation calls gpu_gaussian_blur with a 15x15 kernel and sigma 5 to accelerate the feathering operation. This GPU acceleration ensures real-time performance during face swapping while softening the binary mask edges to prevent visible seams between the original frame and the swapped mouth region.

Where is the mouth mask logic duplicated besides face_masking.py?

An identical implementation of the lower lip masking logic exists in modules/processors/frame/face_swapper.py (lines 32-42), where it performs the same landmark selection and expansion during the active face swap processing pipeline, though face_masking.py contains the canonical version with the full feathering and extraction pipeline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}