How Deep-Live-Cam Applies Color Transfer to the Mouth Region During Face Swaps

Deep-Live-Cam matches the mouth cut-out's color statistics to the swapped face ROI using a LAB color-space transfer algorithm implemented in modules/processors/frame/face_swapper.py.

Deep-Live-Cam synchronizes lip color and illumination with synthetic skin tones by statistically adapting the original mouth cut-out to the lighting conditions of the generated face. This ensures seamless compositing when the mouth region is blended back into the face-swap output. The color transfer implementation relies on per-channel mean and standard deviation matching in CIELAB space, with robust fallback handling to prevent pipeline failures.

The Mouth Region Color Transfer Pipeline

When processing a frame, Deep-Live-Cam executes a multi-step color correction workflow inside the face-swapping logic. The pipeline extracts the region of interest, normalizes the mouth cut-out's color distribution, and composites it with feathered edges.

Extracting the ROI and Resizing

After the initial face swap occurs, the code isolates the rectangular region containing the swapped face. The system calculates bounding coordinates (min_y:max_y, min_x:max_x) to extract the ROI from the frame. The original mouth cut-out is then resized to match these dimensions using GPU-accelerated operations before color processing begins.

Validating Image Channels

Before invoking the transfer algorithm, the implementation verifies that both images contain three BGR channels. This validation occurs at lines 65–74 in modules/processors/frame/face_swapper.py:

# Ensure both images are 3 channels for color transfer

if len(resized_mouth_cutout.shape) == 3 and resized_mouth_cutout.shape[2] == 3 and \
   len(roi.shape) == 3 and roi.shape[2] == 3:
    color_corrected_mouth = apply_color_transfer(resized_mouth_cutout, roi)

If the channel check fails, the pipeline proceeds with the uncorrected resized mouth cut-out, ensuring the frame remains usable.

LAB Color Space Implementation

The core color adaptation logic resides in the apply_color_transfer helper function (lines 1147–1262). This function performs a classic statistical color transfer by aligning the mouth cut-out's LAB color distribution with that of the target ROI.

The apply_color_transfer Algorithm

The function converts both source (mouth) and target (ROI) images from BGR to the CIELAB color space, which separates luminance from color information. It then scales the source's per-channel values to match the target's statistical moments:

def apply_color_transfer(source, target):
    """Apply color transfer using LAB color space."""
    source_float = source.astype(np.float32) / 255.0
    target_float = target.astype(np.float32) / 255.0
    source_lab = cv2.cvtColor(source_float, cv2.COLOR_BGR2LAB)
    target_lab = cv2.cvtColor(target_float, cv2.COLOR_BGR2LAB)

    source_mean, source_std = cv2.meanStdDev(source_lab)
    target_mean, target_std = cv2.meanStdDev(target_lab)

    epsilon = 1e-6
    source_std = np.maximum(source_std, epsilon)

    result_lab = (source_lab - source_mean) * (target_std / source_std) + target_mean
    result_bgr_float = cv2.cvtColor(result_lab, cv2.COLOR_LAB2BGR)
    result_bgr = (np.clip(result_bgr_float, 0.0, 1.0) * 255.0).astype("uint8")
    return result_bgr

This algorithm computes mean and standard deviation for each LAB channel, then linearly transforms the source pixels so that (source - mean) / std matches the target's distribution. The epsilon value prevents division-by-zero errors on uniform regions.

Blending and Compositing

After color correction, the system blends the adapted mouth region back into the swapped face using a feathered polygon mask. This creates smooth transitions at the mouth boundaries and prevents hard edges.

GPU-Accelerated Preparation

The resizing operations leverage GPU acceleration via gpu_resize from modules/gpu_processing.py, ensuring real-time performance during live camera feeds. The color transfer itself operates on CPU-processed numpy arrays after GPU preprocessing.

Feathered Mask Integration

The corrected mouth is combined with the ROI using an inverse mask strategy. The typical integration inside the face-swap pipeline (around lines 86–100) follows this pattern:

# After resizing the mouth cut-out to ROI size...

color_corrected_mouth = resized_mouth_cutout  # fallback

if resized_mouth_cutout.shape[2] == 3 and roi.shape[2] == 3:
    color_corrected_mouth = apply_color_transfer(resized_mouth_cutout, roi)

# …create feathered mask and blend (see lines 86‑100 in the file) …

blended_roi = (color_corrected_mouth * combined_mask_f32 + roi * inv_mask)
frame[min_y:max_y, min_x:max_x] = blended_roi.astype(np.uint8)

This alpha blending operation preserves the swapped face's details outside the mouth zone while inserting the color-matched lips.

Error Handling and Fallbacks

The entire color transfer workflow is wrapped in try/except blocks to handle OpenCV errors or dimension mismatches gracefully. If apply_color_transfer raises an exception, the pipeline falls back to the uncorrected resized_mouth_cutout, maintaining frame continuity even when color statistics cannot be computed.

Using the Color Transfer Function in Custom Pipelines

Developers can import and apply the color transfer logic independently for custom image processing workflows:

import cv2
import numpy as np
from modules.processors.frame.face_swapper import apply_color_transfer

# source: mouth cut-out (BGR uint8)

# target: swapped face ROI (BGR uint8)

corrected_mouth = apply_color_transfer(source_mouth, target_roi)

# Blend back (simple linear blend)

alpha = 0.6
blended = cv2.addWeighted(corrected_mouth, alpha, target_roi, 1 - alpha, 0)

For integration within the standard face-swap pipeline, the function is invoked automatically during apply_mouth_area processing, contingent on the color_correction toggle defined in modules/globals.py and exposed through modules/ui.py.

Summary

• Deep-Live-Cam performs color transfer in modules/processors/frame/face_swapper.py to match mouth regions with swapped faces.
• The apply_color_transfer function uses LAB color space statistics to align the mouth cut-out's mean and standard deviation with the target ROI.
• Channel validation ensures both source and target have three BGR channels before processing (lines 65–74).
• GPU acceleration handles resizing via gpu_resize, while the statistical transfer occurs on CPU-processed float32 arrays.
• Feathered masking blends the corrected mouth back into the frame, with try/except fallback handling to ensure pipeline stability.

Frequently Asked Questions

How does Deep-Live-Cam prevent color mismatches between the original mouth and swapped face?

Deep-Live-Cam prevents color mismatches by statistically transforming the mouth cut-out's color distribution to match the swapped face ROI. The apply_color_transfer function converts both images to LAB color space, computes per-channel means and standard deviations, and scales the mouth pixels to match the face's lighting statistics before compositing.

What color space does the apply_color_transfer function use?

The function uses the CIELAB (or LAB) color space, which separates luminance (L) from color components (A and B). This separation allows the algorithm to match lighting conditions independently of hue, producing more natural results than RGB-space blending. The conversion uses OpenCV's COLOR_BGR2LAB and COLOR_LAB2BGR constants.

Where is the color transfer logic located in the Deep-Live-Cam repository?

The primary implementation resides in modules/processors/frame/face_swapper.py. The apply_color_transfer helper function spans lines 1147–1262, while the calling logic that validates channels and invokes the transfer appears at lines 65–74. GPU preprocessing utilities are imported from modules/gpu_processing.py.

What happens if the color transfer fails during processing?

If color transfer fails due to OpenCV errors, dimension mismatches, or invalid channel configurations, the pipeline catches the exception and falls back to the uncorrected resized mouth cut-out. This ensures the face-swap continues without crashing, though the color match may be less seamless for that specific frame.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}