Deep-Live-Cam Video Processing Pipeline: Frame Extraction, Processing, and Encoding with keep_fps

The Deep-Live-Cam video processing pipeline decodes input videos into PNG frame sequences, applies AI-driven face swapping through modular processors, and re-encodes the output while optionally preserving the original frame rate via the --keep-fps flag.

Deep-Live-Cam processes source media through a three-stage pipeline managed by modules/core.py. The system extracts raw frames using ffmpeg, processes each frame through configurable AI processors, and re-assembles the final video while respecting the keep_fps configuration to either maintain original timing or default to 30 fps.

Pipeline Architecture Overview

The video processing workflow operates in three distinct stages orchestrated by the start() function in modules/core.py:

1. Frame Extraction: The target video is decoded into individual PNG images stored in a temporary directory
2. Frame-Level Processing: Each extracted frame passes through enabled processors (face swapper, enhancer) that modify the image in-place
3. Video Encoding: Processed frames are re-encoded into the final video format with configurable frame rate handling

The global configuration object in modules/globals.py stores the keep_fps boolean flag (line 25), which determines whether the pipeline preserves the source video's original frame rate or falls back to a default 30 fps during encoding.

Frame Extraction with Hardware Acceleration

The extraction phase is handled by extract_frames() in modules/utilities.py (lines 64-76). This function utilizes ffmpeg with hardware-accelerated decoding when available, converting the input video into a sequence of numbered PNG files:

def extract_frames(target_path: str) -> None:
    """Extract frames with hardware acceleration and optimized settings."""
    temp_directory_path = get_temp_directory_path(target_path)
    run_ffmpeg([
        "-i", target_path,
        "-vf", "format=rgb24",          # fast format conversion

        "-vsync", "0",                  # avoid duplication

        "-frame_pts", "1",              # keep original timestamps

        os.path.join(temp_directory_path, "%04d.png"),
    ])

The function creates a temporary directory via create_temp() and writes frames using the pattern %04d.png to maintain sequential ordering. The -vsync 0 flag prevents frame duplication, while -frame_pts 1 preserves original presentation timestamps for accurate temporal reconstruction.

Frame-Level AI Processing

After extraction, core.start() (lines 35-44) gathers the list of temporary frame paths and iterates through enabled frame processors:

temp_frame_paths = get_temp_frame_paths(modules.globals.target_path)
for frame_processor in get_frame_processors_modules(modules.globals.frame_processors):
    frame_processor.process_video(modules.globals.source_path, temp_frame_paths)

Each processor implements a standardized process_video() interface that receives the source media path and a list of frame file paths. Processors load each PNG frame, apply their specific AI algorithms (such as face detection and swapping), and write the modified image back to the same temporary location. This modular design allows users to chain multiple processors without modifying the core pipeline logic.

FPS Handling and the keep_fps Logic

The --keep-fps command-line argument fundamentally alters how the final video is encoded. The flag is defined in modules/core.py (lines 43-75) within parse_args():

program.add_argument('--keep-fps', help='keep original fps',
                    dest='keep_fps', action='store_true', default=False)

When present, the value propagates to modules.globals.keep_fps, influencing the encoding decision point in core.start() (lines 48-57).

Detecting Original Frame Rates

When keep_fps is enabled, the pipeline calls detect_fps() from modules/utilities.py (lines 42-59) to query the source video's frame rate using ffprobe:

def detect_fps(target_path: str) -> float:
    command = [
        "ffprobe", "-v", "error", "-select_streams", "v:0",
        "-show_entries", "stream=r_frame_rate",
        "-of", "default=noprint_wrappers=1:nokey=1", target_path,
    ]
    output = subprocess.check_output(command).decode().strip().split("/")
    numerator, denominator = map(int, output)
    return numerator / denominator

This function parses the rational frame rate representation (e.g., "30000/1001") and returns the decimal value. If detection fails, the system falls back to 30 fps to ensure pipeline continuity.

Encoding with Preserved or Default FPS

The final encoding occurs in create_video() within modules/utilities.py (lines 80-88). The function accepts an fps parameter that controls the output frame rate via ffmpeg's -r flag:

def create_video(target_path: str, fps: float = 30.0) -> None:
    temp_output_path = get_temp_output_path(target_path)
    temp_directory_path = get_temp_directory_path(target_path)

    ffmpeg_args = [
        "-r", str(fps),                               # <-- forces fps

        "-i", os.path.join(temp_directory_path, "%04d.png"),
        "-c:v", encoder,
        "-pix_fmt", "yuv420p",
        "-movflags", "+faststart",
        "-vf", "colorspace=bt709:iall=bt601-6-625:fast=1",
        "-y", temp_output_path,
    ]
    run_ffmpeg(ffmpeg_args)

In core.start(), the conditional logic determines which fps value reaches the encoder:

if modules.globals.keep_fps:
    fps = detect_fps(modules.globals.target_path)
    create_video(modules.globals.target_path, fps)
else:
    create_video(modules.globals.target_path)   # defaults to 30 fps

When keep_fps is false, the pipeline defaults to 30 fps regardless of the source timing, which may introduce temporal jitter or audio desynchronization in videos with non-standard frame rates.

Audio Restoration and Cleanup Operations

If the --keep-audio flag is set (enabled by default), the pipeline restores the original audio track after video encoding via restore_audio() in core.start() (lines 62-68). When keep_fps is disabled, the system displays a warning about potential audio synchronization issues due to the frame rate mismatch.

Temporary frame files are automatically removed by clean_temp() in modules/utilities.py (lines 54-60) unless the --keep-frames debug flag is specified. This ensures efficient disk space management during batch processing operations.

Practical Usage Examples

To preserve the original frame rate and maintain audio synchronization:

python run.py -s source.png -t input_video.mp4 -o output_video.mp4 --keep-fps

To process with the default 30 fps frame rate while retaining extracted frames for debugging:

python run.py -s source.png -t input_video.mp4 -o output_video.mp4 --keep-frames

Summary

• The video processing pipeline in Deep-Live-Cam consists of three stages: ffmpeg-based frame extraction, modular AI processing, and parameterized video encoding.
• The --keep-fps flag triggers detect_fps() to query the source video's frame rate via ffprobe, ensuring temporal fidelity in the output.
• Frame extraction utilizes hardware-accelerated ffmpeg with RGB24 formatting and timestamp preservation via extract_frames() in modules/utilities.py.
• Encoding logic in create_video() defaults to 30 fps but accepts custom rates through the -r ffmpeg parameter when keep_fps is enabled.
• The pipeline maintains modularity through the process_video() interface, allowing arbitrary frame processors to operate on the extracted PNG sequence before final assembly.

Frequently Asked Questions

How does Deep-Live-Cam handle videos with variable frame rates?

Deep-Live-Cam detects the base frame rate using ffprobe's r_frame_rate stream attribute in modules/utilities.py. While the extraction process preserves timestamps via -frame_pts 1, the encoding stage requires a fixed frame rate value. For variable frame rate sources, the detected rate represents the average or nominal value, which is then enforced consistently throughout the output video via the -r ffmpeg flag.

What happens if I don't use the --keep-fps flag?

Without --keep-fps, the pipeline calls create_video() without specifying a frame rate parameter, causing the function to use its default value of 30.0 fps. This hard-coded default may cause temporal distortion if your source video uses 24 fps, 25 fps, or 60 fps, potentially resulting in audio desynchronization or altered playback speed. The system generates a warning about audio sync issues when --keep-audio is combined with the default 30 fps setting.

Where are the temporary frame files stored during processing?

The get_temp_directory_path() function in modules/utilities.py generates a temporary directory adjacent to the target video file path. Frames are written as %04d.png (four-digit zero-padded numbering) within this folder during the extraction phase. The directory is automatically deleted by clean_temp() after successful encoding unless you specify the --keep-frames flag for debugging purposes.

Can I modify the default 30 fps fallback value?

The default frame rate of 30 fps is hard-coded as the function parameter default in create_video(target_path: str, fps: float = 30.0) within modules/utilities.py. To change this behavior, you would need to modify the default parameter value in the source code or implement additional command-line argument parsing in modules/core.py to expose a custom fps option alongside the existing --keep-fps boolean flag.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}