How ffmpeg Extracts Video Frames in the claude-video Repository

ffmpeg serves as the core decoding and frame extraction engine in the claude-video repository, converting compressed video streams into standardized JPEG images through multiple strategies including uniform sampling, scene detection, and keyframe selection.

The claude-video project relies on ffmpeg to transform arbitrary video files into AI-ready image datasets. Located in skills/watch/scripts/frames.py, the extraction pipeline leverages ffmpeg's comprehensive codec support and filtering capabilities to produce deterministic frame sets without requiring heavy external dependencies.

ffmpeg as the Core Decoding Engine

At the heart of the watch skill, ffmpeg handles the computationally intensive work of parsing video containers and decompressing frames. The repository treats ffmpeg as the universal adapter that normalizes input from any video format into a consistent output of JPEG images.

Metadata Inspection with ffprobe

Before extraction begins, the code calls ffprobe (ffmpeg's companion binary) to inspect container metadata. The get_metadata() function in skills/watch/scripts/frames.py (lines 86-99) retrieves duration, resolution, codec information, and audio presence. This data drives downstream decisions about frame sampling rates and processing budgets.

Frame Selection and Rescaling

ffmpeg executes four distinct extraction strategies depending on content requirements. Each strategy uses filters to minimize output volume while preserving visual information:

• Uniform sampling selects frames at regular intervals based on target FPS
• Scene detection identifies visual discontinuities using the select filter
• Keyframe extraction grabs only intra-coded frames for coarse analysis
• Timestamp targeting seeks to specific seconds for transcript alignment

The _scale_filter() helper (lines 42-46) ensures all output respects a maximum resolution default of 512 pixels while preserving aspect ratio, keeping downstream processing predictable.

Frame Extraction Strategies in frames.py

The module implements four specialized functions that construct specific ffmpeg command pipelines.

Uniform Frame Extraction

The extract() function (lines 71-102) implements the default fps-based sampling. It builds a command that optionally seeks to a start time (-ss), enforces the target framerate, applies scaling, and writes sequentially numbered JPEGs (frame_0001.jpg, etc.).

# Simplified representation of the extraction logic

cmd = [
    "ffmpeg",
    "-ss", str(start_time),  # Optional seek

    "-i", video_path,
    "-vf", f"fps={fps},{scale_filter}",
    "-q:v", "4",  # Quality setting for JPEG

    f"{output_dir}/frame_%04d.jpg"
]

Scene-Change Detection

For content-rich analysis, extract_scene_candidates() (lines 40-66) employs ffmpeg's scene detection filter. The command uses select='eq(n\\,0)+gt(scene\\,THRESH)' to output the first frame plus any frame where visual difference exceeds a configurable threshold.

The filter also emits showinfo metadata to stderr, which the Python code parses using SHOWINFO_TS_RE (line 39) to capture exact timestamps for each detected scene change.

Keyframe-Only Extraction

The extract_keyframes() function (lines 100-121) provides a fast, coarse extraction tier. By passing -skip_frame nokey, ffmpeg outputs only keyframes (intra-coded frames), which roughly correspond to scene cuts. This method sacrifices granularity for speed when processing long videos.

Timestamp-Driven Extraction

For aligning frames with transcript cue points, extract_at_timestamps() (lines 61-76) iterates through specific timestamps. For each cue, it constructs a command seeking to the exact second (-ss T) and capturing a single frame (-frames:v 1), ensuring precise alignment between visual content and text transcripts.

Post-Processing and Thumbnail Generation

After ffmpeg writes the primary JPEGs, the pipeline supports deduplication through thumbnail comparison. The _thumb_frames() function (lines 42-50) uses ffmpeg to generate tiny thumbnail versions of extracted frames. These low-resolution images enable fast perceptual hashing to remove near-duplicate frames without re-processing full-size images.

Practical Code Examples

Extract Uniform Frames via CLI

python -m skills.watch.scripts.frames /path/to/video.mp4 ./frames_out \
    --fps 1.5 \
    --resolution 256

This invokes the extract() function with custom framerate and resolution limits.

Extract Scene-Change Frames Programmatically

from pathlib import Path
from skills.watch.scripts.frames import extract_scene_candidates

candidates = extract_scene_candidates(
    video_path="example.mov",
    out_dir=Path("./scene_frames"),
    resolution=512,
    max_frames=None,
)
print(f"Detected {len(candidates)} scene frames")

This returns a list of dictionaries containing JPEG paths, timestamps, and the reason "scene-change".

Extract Frames at Specific Timestamps

from pathlib import Path
from skills.watch.scripts.frames import extract_at_timestamps

frames, meta = extract_at_timestamps(
    video_path="lecture.mp4",
    out_dir=Path("./cue_frames"),
    timestamps=[12.5, 45.0, 78.3],
    resolution=512,
)

Each timestamp triggers a separate ffmpeg invocation with precise seeking.

Summary

• ffmpeg handles all decoding in skills/watch/scripts/frames.py, normalizing arbitrary video formats into JPEG outputs.
• Four extraction strategies provide flexibility: uniform FPS sampling, scene-change detection, keyframe extraction, and timestamp-specific seeking.
• ffprobe integration enables intelligent preprocessing by reading container metadata before frame extraction begins.
• Rescaling and quality controls ensure consistent output dimensions (default 512px) and reasonable file sizes through JPEG compression.
• Metadata parsing of ffmpeg's showinfo output provides accurate timestamps for scene detection and transcript alignment.

Frequently Asked Questions

What ffmpeg filters does claude-video use for scene detection?

The repository uses the select filter with the expression eq(n\,0)+gt(scene\,THRESH) to identify scene changes. This filter compares each frame against the previous one and outputs frames where the difference exceeds a configurable threshold. The showinfo filter runs simultaneously to emit timestamp metadata that Python parses to build accurate frame timelines.

How does the repository handle different video codecs and formats?

ffmpeg acts as the universal decoder, supporting any container or codec that ffmpeg itself recognizes. The Python code does not implement codec-specific logic; instead, it relies on ffmpeg's -i input flag to auto-detect formats. This allows the watch skill to process MP4, MOV, AVI, MKV, and webm files without code changes.

Why does the code extract keyframes separately from scene detection?

Keyframe extraction uses -skip_frame nokey to grab only intra-coded frames, providing a coarse but computationally cheap sampling method suitable for long videos or initial previews. Scene detection performs full-frame analysis using the select filter, which is more accurate but slower. The repository offers both to balance speed against precision depending on the use case.

How are frame timestamps determined during extraction?

For uniform and keyframe extraction, timestamps derive from the frame index divided by the specified FPS. For scene detection, the code parses stderr output from ffmpeg's showinfo filter using the regex pattern SHOWINFO_TS_RE defined at line 39, which extracts the exact presentation timestamp (PTS) for each selected frame.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}