# How ffmpeg Extracts Video Frames in the claude-video Repository

> Discover how ffmpeg extracts video frames in the claude-video repository. Learn about uniform sampling, scene detection, and keyframe selection for efficient image conversion.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: internals
- Published: 2026-07-27

---

**ffmpeg serves as the core decoding and frame extraction engine in the claude-video repository, converting compressed video streams into standardized JPEG images through multiple strategies including uniform sampling, scene detection, and keyframe selection.**

The claude-video project relies on ffmpeg to transform arbitrary video files into AI-ready image datasets. Located in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), the extraction pipeline leverages ffmpeg's comprehensive codec support and filtering capabilities to produce deterministic frame sets without requiring heavy external dependencies.

## ffmpeg as the Core Decoding Engine

At the heart of the `watch` skill, ffmpeg handles the computationally intensive work of parsing video containers and decompressing frames. The repository treats ffmpeg as the universal adapter that normalizes input from any video format into a consistent output of JPEG images.

### Metadata Inspection with ffprobe

Before extraction begins, the code calls `ffprobe` (ffmpeg's companion binary) to inspect container metadata. The `get_metadata()` function in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) (lines 86-99) retrieves duration, resolution, codec information, and audio presence. This data drives downstream decisions about frame sampling rates and processing budgets.

### Frame Selection and Rescaling

ffmpeg executes four distinct extraction strategies depending on content requirements. Each strategy uses filters to minimize output volume while preserving visual information:

- **Uniform sampling** selects frames at regular intervals based on target FPS
- **Scene detection** identifies visual discontinuities using the `select` filter
- **Keyframe extraction** grabs only intra-coded frames for coarse analysis
- **Timestamp targeting** seeks to specific seconds for transcript alignment

The `_scale_filter()` helper (lines 42-46) ensures all output respects a maximum resolution default of 512 pixels while preserving aspect ratio, keeping downstream processing predictable.

## Frame Extraction Strategies in frames.py

The module implements four specialized functions that construct specific ffmpeg command pipelines.

### Uniform Frame Extraction

The `extract()` function (lines 71-102) implements the default fps-based sampling. It builds a command that optionally seeks to a start time (`-ss`), enforces the target framerate, applies scaling, and writes sequentially numbered JPEGs (`frame_0001.jpg`, etc.).

```python

# Simplified representation of the extraction logic

cmd = [
    "ffmpeg",
    "-ss", str(start_time),  # Optional seek

    "-i", video_path,
    "-vf", f"fps={fps},{scale_filter}",
    "-q:v", "4",  # Quality setting for JPEG

    f"{output_dir}/frame_%04d.jpg"
]

```

### Scene-Change Detection

For content-rich analysis, `extract_scene_candidates()` (lines 40-66) employs ffmpeg's scene detection filter. The command uses `select='eq(n\\,0)+gt(scene\\,THRESH)'` to output the first frame plus any frame where visual difference exceeds a configurable threshold.

The filter also emits `showinfo` metadata to stderr, which the Python code parses using `SHOWINFO_TS_RE` (line 39) to capture exact timestamps for each detected scene change.

### Keyframe-Only Extraction

The `extract_keyframes()` function (lines 100-121) provides a fast, coarse extraction tier. By passing `-skip_frame nokey`, ffmpeg outputs only keyframes (intra-coded frames), which roughly correspond to scene cuts. This method sacrifices granularity for speed when processing long videos.

### Timestamp-Driven Extraction

For aligning frames with transcript cue points, `extract_at_timestamps()` (lines 61-76) iterates through specific timestamps. For each cue, it constructs a command seeking to the exact second (`-ss T`) and capturing a single frame (`-frames:v 1`), ensuring precise alignment between visual content and text transcripts.

## Post-Processing and Thumbnail Generation

After ffmpeg writes the primary JPEGs, the pipeline supports deduplication through thumbnail comparison. The `_thumb_frames()` function (lines 42-50) uses ffmpeg to generate tiny thumbnail versions of extracted frames. These low-resolution images enable fast perceptual hashing to remove near-duplicate frames without re-processing full-size images.

## Practical Code Examples

### Extract Uniform Frames via CLI

```bash
python -m skills.watch.scripts.frames /path/to/video.mp4 ./frames_out \
    --fps 1.5 \
    --resolution 256

```

This invokes the `extract()` function with custom framerate and resolution limits.

### Extract Scene-Change Frames Programmatically

```python
from pathlib import Path
from skills.watch.scripts.frames import extract_scene_candidates

candidates = extract_scene_candidates(
    video_path="example.mov",
    out_dir=Path("./scene_frames"),
    resolution=512,
    max_frames=None,
)
print(f"Detected {len(candidates)} scene frames")

```

This returns a list of dictionaries containing JPEG paths, timestamps, and the reason `"scene-change"`.

### Extract Frames at Specific Timestamps

```python
from pathlib import Path
from skills.watch.scripts.frames import extract_at_timestamps

frames, meta = extract_at_timestamps(
    video_path="lecture.mp4",
    out_dir=Path("./cue_frames"),
    timestamps=[12.5, 45.0, 78.3],
    resolution=512,
)

```

Each timestamp triggers a separate ffmpeg invocation with precise seeking.

## Summary

- **ffmpeg handles all decoding** in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), normalizing arbitrary video formats into JPEG outputs.
- **Four extraction strategies** provide flexibility: uniform FPS sampling, scene-change detection, keyframe extraction, and timestamp-specific seeking.
- **ffprobe integration** enables intelligent preprocessing by reading container metadata before frame extraction begins.
- **Rescaling and quality controls** ensure consistent output dimensions (default 512px) and reasonable file sizes through JPEG compression.
- **Metadata parsing** of ffmpeg's `showinfo` output provides accurate timestamps for scene detection and transcript alignment.

## Frequently Asked Questions

### What ffmpeg filters does claude-video use for scene detection?

The repository uses the `select` filter with the expression `eq(n\,0)+gt(scene\,THRESH)` to identify scene changes. This filter compares each frame against the previous one and outputs frames where the difference exceeds a configurable threshold. The `showinfo` filter runs simultaneously to emit timestamp metadata that Python parses to build accurate frame timelines.

### How does the repository handle different video codecs and formats?

ffmpeg acts as the universal decoder, supporting any container or codec that ffmpeg itself recognizes. The Python code does not implement codec-specific logic; instead, it relies on ffmpeg's `-i` input flag to auto-detect formats. This allows the `watch` skill to process MP4, MOV, AVI, MKV, and webm files without code changes.

### Why does the code extract keyframes separately from scene detection?

Keyframe extraction uses `-skip_frame nokey` to grab only intra-coded frames, providing a coarse but computationally cheap sampling method suitable for long videos or initial previews. Scene detection performs full-frame analysis using the `select` filter, which is more accurate but slower. The repository offers both to balance speed against precision depending on the use case.

### How are frame timestamps determined during extraction?

For uniform and keyframe extraction, timestamps derive from the frame index divided by the specified FPS. For scene detection, the code parses stderr output from ffmpeg's `showinfo` filter using the regex pattern `SHOWINFO_TS_RE` defined at line 39, which extracts the exact presentation timestamp (PTS) for each selected frame.