# How the Keyframe Engine Works for Frame Extraction in Claude-Video

> Discover how the keyframe engine in claude-video efficiently extracts frames using ffmpeg's nokey flag for fast, low-cost sampling. Learn its fallback mechanism.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: internals
- Published: 2026-08-09

---

**The keyframe engine in `claude-video` uses ffmpeg's `-skip_frame nokey` flag to extract only I-frames (keyframes), providing a fast, low-cost sampling method that falls back to uniform extraction if fewer than 4 keyframes are found.**

The **keyframe engine** is the fastest tier for frame extraction in the `bradautomates/claude-video` repository. Located in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), this engine leverages the fact that video encoders insert keyframes (I-frames) at scene changes, allowing near-instant sampling without decoding every picture. By extracting only these distinct moments, the system minimizes computational overhead while preserving visual diversity.

## Core Architecture of the Keyframe Engine

### Fast I-Frame Extraction with ffmpeg

The main entry point is **`extract_keyframes`**, which constructs an ffmpeg command targeting only keyframes. The command includes the `-skip_frame nokey` flag, instructing ffmpeg to **skip all P/B frames** and output only I-frames.

The implementation also applies the `showinfo` video filter (lines 100-118 in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py)), causing ffmpeg to print timestamps for each decoded frame to `stderr`. This output capture enables precise temporal indexing without additional file I/O operations.

### Timestamp Parsing and Candidate Generation

After ffmpeg execution completes, the function parses the captured `stderr` output using the compiled regex pattern `SHOWINFO_TS_RE` (defined at line 39). The parsing loop extracts timestamps and assembles a list of candidate frames, where each entry contains:

- Frame index
- Timestamp in seconds
- File path to the extracted JPEG
- Extraction reason (set to `"keyframe"`)

This candidate list serves as the foundation for subsequent deduplication and sampling stages.

## Fallback and Coverage Guarantees

If the video yields **fewer than `KEYFRAME_MIN` (4) keyframes**, the engine automatically falls back to the **uniform extractor** (`extract`). This threshold ensures that short clips or poorly encoded videos still provide adequate visual coverage.

The fallback sequence (lines 136-161) executes three steps:

1. **Metadata retrieval** – Calls `get_metadata` to determine total video duration
2. **Frame rate calculation** – Uses `auto_fps` (lines 22-38) to compute an appropriate sampling rate that respects the user-provided `max_frames` budget
3. **Uniform extraction** – Runs the uniform extractor, followed by optional perceptual deduplication via `dedupe_perceptual`

This graceful degradation guarantees that callers always receive a useful frame set regardless of video encoding characteristics.

## Post-Processing Pipeline

When sufficient keyframes are present, the engine proceeds through **deduplication** and **capping** stages to refine the candidate set.

### Perceptual Deduplication

The **`dedupe_perceptual`** function (line 64) eliminates near-identical frames that may occur during static scenes. The process generates 16×16 grayscale thumbnails for each candidate using `_thumb_frames`, then calculates the mean-absolute-per-pixel difference between consecutive frames.

Frames with a difference **≤ `DEDUP_THRESHOLD` (2.0)** are considered duplicates. The function deletes the corresponding JPEG files on-the-fly (lines 63-78) and removes them from the candidate list, ensuring only visually distinct frames survive.

### Even Sampling and Frame Capping

Even after deduplication, the candidate count may exceed the user's `max_frames` limit. The **`_even_sample`** helper (lines 83-92) selects *n* evenly-spaced frames while always preserving the first and last frames to maintain temporal boundaries.

This function deletes unused JPEG files and re-indexes the survivors, producing a final list that respects the frame budget without clustering selections in specific video segments.

## Implementation Details and Code Examples

The following Python example demonstrates direct usage of the keyframe engine:

```python
from pathlib import Path
from skills.watch.scripts.frames import extract_keyframes

video = "example.mp4"
out_dir = Path("./frames")
frames, meta = extract_keyframes(
    video_path=video,
    out_dir=out_dir,
    resolution=512,
    max_frames=30,      # cap to 30 frames

    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Engine:", meta["engine"])
print("Selected frames:", len(frames))
print(frames[0])   # first frame dict

```

The engine also exposes a command-line interface:

```bash
python -m skills.watch.scripts.frames \
    example.mp4 ./frames \
    --max-frames 30 \
    --resolution 512

```

Both interfaces return a tuple `(selected_frames, meta)` where `meta` contains:
- `"engine"`: `"keyframe"` (or `"uniform"` if fallback occurred)
- Candidate count, deduplication count, and final selection count
- Boolean flag indicating whether fallback extraction was triggered

## Summary

- The keyframe engine resides in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) and uses **`extract_keyframes`** as its primary entry point.
- It leverages ffmpeg's `-skip_frame nokey` flag to decode only I-frames, parsing timestamps via `SHOWINFO_TS_RE`.
- If fewer than 4 keyframes are detected, the system automatically falls back to uniform extraction using `auto_fps` for frame rate calculation.
- **Perceptual deduplication** compares 16×16 thumbnails and removes frames below a difference threshold of 2.0.
- **`_even_sample`** ensures the final output respects `max_frames` while preserving temporal coverage across the video duration.

## Frequently Asked Questions

### What makes the keyframe engine faster than uniform frame extraction?

The keyframe engine avoids decoding intermediate P-frames and B-frames by passing `-skip_frame nokey` to ffmpeg. Because I-frames represent complete images stored independently in the video stream, ffmpeg can locate and extract them without processing intervening delta frames, reducing CPU usage and I/O operations significantly compared to decoding the entire video.

### How does the engine handle videos with few scene changes?

When a video contains fewer than `KEYFRAME_MIN` (4) keyframes—common in static recordings or slideshows—the engine triggers a fallback to uniform extraction. It calculates an appropriate frames-per-second rate using `auto_fps` to distribute samples evenly across the duration, ensuring the user still receives the requested number of frames up to `max_frames`.

### What is the deduplication threshold and how does it work?

The `DEDUP_THRESHOLD` is set to **2.0**, representing the maximum mean-absolute-per-pixel difference between 16×16 grayscale thumbnails. If two consecutive frames differ by 2.0 or less, they are considered visually identical, and the duplicate is deleted immediately. This threshold balances removal of static frames against preservation of subtle motion.

### Can I disable deduplication when using the keyframe engine?

Yes. Both the Python API and CLI support disabling deduplication by setting `dedup=False` (Python) or omitting the deduplication flag. When disabled, the engine skips the `dedupe_perceptual` call and proceeds directly to even sampling, preserving all extracted keyframes regardless of visual similarity.