# How Claude Video Processes and Formats Extracted Frames: The Complete Pipeline

> Discover how Claude Video processes and formats extracted frames. Learn about JPEGs, scene detection, deduplication, and timestamping to understand the complete pipeline.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: internals
- Published: 2026-07-13

---

**Claude Video converts video files into structured JSON frame descriptors by extracting JPEGs at calculated intervals, detecting scene changes and keyframes, applying perceptual deduplication, and formatting each frame with timestamps and selection reasons.**

The `bradautomates/claude-video` repository provides a sophisticated frame extraction pipeline that transforms raw video into model-ready data. Located primarily in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), this system processes extracted frames through ten distinct stages—from metadata discovery to final JSON formatting—to ensure optimal visual coverage without overwhelming context windows.

## Metadata Discovery and FPS Budgeting

The pipeline begins with `get_metadata()` in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) (lines 86-119), which invokes **ffprobe** to read video duration, resolution, codec, and audio presence. This metadata drives the frame sampling strategy.

The `auto_fps()` function (lines 22-60) calculates a suitable frames-per-second value that respects a maximum frame budget (default 100) while capping FPS at 2. For focused time ranges, `auto_fps_focus()` applies a denser "focus" budget to capture more detail in specific segments.

## Frame Extraction Strategies

Claude Video employs multiple extraction engines depending on video content:

**Uniform Sampling** – The `extract()` function (lines 62-99) builds an **ffmpeg** command using `-vf fps=...,scale...` to generate JPEGs at the calculated FPS. Each frame is rescaled to a configurable resolution (default 512px) while preserving aspect ratio, producing files named `frame_%04d.jpg`.

**Scene-Change Detection** – When the video contains cuts, `extract_scene_candidates()` (lines 170-178) runs ffmpeg with a scene filter (`gt(scene,...)`) and `showinfo` to capture exact timestamps of visual transitions, ensuring major visual shifts are always represented.

**Keyframe Extraction** – The `extract_keyframes()` function (lines 76-82) uses `-skip_frame nokey` to extract only I-frames, falling back to uniform extraction if insufficient keyframes are present.

## Perceptual Deduplication and Sampling

After extraction, `dedupe_perceptual()` eliminates near-identical frames to reduce redundancy. The implementation (lines 62-84) creates 16×16 grayscale thumbnails via `_thumb_frames()`, then drops frames where the mean-pixel difference is ≤ 2.0 through `_dedupe_by_deltas()`.

Following deduplication, `_even_sample()` (lines 93-102) enforces the maximum frame count by keeping the first and last frames and selecting evenly-spaced representatives from the remaining set.

When transcripts are available, `extract_at_timestamps()` generates "cue" frames at speech timestamps, which `merge_frames()` (lines 112-140) combines with the primary frame list while preserving all cue frames and re-indexing the sequence.

## Final Frame Format and JSON Structure

The [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) entry point (lines 39-55) orchestrates the pipeline and outputs a JSON object containing metadata, FPS, target counts, and the ordered list of frame descriptors.

Each extracted frame is represented as a dictionary with these fields:

```json
{
  "index": 0,
  "timestamp_seconds": 0.0,
  "path": "/tmp/frames/frame_0000.jpg",
  "reason": "uniform"
}

```

The `reason` field indicates why the frame was selected: `uniform`, `scene-change`, `keyframe`, `first-frame`, or `transcript-cue`. This standardized structure allows Claude Video to present frames consistently regardless of which extraction engine produced them.

## Implementation Example

You can interact with the frame processing pipeline directly in Python:

```python
from pathlib import Path
from skills.watch.scripts.frames import get_metadata, auto_fps, extract, dedupe_perceptual

video = "sample.mp4"
out_dir = Path("tmp/frames")
meta = get_metadata(video)

# Choose FPS based on clip length (default max 100 frames)

fps, target = auto_fps(meta["duration_seconds"], max_frames=100)

# Extract frames at calculated FPS

frames = extract(video, out_dir, fps=fps, resolution=512, max_frames=target)

# Remove near-duplicate frames

frames, dropped = dedupe_perceptual(frames)

print(f"Extracted {len(frames)} frames, dropped {dropped} duplicates")
print(frames[:3])   # show first three descriptors

```

Command-line usage provides the same functionality:

```bash
python -m skills.watch.scripts.frames \
    sample.mp4 ./tmp/frames \
    --resolution 512 \
    --max-frames 80

```

Both methods produce a JSON-compatible list of frame descriptors ready for model consumption.

## Summary

- Claude Video processes extracted frames through a ten-stage pipeline defined in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) and orchestrated by [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py).
- **Metadata discovery** via ffprobe informs intelligent FPS selection that respects a 100-frame default budget.
- **Multiple extraction strategies** include uniform sampling, scene-change detection, and keyframe extraction to capture relevant visual content.
- **Perceptual deduplication** uses 16×16 grayscale thumbnails and mean-pixel difference thresholds (≤ 2.0) to collapse identical shots.
- **Final output** is a JSON array of frame descriptors containing `index`, `timestamp_seconds`, `path`, and `reason` fields for consistent downstream processing.

## Frequently Asked Questions

### What is the default maximum frame count in Claude Video?

Claude Video defaults to a maximum of **100 frames** per video. The `auto_fps()` function calculates the optimal frames-per-second rate to distribute this budget evenly across the video duration while never exceeding 2 FPS, ensuring manageable token counts for model context windows.

### How does Claude Video detect scene changes?

The `extract_scene_candidates()` function in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) (lines 170-178) uses ffmpeg's scene detection filter (`gt(scene,...)` combined with `showinfo`) to identify exact timestamps where significant visual changes occur. This ensures that abrupt cuts or transitions are always captured in the frame list, regardless of the uniform sampling interval.

### What is perceptual deduplication and why does Claude Video use it?

Perceptual deduplication prevents redundant frames from wasting the frame budget on visually similar content. The `dedupe_perceptual()` function generates 16×16 grayscale thumbnails and calculates mean-pixel differences between consecutive frames; if the difference is ≤ 2.0, the frame is discarded. This efficiently collapses static or slow-changing scenes while preserving visual variance.

### How are frame timestamps calculated?

Timestamps are computed during the extraction phase based on the start offset and selected FPS. The `extract()` function (lines 106-113) generates a list of dictionaries where `timestamp_seconds` represents the exact temporal position of each frame in the video, allowing the model to correlate visual content with transcript or audio cues.