How Claude Video Processes and Formats Extracted Frames: The Complete Pipeline

Claude Video converts video files into structured JSON frame descriptors by extracting JPEGs at calculated intervals, detecting scene changes and keyframes, applying perceptual deduplication, and formatting each frame with timestamps and selection reasons.

The bradautomates/claude-video repository provides a sophisticated frame extraction pipeline that transforms raw video into model-ready data. Located primarily in skills/watch/scripts/frames.py, this system processes extracted frames through ten distinct stages—from metadata discovery to final JSON formatting—to ensure optimal visual coverage without overwhelming context windows.

Metadata Discovery and FPS Budgeting

The pipeline begins with get_metadata() in skills/watch/scripts/frames.py (lines 86-119), which invokes ffprobe to read video duration, resolution, codec, and audio presence. This metadata drives the frame sampling strategy.

The auto_fps() function (lines 22-60) calculates a suitable frames-per-second value that respects a maximum frame budget (default 100) while capping FPS at 2. For focused time ranges, auto_fps_focus() applies a denser "focus" budget to capture more detail in specific segments.

Frame Extraction Strategies

Claude Video employs multiple extraction engines depending on video content:

Uniform Sampling – The extract() function (lines 62-99) builds an ffmpeg command using -vf fps=...,scale... to generate JPEGs at the calculated FPS. Each frame is rescaled to a configurable resolution (default 512px) while preserving aspect ratio, producing files named frame_%04d.jpg.

Scene-Change Detection – When the video contains cuts, extract_scene_candidates() (lines 170-178) runs ffmpeg with a scene filter (gt(scene,...)) and showinfo to capture exact timestamps of visual transitions, ensuring major visual shifts are always represented.

Keyframe Extraction – The extract_keyframes() function (lines 76-82) uses -skip_frame nokey to extract only I-frames, falling back to uniform extraction if insufficient keyframes are present.

Perceptual Deduplication and Sampling

After extraction, dedupe_perceptual() eliminates near-identical frames to reduce redundancy. The implementation (lines 62-84) creates 16×16 grayscale thumbnails via _thumb_frames(), then drops frames where the mean-pixel difference is ≤ 2.0 through _dedupe_by_deltas().

Following deduplication, _even_sample() (lines 93-102) enforces the maximum frame count by keeping the first and last frames and selecting evenly-spaced representatives from the remaining set.

When transcripts are available, extract_at_timestamps() generates "cue" frames at speech timestamps, which merge_frames() (lines 112-140) combines with the primary frame list while preserving all cue frames and re-indexing the sequence.

Final Frame Format and JSON Structure

The watch.py entry point (lines 39-55) orchestrates the pipeline and outputs a JSON object containing metadata, FPS, target counts, and the ordered list of frame descriptors.

Each extracted frame is represented as a dictionary with these fields:

{
  "index": 0,
  "timestamp_seconds": 0.0,
  "path": "/tmp/frames/frame_0000.jpg",
  "reason": "uniform"
}

The reason field indicates why the frame was selected: uniform, scene-change, keyframe, first-frame, or transcript-cue. This standardized structure allows Claude Video to present frames consistently regardless of which extraction engine produced them.

Implementation Example

You can interact with the frame processing pipeline directly in Python:

from pathlib import Path
from skills.watch.scripts.frames import get_metadata, auto_fps, extract, dedupe_perceptual

video = "sample.mp4"
out_dir = Path("tmp/frames")
meta = get_metadata(video)

# Choose FPS based on clip length (default max 100 frames)

fps, target = auto_fps(meta["duration_seconds"], max_frames=100)

# Extract frames at calculated FPS

frames = extract(video, out_dir, fps=fps, resolution=512, max_frames=target)

# Remove near-duplicate frames

frames, dropped = dedupe_perceptual(frames)

print(f"Extracted {len(frames)} frames, dropped {dropped} duplicates")
print(frames[:3])   # show first three descriptors

Command-line usage provides the same functionality:

python -m skills.watch.scripts.frames \
    sample.mp4 ./tmp/frames \
    --resolution 512 \
    --max-frames 80

Both methods produce a JSON-compatible list of frame descriptors ready for model consumption.

Summary

• Claude Video processes extracted frames through a ten-stage pipeline defined in skills/watch/scripts/frames.py and orchestrated by watch.py.
• Metadata discovery via ffprobe informs intelligent FPS selection that respects a 100-frame default budget.
• Multiple extraction strategies include uniform sampling, scene-change detection, and keyframe extraction to capture relevant visual content.
• Perceptual deduplication uses 16×16 grayscale thumbnails and mean-pixel difference thresholds (≤ 2.0) to collapse identical shots.
• Final output is a JSON array of frame descriptors containing index, timestamp_seconds, path, and reason fields for consistent downstream processing.

Frequently Asked Questions

What is the default maximum frame count in Claude Video?

Claude Video defaults to a maximum of 100 frames per video. The auto_fps() function calculates the optimal frames-per-second rate to distribute this budget evenly across the video duration while never exceeding 2 FPS, ensuring manageable token counts for model context windows.

How does Claude Video detect scene changes?

The extract_scene_candidates() function in skills/watch/scripts/frames.py (lines 170-178) uses ffmpeg's scene detection filter (gt(scene,...) combined with showinfo) to identify exact timestamps where significant visual changes occur. This ensures that abrupt cuts or transitions are always captured in the frame list, regardless of the uniform sampling interval.

What is perceptual deduplication and why does Claude Video use it?

Perceptual deduplication prevents redundant frames from wasting the frame budget on visually similar content. The dedupe_perceptual() function generates 16×16 grayscale thumbnails and calculates mean-pixel differences between consecutive frames; if the difference is ≤ 2.0, the frame is discarded. This efficiently collapses static or slow-changing scenes while preserving visual variance.

How are frame timestamps calculated?

Timestamps are computed during the extraction phase based on the start offset and selected FPS. The extract() function (lines 106-113) generates a list of dictionaries where timestamp_seconds represents the exact temporal position of each frame in the video, allowing the model to correlate visual content with transcript or audio cues.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}