How Claude Video's Efficient Mode Works for Frame Extraction

Claude Video's efficient mode extracts only keyframes from video files, capping the total at 50 frames and optionally deduplicating visually similar images to minimize token usage while preserving essential visual context.

The bradautomates/claude-video repository provides a Python-based video processing pipeline that optimizes frame extraction for AI vision models. When processing long videos, the efficient mode offers a high-speed, low-token alternative to comprehensive scene analysis by strategically selecting only the most critical frames. This approach significantly reduces processing time and API costs while maintaining sufficient visual information for most video understanding tasks.

Selecting the Keyframe Engine in watch.py

The efficient mode selection logic resides in watch.py at lines 198-204. When the detail parameter is set to "efficient", the system bypasses the computationally expensive scene-aware analysis in favor of a keyframe-only extraction strategy.

engine_label = "keyframes" if detail == "efficient" else "scene-aware frames"

This conditional assignment determines which processing pipeline activates. The keyframes engine prioritizes speed over comprehensive scene detection, making it ideal for longer videos where token budgets must be strictly managed.

Calculating the Frame Budget in config.py

Before extraction begins, the system calculates a strict frame budget to prevent excessive token usage. In config.py at lines 66-67, the frame_cap() function returns a hard limit based on the selected detail level:

if detail == "efficient":  # in config.frame_cap()

    return 50

For efficient mode, this 50-frame cap represents the maximum number of images that will be extracted and sent to the vision model. The code then calls auto_fps_focus (or auto_fps for full-video scans) to determine an appropriate sampling rate that respects this budget while distributing frames across the video duration.

Extracting Keyframes with FFmpeg in frames.py

The heavy lifting occurs in frames.py within the extract_keyframes function (lines 576-610). This implementation leverages ffmpeg with the -skip_frame nokey flag to extract only intra-coded frames—compression points where the video stores complete image data rather than motion deltas.

def extract_keyframes(...):
    # ffmpeg command selects keyframes only and respects the cap

This method ensures that extracted frames represent significant visual changes or camera cuts without requiring computationally expensive frame-by-frame analysis. The extraction stops immediately once the max_frames budget (50 for efficient mode) is reached, ensuring predictable processing times regardless of video length.

Optional Perceptual Deduplication

By default, the extracted keyframes undergo a perceptual hash deduplication step via the dedupe_perceptual function. This process identifies and removes visually identical or near-identical frames that may occur during static scenes or slow camera movements.

This deduplication step occurs after initial extraction but before final processing, ensuring that the 50-frame budget contains only visually distinct information. The result is a highly compressed representation that eliminates redundancy while preserving scene diversity.

Handling Transcript Cue Frames

When working with video transcripts, the system extracts cue frames at transcript timestamps separately from the keyframe budget. These cue frames are merged with the keyframe collection after extraction.

Critically, the merge operation does not consume the 50-frame budget. The system counts cue frames first, then fills the remaining budget with keyframes extracted from the intervening video segments. This prioritization ensures that spoken content aligns with relevant visual context without sacrificing the efficiency benefits of the keyframe-only approach.

Summary

• Engine Selection: Efficient mode selects the keyframes engine in watch.py (lines 198-204), bypassing scene-aware analysis for speed.
• Frame Cap: A hard limit of 50 frames is enforced via config.py (lines 66-67) to control token costs.
• FFmpeg Pipeline: The extract_keyframes function in frames.py (lines 576-610) uses -skip_frame nokey to extract only compression keyframes.
• Deduplication: The dedupe_perceptual step removes visually redundant frames by default, maximizing information density.
• Cue Priority: Transcript timestamp frames are extracted and merged first, with the remaining budget filled by keyframes.

Frequently Asked Questions

What is the maximum number of frames extracted in efficient mode?

Claude Video's efficient mode extracts a maximum of 50 frames per video. This cap is defined in config.py and enforced throughout the extraction pipeline to ensure minimal token usage when processing video content with AI models.

How does efficient mode differ from scene-aware extraction?

Efficient mode uses the keyframes engine which extracts only ffmpeg keyframes (intra-coded frames), while scene-aware mode analyzes visual content to detect actual scene changes. The keyframe approach is significantly faster and uses fewer computational resources, though it may miss subtle scene transitions that occur within keyframe intervals.

Does efficient mode remove duplicate frames automatically?

Yes, by default efficient mode runs extracted frames through a perceptual hash deduplication process (dedupe_perceptual) that identifies and removes visually identical or near-identical images. This ensures the 50-frame budget contains only distinct visual information rather than redundant static shots.

Can transcript timestamps affect the frame budget in efficient mode?

Transcript cue frames are extracted separately from the keyframe budget. The system extracts cue frames at transcript timestamps first, then merges them with the keyframe collection. The 50-frame budget applies only to the keyframe extraction, meaning videos with transcripts may result in slightly more total frames, with cue frames prioritized for inclusion.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}