Frame Candidate Detection vs Frame Selection in the Video Extraction Pipeline

Frame candidate detection discovers all plausible frames using visual analysis engines, while frame selection applies perceptual deduplication and budget-aware sampling to deliver the final curated set.

The bradautomates/claude-video repository implements a strict separation between these two phases in skills/watch/scripts/frames.py. Understanding how frame candidate detection differs from frame selection helps developers optimize extraction workflows and control computational costs when processing video content.

Frame Candidate Detection: The Discovery Phase

Frame candidate detection operates as an uncapped discovery mechanism that scans the entire video (or a specified range) to identify potentially meaningful moments. This phase is agnostic of any frame budget and focuses solely on finding visual landmarks through three distinct engines.

Scene-Change Detection Engine

The primary engine uses ffmpeg's scene detection filter to identify cuts. In extract_scene_candidates() (lines 17-28), the system executes ffmpeg with select='gt(scene,…)' to emit the first frame and every subsequent frame where a scene cut exceeds the SCENE_THRESHOLD.

from pathlib import Path
from skills.watch.scripts import frames

candidates = frames.extract_scene_candidates(
    video_path="sample.mp4",
    out_dir=Path("tmp/frames"),
    resolution=512,
    max_frames=None,          # Uncapped detection

    start_seconds=None,
    end_seconds=None,
    threshold=frames.SCENE_THRESHOLD,
)

Each returned candidate dictionary contains the frame index, timestamp in seconds, file path, and a reason tag set to "scene-change".

Key-Frame Detection Engine

For videos where I-frames provide sufficient coverage, extract_keyframes() (lines 76-85) invokes ffmpeg with the -skip_frame nokey flag. This decodes only keyframes, dramatically reducing processing time while maintaining structural completeness.

candidates, meta = frames.extract_keyframes(
    video_path="sample.mp4",
    out_dir=Path("tmp/keyframes"),
    resolution=512,
    max_frames=50,
)

The function returns metadata indicating the engine type and whether a fallback occurred.

Timestamp-Cue Extraction Engine

When users specify exact moments of interest, extract_at_timestamps() (lines 24-33) grabs single frames at each supplied timestamp. This engine generates candidates with the reason tag "transcript-cue" or similar contextual markers.

Frame Selection: The Filtering Phase

Frame selection transforms the raw candidate list into the final deliverable through budget-constrained filtering. This phase ensures the output respects user-defined limits while eliminating redundancy.

Perceptual Deduplication

The dedupe_perceptual() function (lines 66-73) implements the first filtering stage. It generates thumbnails of the JPEG candidates and computes mean-pixel differences between consecutive frames. Any frame with a difference ≤ DEDUP_THRESHOLD is classified as a near-duplicate and removed from the candidate pool.

deduped_candidates, removed_count = frames.dedupe_perceptual(candidates)

This perceptual hashing approach prevents visually similar frames from wasting the frame budget.

Even Sampling and Budget Enforcement

When the deduplicated list exceeds the target count, _even_sample() (lines 94-101) performs stratified sampling. The helper _even_indices() (lines 84-92) calculates evenly spaced indices while preserving the first and last frames to maintain temporal boundaries.

selected, stats = frames._even_sample(
    deduped_candidates,
    n=30,  # Hard budget limit

)

Surviving frames are renumbered 0…N-1, and discarded JPEG files are physically removed from the output directory.

Complete Pipeline Implementation

The extract_scene_or_uniform() function orchestrates both phases, automatically falling back to uniform sampling if scene detection yields insufficient candidates.

selected, meta = frames.extract_scene_or_uniform(
    video_path="sample.mp4",
    out_dir=Path("tmp/out"),
    fps=2.0,
    target_frames=80,
    resolution=512,
    max_frames=80,
    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

According to the source code in skills/watch/scripts/frames.py, this high-level interface handles the transition from detection to selection transparently, returning metadata that identifies which engine succeeded and final frame counts.

Summary

• Detection phase generates an uncapped list of candidates using scene-change analysis, key-frame extraction, or timestamp cues, operating independently of output constraints.
• Selection phase applies dedupe_perceptual() to remove visual duplicates and _even_sample() to enforce frame budgets while preserving temporal endpoints.
• The pipeline architecture in bradautomates/claude-video ensures flexible, cost-effective extraction across all Agent-Skills hosts by decoupling discovery from curation.

Frequently Asked Questions

What triggers the scene-change detection algorithm?

Scene-change detection activates when calling extract_scene_candidates() with a video path and threshold parameter. The function uses ffmpeg's select='gt(scene,SCENE_THRESHOLD)' filter to compare consecutive frames and emit candidates only when the difference metric exceeds the configured threshold, typically capturing the first frame and every significant visual cut.

How does the deduplication threshold work?

The DEDUP_THRESHOLD constant in skills/watch/scripts/frames.py defines the mean-pixel difference cutoff used by dedupe_perceptual(). When thumbnails of two consecutive candidates differ by less than or equal to this threshold, the latter frame is discarded as perceptually redundant, ensuring the final selection contains only visually distinct content.

Can I use timestamp cues with frame budget limits?

Yes. When using extract_at_timestamps(), the function generates candidates at exact specified times, but these still flow through the standard selection pipeline. If you pass the resulting candidates to _even_sample() with a budget parameter n, the system will deduplicate first, then evenly sample from your timestamp-specific frames to meet the limit.

What happens if candidate detection returns fewer frames than the budget?

If the detection phase yields fewer candidates than the requested max_frames or target_frames value, the selection phase simply returns all available candidates without synthetic interpolation. The extract_scene_or_uniform() wrapper specifically checks for this condition and can fallback to uniform fps-based extraction if scene detection produces an insufficient set.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}