# Frame Candidate Detection vs Frame Selection in the Video Extraction Pipeline

> Understand frame candidate detection vs frame selection in video extraction. Discover how visual analysis and perceptual deduplication create a curated frame set.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: deep-dive
- Published: 2026-08-08

---

**Frame candidate detection discovers all plausible frames using visual analysis engines, while frame selection applies perceptual deduplication and budget-aware sampling to deliver the final curated set.**

The `bradautomates/claude-video` repository implements a strict separation between these two phases in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py). Understanding how frame candidate detection differs from frame selection helps developers optimize extraction workflows and control computational costs when processing video content.

## Frame Candidate Detection: The Discovery Phase

Frame candidate detection operates as an **uncapped discovery mechanism** that scans the entire video (or a specified range) to identify potentially meaningful moments. This phase is agnostic of any frame budget and focuses solely on finding visual landmarks through three distinct engines.

### Scene-Change Detection Engine

The primary engine uses ffmpeg's scene detection filter to identify cuts. In `extract_scene_candidates()` (lines 17-28), the system executes ffmpeg with `select='gt(scene,…)'` to emit the first frame and every subsequent frame where a scene cut exceeds the `SCENE_THRESHOLD`.

```python
from pathlib import Path
from skills.watch.scripts import frames

candidates = frames.extract_scene_candidates(
    video_path="sample.mp4",
    out_dir=Path("tmp/frames"),
    resolution=512,
    max_frames=None,          # Uncapped detection

    start_seconds=None,
    end_seconds=None,
    threshold=frames.SCENE_THRESHOLD,
)

```

Each returned candidate dictionary contains the frame index, timestamp in seconds, file path, and a reason tag set to `"scene-change"`.

### Key-Frame Detection Engine

For videos where I-frames provide sufficient coverage, `extract_keyframes()` (lines 76-85) invokes ffmpeg with the `-skip_frame nokey` flag. This decodes only keyframes, dramatically reducing processing time while maintaining structural completeness.

```python
candidates, meta = frames.extract_keyframes(
    video_path="sample.mp4",
    out_dir=Path("tmp/keyframes"),
    resolution=512,
    max_frames=50,
)

```

The function returns metadata indicating the engine type and whether a fallback occurred.

### Timestamp-Cue Extraction Engine

When users specify exact moments of interest, `extract_at_timestamps()` (lines 24-33) grabs single frames at each supplied timestamp. This engine generates candidates with the reason tag `"transcript-cue"` or similar contextual markers.

## Frame Selection: The Filtering Phase

Frame selection transforms the raw candidate list into the final deliverable through **budget-constrained filtering**. This phase ensures the output respects user-defined limits while eliminating redundancy.

### Perceptual Deduplication

The `dedupe_perceptual()` function (lines 66-73) implements the first filtering stage. It generates thumbnails of the JPEG candidates and computes mean-pixel differences between consecutive frames. Any frame with a difference ≤ `DEDUP_THRESHOLD` is classified as a near-duplicate and removed from the candidate pool.

```python
deduped_candidates, removed_count = frames.dedupe_perceptual(candidates)

```

This perceptual hashing approach prevents visually similar frames from wasting the frame budget.

### Even Sampling and Budget Enforcement

When the deduplicated list exceeds the target count, `_even_sample()` (lines 94-101) performs stratified sampling. The helper `_even_indices()` (lines 84-92) calculates evenly spaced indices while **preserving the first and last frames** to maintain temporal boundaries.

```python
selected, stats = frames._even_sample(
    deduped_candidates,
    n=30,  # Hard budget limit

)

```

Surviving frames are renumbered 0…N-1, and discarded JPEG files are physically removed from the output directory.

## Complete Pipeline Implementation

The `extract_scene_or_uniform()` function orchestrates both phases, automatically falling back to uniform sampling if scene detection yields insufficient candidates.

```python
selected, meta = frames.extract_scene_or_uniform(
    video_path="sample.mp4",
    out_dir=Path("tmp/out"),
    fps=2.0,
    target_frames=80,
    resolution=512,
    max_frames=80,
    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

```

According to the source code in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), this high-level interface handles the transition from detection to selection transparently, returning metadata that identifies which engine succeeded and final frame counts.

## Summary

- **Detection phase** generates an uncapped list of candidates using scene-change analysis, key-frame extraction, or timestamp cues, operating independently of output constraints.
- **Selection phase** applies `dedupe_perceptual()` to remove visual duplicates and `_even_sample()` to enforce frame budgets while preserving temporal endpoints.
- The pipeline architecture in `bradautomates/claude-video` ensures flexible, cost-effective extraction across all Agent-Skills hosts by decoupling discovery from curation.

## Frequently Asked Questions

### What triggers the scene-change detection algorithm?

Scene-change detection activates when calling `extract_scene_candidates()` with a video path and threshold parameter. The function uses ffmpeg's `select='gt(scene,SCENE_THRESHOLD)'` filter to compare consecutive frames and emit candidates only when the difference metric exceeds the configured threshold, typically capturing the first frame and every significant visual cut.

### How does the deduplication threshold work?

The `DEDUP_THRESHOLD` constant in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) defines the mean-pixel difference cutoff used by `dedupe_perceptual()`. When thumbnails of two consecutive candidates differ by less than or equal to this threshold, the latter frame is discarded as perceptually redundant, ensuring the final selection contains only visually distinct content.

### Can I use timestamp cues with frame budget limits?

Yes. When using `extract_at_timestamps()`, the function generates candidates at exact specified times, but these still flow through the standard selection pipeline. If you pass the resulting candidates to `_even_sample()` with a budget parameter `n`, the system will deduplicate first, then evenly sample from your timestamp-specific frames to meet the limit.

### What happens if candidate detection returns fewer frames than the budget?

If the detection phase yields fewer candidates than the requested `max_frames` or `target_frames` value, the selection phase simply returns all available candidates without synthetic interpolation. The `extract_scene_or_uniform()` wrapper specifically checks for this condition and can fallback to uniform fps-based extraction if scene detection produces an insufficient set.