Why Transcript-Cue Frames Are Prioritized Over Scene-Detected Frames in claude-video

Transcript-cue frames are prioritized over scene-detected frames in claude-video because they are marked as "pinned" and reserved in the frame budget first, ensuring critical visual evidence aligns with every timestamp mentioned in the transcript.

The bradautomates/claude-video repository implements a deliberate two-tier frame extraction system for its /watch skill. When processing video content, the system extracts both detail frames (scene/keyframe based) and transcript-cue frames (timestamp-aligned), but guarantees that cue frames survive any budget constraints that might cull scene-detected alternatives.

How Frame Prioritization Works in claude-video

The prioritization logic operates through three coordinated steps in the codebase:

Step 1: Extract Cue Frames With Metadata Tagging

In skills/watch/scripts/frames.py, the extract_at_timestamps function generates frames at each transcript timestamp and explicitly tags them:

cue_frames, cue_meta = extract_at_timestamps(
    video_path,
    work / "frames",
    cue_timestamps,
    resolution=args.resolution,
    max_frames=max_frames,
    start_seconds=start_sec,
    end_seconds=end_sec,
)

Each frame receives reason: "transcript-cue" (lines 376-381), marking it for protected status in downstream processing.

Step 2: Reserve Budget Before Scene Extraction

The orchestration in skills/watch/scripts/watch.py enforces prioritization through budget subtraction:

detail_budget = max_frames if max_frames is None else max(0, max_frames - len(cue_frames))

This calculation (lines 195-197) reduces the available slots for scene detection by the exact count of cue frames, guaranteeing their reservation regardless of how the detail engine operates.

Step 3: Merge Without Dropping Pinned Frames

The merge_frames function (lines 312-321) concatenates both frame lists, sorts chronologically, and re-indexes—but never discards cue frames. The docstring explicitly states this behavior, with the cap already enforced through budget reservation.

The final merge call appears in watch.py (lines 227-229):

if cue_frames:
    frames = merge_frames(frames, cue_frames)   # never drops pinned frames

Why This Design Prioritizes Transcript Alignment

Accuracy over coverage. The scene detection engine provides visual variety but cannot guarantee relevance to specific transcript moments. By pinning cue frames, claude-video ensures that any user question referencing a timestamp has corresponding visual evidence—critical for accurate video understanding.

Deterministic behavior. The budget-reservation pattern makes frame counts predictable: total_frames = cue_frames + min(scene_frames, remaining_budget). Users relying on transcript data never encounter missing reference images.

Performance efficiency. Cue frame extraction runs once, early in the pipeline. The scene engine then operates within constrained resources, avoiding redundant processing of frames that would later be discarded.

Key Source Files and Functions

File	Function	Purpose
skills/watch/scripts/frames.py	extract_at_timestamps	Creates cue frames with metadata tagging
skills/watch/scripts/frames.py	merge_frames	Concatenates lists while preserving pinned frames
skills/watch/scripts/watch.py	Budget calculation (lines 195-197)	Reserves slots for cues before scene extraction
tests/test_timestamps.py	Frame retention tests	Validates that pinned frames survive merging

Summary

• Transcript-cue frame prioritization is implemented through three coordinated steps: extraction with metadata tagging, budget reservation, and protected merging.
• Pinned frame status ensures cue frames are never dropped, even under strict max_frames limits.
• Source locations in frames.py and watch.py demonstrate intentional design, not incidental behavior.
• Test coverage in test_timestamps.py provides regression protection for this prioritization logic.

Frequently Asked Questions

What happens if there are more transcript cues than max_frames allows?

The detail_budget calculation uses max(0, max_frames - len(cue_frames)), which bottoms at zero. If cues exceed the limit, scene frames receive zero budget. The current implementation does not truncate cues—this edge case would require explicit handling or user warning, which the source code does not currently implement.

Can users disable transcript-cue frame prioritization?

No configuration option exists in the analyzed codebase. The prioritization is hardcoded in the orchestration logic. Users seeking scene-only frames would need to modify watch.py to bypass the extract_at_timestamps call and remove the budget reservation logic.

How does merge_frames distinguish pinned from regular frames?

The function relies on the temporal separation enforced by budget reservation rather than runtime filtering. Since cues are extracted and reserved first, the merge operation simply concatenates both lists. The docstring documents this as "never dropping pinned frames," with actual enforcement occurring through prior budget allocation.

Why not use scene detection to find frames near transcript timestamps?

Scene detection optimizes for visual change, not temporal alignment. A transcript timestamp might fall during a static shot the scene engine would skip, or during rapid cuts where the nearest keyframe lacks relevant detail. Explicit extraction guarantees exact temporal correspondence regardless of visual activity.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}