# Why Transcript-Cue Frames Are Prioritized Over Scene-Detected Frames in claude-video

> Discover why claude-video prioritizes transcript-cue frames over scene-detected frames. Learn how pinned frames ensure critical visual evidence matches transcript timestamps for better analysis.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: internals
- Published: 2026-08-05

---

**Transcript-cue frames are prioritized over scene-detected frames in `claude-video` because they are marked as "pinned" and reserved in the frame budget first, ensuring critical visual evidence aligns with every timestamp mentioned in the transcript.**

The `bradautomates/claude-video` repository implements a deliberate two-tier frame extraction system for its **/watch** skill. When processing video content, the system extracts both **detail frames** (scene/keyframe based) and **transcript-cue frames** (timestamp-aligned), but guarantees that cue frames survive any budget constraints that might cull scene-detected alternatives.

## How Frame Prioritization Works in claude-video

The prioritization logic operates through three coordinated steps in the codebase:

### Step 1: Extract Cue Frames With Metadata Tagging

In [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), the `extract_at_timestamps` function generates frames at each transcript timestamp and explicitly tags them:

```python
cue_frames, cue_meta = extract_at_timestamps(
    video_path,
    work / "frames",
    cue_timestamps,
    resolution=args.resolution,
    max_frames=max_frames,
    start_seconds=start_sec,
    end_seconds=end_sec,
)

```

Each frame receives `reason: "transcript-cue"` ([lines 376-381](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py#L376-L381)), marking it for protected status in downstream processing.

### Step 2: Reserve Budget Before Scene Extraction

The orchestration in [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py) enforces prioritization through budget subtraction:

```python
detail_budget = max_frames if max_frames is None else max(0, max_frames - len(cue_frames))

```

This calculation ([lines 195-197](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py#L195-L197)) reduces the available slots for scene detection by the exact count of cue frames, guaranteeing their reservation regardless of how the detail engine operates.

### Step 3: Merge Without Dropping Pinned Frames

The `merge_frames` function ([lines 312-321](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py#L312-L321)) concatenates both frame lists, sorts chronologically, and re-indexes—**but never discards cue frames**. The docstring explicitly states this behavior, with the cap already enforced through budget reservation.

The final merge call appears in [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) ([lines 227-229](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py#L227-L229)):

```python
if cue_frames:
    frames = merge_frames(frames, cue_frames)   # never drops pinned frames

```

## Why This Design Prioritizes Transcript Alignment

**Accuracy over coverage.** The scene detection engine provides visual variety but cannot guarantee relevance to specific transcript moments. By pinning cue frames, `claude-video` ensures that any user question referencing a timestamp has corresponding visual evidence—critical for accurate video understanding.

**Deterministic behavior.** The budget-reservation pattern makes frame counts predictable: `total_frames = cue_frames + min(scene_frames, remaining_budget)`. Users relying on transcript data never encounter missing reference images.

**Performance efficiency.** Cue frame extraction runs once, early in the pipeline. The scene engine then operates within constrained resources, avoiding redundant processing of frames that would later be discarded.

## Key Source Files and Functions

| File | Function | Purpose |
|------|----------|---------|
| [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) | `extract_at_timestamps` | Creates cue frames with metadata tagging |
| [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) | `merge_frames` | Concatenates lists while preserving pinned frames |
| [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py) | Budget calculation (lines 195-197) | Reserves slots for cues before scene extraction |
| [`tests/test_timestamps.py`](https://github.com/bradautomates/claude-video/blob/main/tests/test_timestamps.py) | Frame retention tests | Validates that pinned frames survive merging |

## Summary

- **Transcript-cue frame prioritization** is implemented through three coordinated steps: extraction with metadata tagging, budget reservation, and protected merging.
- **Pinned frame status** ensures cue frames are never dropped, even under strict `max_frames` limits.
- **Source locations** in [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py) and [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) demonstrate intentional design, not incidental behavior.
- **Test coverage** in [`test_timestamps.py`](https://github.com/bradautomates/claude-video/blob/main/test_timestamps.py) provides regression protection for this prioritization logic.

## Frequently Asked Questions

### What happens if there are more transcript cues than max_frames allows?

The `detail_budget` calculation uses `max(0, max_frames - len(cue_frames))`, which bottoms at zero. If cues exceed the limit, scene frames receive zero budget. The current implementation does not truncate cues—this edge case would require explicit handling or user warning, which the source code does not currently implement.

### Can users disable transcript-cue frame prioritization?

No configuration option exists in the analyzed codebase. The prioritization is hardcoded in the orchestration logic. Users seeking scene-only frames would need to modify [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) to bypass the `extract_at_timestamps` call and remove the budget reservation logic.

### How does merge_frames distinguish pinned from regular frames?

The function relies on the temporal separation enforced by budget reservation rather than runtime filtering. Since cues are extracted and reserved first, the merge operation simply concatenates both lists. The docstring documents this as "never dropping pinned frames," with actual enforcement occurring through prior budget allocation.

### Why not use scene detection to find frames near transcript timestamps?

Scene detection optimizes for visual change, not temporal alignment. A transcript timestamp might fall during a static shot the scene engine would skip, or during rapid cuts where the nearest keyframe lacks relevant detail. Explicit extraction guarantees exact temporal correspondence regardless of visual activity.