# How Claude Video Combines Cue Frames with Detail Frames Using `merge_frames`

> Learn how Claude Video's merge_frames function combines cue and detail frames by sorting and re-indexing for unified frame analysis.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: deep-dive
- Published: 2026-08-02

---

**The `merge_frames` function in Claude Video concatenates detail frames and cue frames, sorts them chronologically by `timestamp_seconds`, and re-indexes the result to produce a unified frame sequence for analysis.**

Claude Video, an open-source video analysis tool by bradautomates, extracts two distinct frame collections to build comprehensive video understanding: **detail frames** from visual analysis engines and **cue frames** from transcript timestamps. The `merge_frames` function in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) unifies these collections into a single chronological sequence that powers the final report.

## What `merge_frames` Does: Step by Step

### Frame Collection Overview

Claude Video generates frames from two sources before merging:

| Frame Type | Source | Purpose |
|------------|--------|---------|
| **Detail frames** | Key-frame, scene-change, or uniform extraction engines | Provide broad visual coverage across the video |
| **Cue frames** | Transcript timestamp analysis (captions or Whisper-generated) | Ensure moments mentioned in speech are visually captured |

The `merge_frames` function receives these as two separate Python lists and produces one consolidated, time-ordered result.

### The Core Algorithm

Located at lines 312-322 of [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), `merge_frames` performs three operations:

```python
def merge_frames(primary: list[dict], pinned: list[dict]) -> list[dict]:
    """Combine two frame lists into one chronological list and reindex 0..n-1.

    *`pinned` frames (transcript cues) are never dropped — this is a plain union,
    so the cap is enforced upstream by reserving budget for the cues.*
    """
    merged = sorted([*primary, *pinned], key=lambda f: f["timestamp_seconds"])
    for i, frame in enumerate(merged):
        frame["index"] = i
    return merged

```

The function executes these steps:

1. **Union** — Concatenates `primary` (detail frames) and `pinned` (cue frames) without removing any entries
2. **Chronological sort** — Orders the combined list by `timestamp_seconds` to match video timeline
3. **Re-indexing** — Assigns sequential indices (0, 1, 2...) to reflect new positions

## Why Cue Frames Are Preserved

The docstring explicitly states: **`pinned` frames are never dropped**. This design guarantees transcript-driven moments remain visible regardless of visual extraction results.

Frame budget management happens **before** the merge. The caller reserves slots for cue frames, ensuring the union operation won't exceed limits. This separation of concerns keeps `merge_frames` simple and predictable.

## Where `merge_frames` Gets Called

The merge step integrates into the main watch workflow at lines 228-229 of [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py):

```python
if cue_frames:
    frames = merge_frames(frames, cue_frames)   # merge detail & cue frames

```

This conditional ensures cue frames only merge when transcript analysis produces them.

## Practical Code Example

Here's how `merge_frames` behaves with sample data:

```python
from skills.watch.scripts.frames import merge_frames

# Detail frames from scene-change detection

detail_frames = [
    {"index": 0, "timestamp_seconds": 3.0, "path": "frame_0000.jpg", "reason": "scene-change"},
    {"index": 1, "timestamp_seconds": 7.5, "path": "frame_0001.jpg", "reason": "scene-change"},
]

# Cue frames from transcript timestamps

cue_frames = [
    {"index": 0, "timestamp_seconds": 5.0, "path": "cue_0000.jpg", "reason": "transcript-cue"},
    {"index": 1, "timestamp_seconds": 12.0, "path": "cue_0001.jpg", "reason": "transcript-cue"},
]

merged = merge_frames(detail_frames, cue_frames)

# Result: chronologically sorted and re-indexed

# [

#   {'index': 0, 'timestamp_seconds': 3.0, 'path': 'frame_0000.jpg', ...},

#   {'index': 1, 'timestamp_seconds': 5.0, 'path': 'cue_0000.jpg', ...},

#   {'index': 2, 'timestamp_seconds': 7.5, 'path': 'frame_0001.jpg', ...},

#   {'index': 3, 'timestamp_seconds': 12.0, 'path': 'cue_0001.jpg', ...},

# ]

```

Note how the 5-second cue frame inserts between the 3-second and 7.5-second detail frames, with indices reset to sequential order.

## Key Files in the Merge Pipeline

| File | Purpose | Location |
|------|---------|----------|
| [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) | `merge_frames` implementation and extraction utilities | [GitHub](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py#L312-L322) |
| [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py) | Orchestrates workflow and invokes `merge_frames` | [GitHub](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py#L228-L229) |
| [`tests/test_timestamps.py`](https://github.com/bradautomates/claude-video/blob/main/tests/test_timestamps.py) | Unit tests for sorting, re-indexing, and cue preservation | [GitHub](https://github.com/bradautomates/claude-video/blob/main/tests/test_timestamps.py) |

## Summary

- **`merge_frames`** performs a plain union of detail and cue frames without dropping entries
- **Chronological sorting** by `timestamp_seconds` ensures the final sequence matches video playback order
- **Re-indexing** provides clean sequential indices for downstream processing
- **Cue preservation** is guaranteed because budget enforcement happens upstream
- **Clean separation** between budget management and merging keeps the code maintainable

## Frequently Asked Questions

### Does `merge_frames` remove duplicate frames at the same timestamp?

No. `merge_frames` performs a plain union with no deduplication. If a detail frame and cue frame share identical `timestamp_seconds` values, both appear in output with their relative order preserved from the sort stability. The function prioritizes simplicity and predictability over duplicate detection.

### How does Claude Video prevent exceeding frame limits when merging?

Frame budget enforcement occurs **before** calling `merge_frames`. The workflow reserves dedicated slots for cue frames when generating detail frames, ensuring the eventual union stays within limits. This design lets `merge_frames` remain a pure merge operation without side effects.

### Can `merge_frames` handle empty cue frame lists?

Yes. The calling code in [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) only invokes `merge_frames` when `cue_frames` exists and is non-empty. If no transcript cues exist, the detail frames pass through unmodified to downstream processing.

### What data structure does each frame dictionary require?

Each frame must contain at minimum a `timestamp_seconds` numeric field for sorting and an `index` field for re-indexing. Typical frames also include `path` (file location) and `reason` (extraction source) fields, though `merge_frames` only accesses `timestamp_seconds` and `index` directly.