How Claude Video Combines Cue Frames with Detail Frames Using `merge_frames`

The merge_frames function in Claude Video concatenates detail frames and cue frames, sorts them chronologically by timestamp_seconds, and re-indexes the result to produce a unified frame sequence for analysis.

Claude Video, an open-source video analysis tool by bradautomates, extracts two distinct frame collections to build comprehensive video understanding: detail frames from visual analysis engines and cue frames from transcript timestamps. The merge_frames function in skills/watch/scripts/frames.py unifies these collections into a single chronological sequence that powers the final report.

What merge_frames Does: Step by Step

Frame Collection Overview

Claude Video generates frames from two sources before merging:

Frame Type	Source	Purpose
Detail frames	Key-frame, scene-change, or uniform extraction engines	Provide broad visual coverage across the video
Cue frames	Transcript timestamp analysis (captions or Whisper-generated)	Ensure moments mentioned in speech are visually captured

The merge_frames function receives these as two separate Python lists and produces one consolidated, time-ordered result.

The Core Algorithm

Located at lines 312-322 of skills/watch/scripts/frames.py, merge_frames performs three operations:

def merge_frames(primary: list[dict], pinned: list[dict]) -> list[dict]:
    """Combine two frame lists into one chronological list and reindex 0..n-1.

    *`pinned` frames (transcript cues) are never dropped — this is a plain union,
    so the cap is enforced upstream by reserving budget for the cues.*
    """
    merged = sorted([*primary, *pinned], key=lambda f: f["timestamp_seconds"])
    for i, frame in enumerate(merged):
        frame["index"] = i
    return merged

The function executes these steps:

1. Union — Concatenates primary (detail frames) and pinned (cue frames) without removing any entries
2. Chronological sort — Orders the combined list by timestamp_seconds to match video timeline
3. Re-indexing — Assigns sequential indices (0, 1, 2...) to reflect new positions

Why Cue Frames Are Preserved

The docstring explicitly states: pinned frames are never dropped. This design guarantees transcript-driven moments remain visible regardless of visual extraction results.

Frame budget management happens before the merge. The caller reserves slots for cue frames, ensuring the union operation won't exceed limits. This separation of concerns keeps merge_frames simple and predictable.

Where merge_frames Gets Called

The merge step integrates into the main watch workflow at lines 228-229 of skills/watch/scripts/watch.py:

if cue_frames:
    frames = merge_frames(frames, cue_frames)   # merge detail & cue frames

This conditional ensures cue frames only merge when transcript analysis produces them.

Practical Code Example

Here's how merge_frames behaves with sample data:

from skills.watch.scripts.frames import merge_frames

# Detail frames from scene-change detection

detail_frames = [
    {"index": 0, "timestamp_seconds": 3.0, "path": "frame_0000.jpg", "reason": "scene-change"},
    {"index": 1, "timestamp_seconds": 7.5, "path": "frame_0001.jpg", "reason": "scene-change"},
]

# Cue frames from transcript timestamps

cue_frames = [
    {"index": 0, "timestamp_seconds": 5.0, "path": "cue_0000.jpg", "reason": "transcript-cue"},
    {"index": 1, "timestamp_seconds": 12.0, "path": "cue_0001.jpg", "reason": "transcript-cue"},
]

merged = merge_frames(detail_frames, cue_frames)

# Result: chronologically sorted and re-indexed

# [

#   {'index': 0, 'timestamp_seconds': 3.0, 'path': 'frame_0000.jpg', ...},

#   {'index': 1, 'timestamp_seconds': 5.0, 'path': 'cue_0000.jpg', ...},

#   {'index': 2, 'timestamp_seconds': 7.5, 'path': 'frame_0001.jpg', ...},

#   {'index': 3, 'timestamp_seconds': 12.0, 'path': 'cue_0001.jpg', ...},

# ]

Note how the 5-second cue frame inserts between the 3-second and 7.5-second detail frames, with indices reset to sequential order.

Key Files in the Merge Pipeline

File	Purpose	Location
skills/watch/scripts/frames.py	merge_frames implementation and extraction utilities	GitHub
skills/watch/scripts/watch.py	Orchestrates workflow and invokes merge_frames	GitHub
tests/test_timestamps.py	Unit tests for sorting, re-indexing, and cue preservation	GitHub

Summary

• merge_frames performs a plain union of detail and cue frames without dropping entries
• Chronological sorting by timestamp_seconds ensures the final sequence matches video playback order
• Re-indexing provides clean sequential indices for downstream processing
• Cue preservation is guaranteed because budget enforcement happens upstream
• Clean separation between budget management and merging keeps the code maintainable

Frequently Asked Questions

Does merge_frames remove duplicate frames at the same timestamp?

No. merge_frames performs a plain union with no deduplication. If a detail frame and cue frame share identical timestamp_seconds values, both appear in output with their relative order preserved from the sort stability. The function prioritizes simplicity and predictability over duplicate detection.

How does Claude Video prevent exceeding frame limits when merging?

Frame budget enforcement occurs before calling merge_frames. The workflow reserves dedicated slots for cue frames when generating detail frames, ensuring the eventual union stays within limits. This design lets merge_frames remain a pure merge operation without side effects.

Can merge_frames handle empty cue frame lists?

Yes. The calling code in watch.py only invokes merge_frames when cue_frames exists and is non-empty. If no transcript cues exist, the detail frames pass through unmodified to downstream processing.

What data structure does each frame dictionary require?

Each frame must contain at minimum a timestamp_seconds numeric field for sorting and an index field for re-indexing. Typical frames also include path (file location) and reason (extraction source) fields, though merge_frames only accesses timestamp_seconds and index directly.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}