How Pinned Cue Frames Ensure Explicit Timestamp Requests Survive the Frame Cap

Cue frames are guaranteed to survive the frame cap because they are extracted first, reserved in the budget calculation, and merged without eviction—ensuring explicit timestamp requests are never dropped.

In the bradautomates/claude-video repository, the watch skill handles two distinct frame types: cue frames (user-requested timestamps) and detail frames (automatically extracted scenes). The architecture prioritizes cue frames through a three-stage pipeline that reserves capacity before any automatic extraction occurs.

The Three-Stage Protection Mechanism

Stage 1: Extract Cue Frames Before Any Budget Enforcement

When cue_timestamps are provided, the system calls extract_at_timestamps() immediately—before the detail engine runs. This ensures explicit requests are satisfied first, independent of the max_frames limit.

In watch.py (lines 76-80):

# 1️⃣ Pull out any user-requested timestamps

if cue_timestamps and video_path:
    cue_frames, cue_meta = extract_at_timestamps(
        video_path,
        work / "frames",
        cue_timestamps,
        resolution=args.resolution,
        max_frames=max_frames,
        start_seconds=start_sec,
        end_seconds=end_sec,
    )
    # Log any timestamps that fell outside the focus range

    if cue_meta.get("dropped_out_of_window"):
        print("[watch] … cue timestamps outside the focus range — dropped", file=sys.stderr)

Only timestamps outside the user-specified focus window are dropped at this stage. The remaining cue frames are locked in and counted toward the final output.

Stage 2: Reserve Budget Exclusively for Detail Extraction

The critical protection happens in the budget calculation. The system subtracts cue frame count from max_frames, creating a separate detail budget that cannot touch the reserved cue frames.

In watch.py (lines 95-96):

# 2️⃣ Reserve space for those frames before running the detail engine

detail_budget = max_frames if max_frames is None else max(0, max_frames - len(cue_frames))

This ensures:

• The detail engine operates on detail_budget only
• Cue frames remain outside the scope of automatic extraction limits
• Frame cap enforcement applies solely to scene-based frames

Stage 3: Merge Without Eviction via merge_frames()

After detail extraction completes, the two lists are combined. The merge_frames() function in frames.py treats cue frames as pinned—performing a simple union operation with no filtering or eviction logic.

In frames.py (lines 312-319):

def merge_frames(primary: list[dict], pinned: list[dict]) -> list[dict]:
    """
    Combine two frame lists into one chronological list and reindex 0..n-1.

    ``pinned`` frames (transcript cues) are never dropped — this is a plain
    union, so the cap is enforced upstream by reserving budget for the cues.
    """
    merged = sorted([*primary, *pinned], key=lambda f: f["timestamp_seconds"])
    for i, frame in enumerate(merged):
        frame["index"] = i
    return merged

The function sorts chronologically and reindexes—never discarding elements. The comment explicitly documents this contract: eviction prevention happens upstream through budget reservation, making the merge operation safe and simple.

Complete Pipeline Example

Here's the full flow from watch.py showing how explicit timestamp requests are protected end-to-end:

# 1️⃣ Pull out any user‑requested timestamps

if cue_timestamps and video_path:
    cue_frames, cue_meta = extract_at_timestamps(
        video_path,
        work / "frames",
        cue_timestamps,
        resolution=args.resolution,
        max_frames=max_frames,
        start_seconds=start_sec,
        end_seconds=end_sec,
    )

# 2️⃣ Reserve space for those frames before running the detail engine

detail_budget = max_frames if max_frames is None else max(0, max_frames - len(cue_frames))

# …run the detail engine (keyframes, scene‑aware, etc.) using `detail_budget`…

# 3️⃣ Merge the two lists – pinned cue frames are never evicted

if cue_frames:
    frames = merge_frames(frames, cue_frames)

Why This Design Guarantees Frame Survival

Protection Layer	Mechanism	Code Location
Temporal priority	Cue extraction runs before detail engine	watch.py:76-80
Budget isolation	detail_budget = max_frames - len(cue_frames)	watch.py:95-96
Non-destructive merge	merge_frames() performs pure union	frames.py:312-319

These layers ensure that explicit timestamp requests are always honored, even when the overall frame cap would otherwise force eviction. The frame cap applies only to automatically generated detail frames—the user's explicit selections bypass the constraint entirely through reservation.

Key Source Files

File	Purpose	Direct Link
skills/watch/scripts/watch.py	Orchestrates cue extraction, budget reservation, and final merge	View on GitHub
skills/watch/scripts/frames.py	Core frame utilities including merge_frames()	View on GitHub
tests/test_timestamps.py	Unit tests verifying cue frame preservation under cap constraints	View on GitHub

Summary

• Cue frames (explicit timestamp requests) are extracted first and reserved in the frame budget
• The detail budget is calculated as max(0, max_frames - len(cue_frames)), isolating automatic extraction from pinned frames
• merge_frames() performs a non-destructive union, with the comment explicitly stating that eviction prevention happens upstream
• Cue frames dropped only if they fall outside the user-specified focus window—never due to frame cap pressure
• Unit tests in test_timestamps.py verify this behavior under cap-constrained conditions

Frequently Asked Questions

What happens if cue timestamps exceed the max_frames value entirely?

The detail_budget calculation uses max(0, max_frames - len(cue_frames)), which floors at zero. This means the detail engine receives no budget and produces no automatic frames, but all valid cue frames are still extracted and preserved. The merge_frames() function returns only the cue list in this case.

Can cue frames be dropped for any reason other than the frame cap?

Yes—cue frames are filtered solely by extract_at_timestamps() if their timestamps fall outside the start_seconds/end_seconds focus window. This is logged via cue_meta.get("dropped_out_of_window"). These drops are user-controlled range constraints, not automatic eviction.

Why is the frame cap passed to extract_at_timestamps() if cue frames are supposed to bypass it?

The max_frames parameter in extract_at_timestamps() is used for input validation and logging purposes, not for limiting cue frame output. The actual enforcement happens downstream through the detail_budget reservation and the non-destructive merge_frames() operation.

How does merge_frames() handle timestamp collisions between cue and detail frames?

The function sorts by timestamp_seconds and reindexes sequentially. If two frames share identical timestamps, Python's stable sort preserves original order. No deduplication occurs—both frames appear in the final output, maintaining the contract that pinned frames are never dropped.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}