How the Transcript-Cue System Pins Specific Timestamps Against the Frame Budget Cap

The transcript-cue system guarantees user-requested timestamps survive frame limits by extracting cue frames first, subtracting their count from the global max_frames allowance, and preventing the detail engine from evicting them.

The bradautomates/claude-video repository implements a sophisticated transcript-cue feature that lets users request exact frames at specific timestamps derived from captions or Whisper transcripts. This system ensures these critical moments are preserved even when operating under strict frame budget constraints. By pinning cue frames before running any detail extraction algorithms, the codebase maintains semantic fidelity to user-specified moments while respecting computational limits.

How the Transcript-Cue System Reserves Frame Budget

The pinning mechanism operates through a strict priority pipeline where transcript cues are processed before any automated frame selection occurs. This sequence ensures that user intent takes precedence over algorithmic sampling.

Parsing User-Provided Timestamps

The workflow begins in skills/watch/scripts/watch.py, where the parse_timestamps function converts user input into a list of seconds. Located at line 81, this utility handles the --timestamps argument and validates the input format before passing the normalized values downstream.

Extracting Cue Frames Before Detail Analysis

Once timestamps are parsed, the system immediately extracts frames at those specific moments via extract_at_timestamps (lines 76-79). This function:

• Discards timestamps outside the focus window
• Respects the user-supplied max_frames cap through even sampling when necessary
• Returns cue_frames that are stored separately from detail-engine output

Because this extraction happens before any keyframe or scene-detection algorithms run, the cue frames are guaranteed to exist regardless of what the detail engine subsequently selects.

Budget Allocation and Pinning Mechanism

After cue extraction, the system performs a critical budget reservation step that mathematically prevents frame over-allocation.

Calculating the Detail Engine Budget

At lines 95-96 in watch.py, the remaining budget for the detail engine is calculated as:

detail_budget = max_frames - len(cue_frames)

This subtraction ensures that the pinned frames consume part of the total allowance, leaving only the residual capacity for automated frame selection. When the detail engine (whether using efficient keyframes or balanced scene-aware extraction) receives its max_frames argument at lines 104-111, it operates within this constrained detail_budget and cannot steal slots from the cue frames.

Preventing Cue Frame Eviction

The final protection occurs in skills/watch/scripts/frames.py within the merge_frames function (lines 12-22). This utility unions the detail-engine frames with the cue frames without filtering:

def merge_frames(primary: list[dict], pinned: list[dict]) -> list[dict]:
    """Combine two frame lists … ``pinned`` frames (transcript cues) are never dropped."""
    merged = sorted([*primary, *pinned], key=lambda f: f["timestamp_seconds"])
    for i, frame in enumerate(merged):
        frame["index"] = i
    return merged

Because the function simply concatenates and sorts the two lists, cue frames persist in the final output even if the detail engine produced more frames than expected.

Handling Budget Overages with Even Sampling

When users request more timestamps than the max_frames cap allows, the system applies intelligent down-sampling rather than failing. In frames.py (lines 54-57), the extract_at_timestamps function implements even sampling:

if max_frames is not None and len(in_window) > max_frames:
    points = [in_window[i] for i in _even_indices(len(in_window), max_frames)]

This algorithm preserves the first and last requested timestamps while distributing selections evenly across the remaining cues. The metadata returned by extract_at_timestamps also records any timestamps that fell outside the focused range in dropped_out_of_window, ensuring transparency when constraints force omissions.

Code Implementation Example

The following demonstrates the complete workflow from command-line invocation through budget reservation:

# Request two specific timestamps while limiting total frames to 8

watch https://youtu.be/abc123 --timestamps "5.0,12.3" --max-frames 8

Inside watch.py, the orchestration logic executes:

# Parse and extract cue frames first

cue_timestamps = parse_timestamps(args.timestamps)               # → [5.0, 12.3]

cue_frames, cue_meta = extract_at_timestamps(
    video_path, work / "frames", cue_timestamps,
    max_frames=args.max_frames
)                                                               # → 2 cue frames

# Reserve remaining budget for detail engine

detail_budget = max_frames - len(cue_frames)                    # → 6 remaining frames

frames, frame_meta = extract_keyframes(..., max_frames=detail_budget)

# Merge without dropping pinned frames

frames = merge_frames(frames, cue_frames)                       # final list = 8 frames

Summary

• Early extraction guarantees cue frames exist before detail algorithms run, preventing eviction by scene-detection logic.
• Budget reservation via detail_budget = max_frames - len(cue_frames) mathematically isolates cue frame capacity from detail-engine allocation.
• Even sampling automatically handles cases where user requests exceed the cap, preserving temporal distribution through first/last spacing.
• Immutable merging in merge_frames ensures pinned timestamps survive the final frame compilation regardless of detail-engine output volume.
• Metadata transparency reports out-of-window drops without counting them against the frame budget.

Frequently Asked Questions

What happens if I request more timestamps than the max_frames limit?

The system applies even sampling to distribute selections across your requested range while preserving the first and last timestamps. As implemented in frames.py, the _even_indices helper selects evenly spaced indices from the input list, ensuring representative coverage without exceeding the cap.

Can the detail engine accidentally remove my pinned cue frames?

No. The detail engine operates on a separate detail_budget calculated after cue extraction. Because merge_frames simply unions the two lists without filtering, and cue frames are extracted first, the detail engine has no mechanism to evict or overwrite pinned timestamps.

How does the system handle timestamps outside the video focus window?

Timestamps falling outside the specified focus window are recorded in the dropped_out_of_window metadata field but are not counted against the frame budget. Only timestamps within the window compete for the max_frames allocation.

Where is the frame budget logic implemented in the source code?

The primary budget calculation occurs in skills/watch/scripts/watch.py at lines 95-96, where detail_budget is derived by subtracting the cue frame count from max_frames. The enforcement happens at lines 104-111, where this budget is passed as the max_frames argument to the chosen detail extraction engine.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}