How Cue Timestamps Are Prioritized Over the Frame Cap in claude-video

In claude-video, user-provided cue timestamps are extracted first and subtracted from the global frame budget, ensuring these pinned frames are preserved while the detail engine only receives the remaining allocation.

The open-source claude-video tool processes video content by balancing automated frame extraction with user-specified requirements. When you provide timestamps via the --timestamps flag, those frames become cue frames that take precedence over automated selection methods. Understanding how cue timestamps are prioritized over the frame cap is essential for controlling exactly which moments appear in your final analysis.

Understanding the Cue Timestamp System

In claude-video, timestamps supplied with the --timestamps flag represent pinned cue frames that must appear in the final report. Unlike frames generated by the detail engine—which includes keyframes, scene-aware frames, or uniform sampling—these user-specified timestamps are treated as hard requirements. The system implements this priority through strict ordering in the extraction pipeline defined in skills/watch/scripts/watch.py.

How the Frame Budget Is Reserved for Cue Timestamps

The prioritization logic follows a four-step process that guarantees cue frames never compete with detail frames for budget space.

Parsing and Validating Timestamps

First, the raw comma-separated timestamp strings are converted into a sorted list of seconds. The parse_timestamps function in skills/watch/scripts/frames.py handles this conversion, normalizing all values before extraction begins.

Extracting Cue Frames Before the Detail Engine

Next, watch.py calls extract_at_timestamps with the full max_frames value. This function drops timestamps outside the focus range (when --start or --end boundaries are set) and applies even-sampling if the number of valid timestamps exceeds the global cap. According to the source code in skills/watch/scripts/frames.py, if more timestamps survive than the supplied max_frames, the function even-samples them to fit the cap, ensuring the total never exceeds the user-specified limit.

Calculating the Remaining Detail Budget

After cue extraction, the remaining budget for the detail engine is calculated:

detail_budget = max_frames if max_frames is None else max(0, max_frames - len(cue_frames))

This line from skills/watch/scripts/watch.py ensures the detail engine can only use the unallocated portion of the cap. If you specify 20 frames total and provide 3 cue timestamps, the detail engine receives a budget of exactly 17 frames.

The Consequences of Cue Frame Priority

This budget reservation strategy creates specific guarantees in the final output. Cue frames are always present in the final report because they are extracted before any budget constraints are applied to the detail engine. If the number of cue frames equals or exceeds the original max_frames, the detail engine receives a budget of 0 and no additional frames are extracted. Finally, the merge_frames function in skills/watch/scripts/frames.py combines both sets while preserving all cue frames, ensuring they are never dropped during the merge process.

Practical Code Examples

To request specific frames while maintaining a global cap, use the --timestamps flag with --max-frames:

claude-video watch https://example.com/video.mp4 \
  --detail balanced \
  --max-frames 20 \
  --timestamps "00:05,00:15,00:45"

In this example, the three timestamps are parsed, extracted first, and count against the 20-frame budget. If the balanced detail engine would normally produce 18 frames, only 17 are kept (20 minus 3).

Programmatically, you can observe the budget calculation:

from frames import parse_timestamps, extract_at_timestamps

timestamps = parse_timestamps("00:05,00:15,00:45")
cue_frames, meta = extract_at_timestamps(
    video_path="video.mp4",
    out_dir=Path("./frames"),
    timestamps=timestamps,
    max_frames=20,            # global cap

)

# cue_frames contains up to 3 frames; remaining slots available for detail engine

Summary

• Cue timestamps are treated as pinned frames that must appear in the final output.
• The extract_at_timestamps function runs before the detail engine with the full max_frames budget.
• The remaining budget is calculated by subtracting the cue frame count from max_frames, ensuring the total never exceeds the cap.
• Cue frames are preserved through the final merge_frames operation and cannot be displaced by detail engine results.

Frequently Asked Questions

What happens if I provide more timestamps than the max-frames cap?

If the number of cue timestamps exceeds max_frames, the extract_at_timestamps function in skills/watch/scripts/frames.py automatically even-samples them to fit within the cap. The detail engine then receives a budget of 0, and only the sampled cue frames appear in the final report.

Do cue timestamps work with the --start and --end flags?

Yes. The system first filters timestamps to only those within the specified focus range before calculating the budget. Timestamps outside the range are dropped, and the remaining valid timestamps are processed against the frame cap in skills/watch/scripts/frames.py.

Why does the detail engine get the remaining budget instead of sharing it equally?

The design treats user-specified timestamps as hard requirements that take precedence over automated extraction. By calculating detail_budget as max(0, max_frames - len(cue_frames)) in skills/watch/scripts/watch.py, the system guarantees that manual selections are never sacrificed to accommodate algorithmic frame selection.

Where is the default frame cap defined for different detail levels?

The default frame caps for detail modes like balanced or high are defined in skills/watch/scripts/config.py. These values serve as the max_frames parameter when users don't explicitly override them with --max-frames.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}