How Transcript-Cue Frames with --timestamps Enable Deictic Moments in Claude-video

Claude-video's --timestamps flag extracts specific video frames at absolute timestamps provided via CLI, creating pinned visual anchors that correspond to transcript-flagged moments and allowing Claude to "look here" during analysis.

The bradautomates/claude-video repository provides a /watch command for processing video content through AI analysis. When you need to ensure Claude examines specific moments mentioned in a transcript—such as when a speaker references a visual element or says "look at this"—the --timestamps feature creates deictic reference points by extracting cue frames at exact temporal coordinates.

How the --timestamps Pipeline Works

The workflow implemented in skills/watch/scripts/watch.py and skills/watch/scripts/frames.py converts raw timestamp strings into prioritized visual data through three distinct phases.

Parsing Absolute Time Formats

The raw string from --timestamps (e.g., 1,3:15,00:02:30) is processed by parse_timestamps in skills/watch/scripts/frames.py (line 95). This function delegates to parse_time (also in frames.py) to handle multiple input formats:

• SS: Seconds only (e.g., 90)
• MM:SS: Minutes and seconds (e.g., 3:15)
• HH:MM:SS: Hours, minutes, seconds (e.g., 00:02:30)

The function returns a sorted list of floating-point seconds that drives the extraction logic.

Extracting Pinned Cue Frames

In skills/watch/scripts/watch.py (line 76), parsed timestamps are stored in the cue_timestamps variable. When cue timestamps are present and a video file is available, the system calls extract_at_timestamps from frames.py (lines 33-41):

cue_frames, cue_meta = extract_at_timestamps(
    video_path,
    work / "frames",
    cue_timestamps,
    resolution=args.resolution,
    max_frames=max_frames,
    start_seconds=start_sec,
    end_seconds=end_sec,
)

The extract_at_timestamps function performs the following operations:

1. Filters timestamps to the user-specified focus window using start_seconds and end_seconds
2. Samples evenly using _even_indices if more timestamps exist than the max_frames budget allows
3. Executes single-frame seeks via ffmpeg using the -ss <t> parameter to capture JPEGs named cue_XXXX.jpg
4. Returns metadata with engine: "timestamps" indicating the extraction source

Critically, these cue frames are pinned—they are extracted before any detail-engine frames and are reserved against the frame cap, meaning the later scene-aware, keyframe, or uniform extraction will never evict them.

Merging with Detail Engine Output

After the regular detail engine produces its frame list, cue frames are merged using merge_frames (lines 12-21 in frames.py):

if cue_frames:
    frames = merge_frames(frames, cue_frames)

This function sorts both lists chronologically and re-indexes them, guaranteeing that cue frames appear in the final markdown report exactly where their timestamps occur. The generated report includes entries such as:

- `tmp/frames/cue_0000.jpg` (t=00:10, reason=transcript-cue)
- `tmp/frames/cue_0001.jpg` (t=02:30, reason=transcript-cue)

And identifies the extraction method with a summary line like:

- **Cue frames:** 3 at transcript-flagged timestamps (transcript-cue)

Practical Usage Examples

Command-Line Interface

Extract frames at 10 seconds and 2 minutes 30 seconds while using the balanced detail engine:

watch https://example.com/video.mp4 --detail balanced --timestamps 10,02:30

Python API Integration

Replicate the CLI workflow programmatically:

from pathlib import Path
from frames import parse_timestamps, extract_at_timestamps, merge_frames

video_path = Path("my_video.mp4")
work_dir = Path("tmp/frames")

# Parse the timestamp string into seconds

timestamps = parse_timestamps("10,02:30")  # → [10.0, 150.0]

# Extract cue frames (pinned against the budget)

cue_frames, cue_meta = extract_at_timestamps(
    video_path,
    work_dir,
    timestamps,
    resolution=512,
    max_frames=None,
    start_seconds=None,
    end_seconds=None,
)

# Merge with existing detail frames from scene/keyframe engines

detail_frames = [...]  # List of frame dicts from other engines

final_frames = merge_frames(detail_frames, cue_frames)

Key Technical Files

The deictic functionality spans these core components:

• skills/watch/scripts/watch.py: Main entry point that orchestrates download, cue-frame extraction, detail-engine selection, and report generation
• skills/watch/scripts/frames.py: Contains parse_timestamps, parse_time, extract_at_timestamps, and merge_frames utilities
• skills/watch/scripts/transcribe.py: Handles VTT caption parsing when transcript segments are required for timestamp identification
• tests/test_timestamps.py: Validates timestamp parsing and extraction logic

Summary

• Transcript-cue frames are extracted via --timestamps using absolute time coordinates in SS, MM:SS, or HH:MM:SS formats
• parse_timestamps and extract_at_timestamps in frames.py convert CLI strings into JPEG frames using ffmpeg single-frame seeks
• Cue frames are pinned and reserved against the max_frames budget, ensuring they survive the detail-engine selection process
• merge_frames chronologically combines cue frames with scene-aware or uniform frames, inserting them at precise temporal locations in the final report
• The resulting deictic anchors provide visual evidence tied to specific transcript moments, enabling Claude to reference exact visual content

Frequently Asked Questions

What timestamp formats does Claude-video support?

Claude-video accepts three formats parsed by parse_time in frames.py: raw seconds (e.g., 90), minutes:seconds (e.g., 3:15), and hours:minutes:seconds (e.g., 00:02:30). The --timestamps flag accepts a comma-separated list combining any of these formats.

How do cue frames interact with the frame budget?

Cue frames are reserved against the max_frames cap. If you specify 10 timestamps but set --max-frames 20, the system extracts all 10 cue frames first, then allows the detail engine (scene, keyframe, or uniform) to fill the remaining 10 slots. If timestamps exceed the budget, _even_indices evenly samples from your requested times.

Can I use --timestamps with any detail engine?

Yes. The --timestamps flag works independently of the --detail engine selection (scene, keyframe, or uniform). Cue frames are extracted prior to the detail engine running and are merged afterward, ensuring they appear regardless of which analysis method you choose.

Where are the cue frames saved in the output?

Cue frames are saved as JPEG files in the working directory's frames subfolder (e.g., tmp/frames/cue_0000.jpg). The final markdown report lists them with their absolute timestamp (t=MM:SS) and labels them with reason=transcript-cue to distinguish them from automatically detected frames.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}