How Transcript-Cue Timestamps Are Extracted and Merged with Detail Frames in claude-video

The claude-video skill extracts transcript-cue timestamps using parse_timestamps() and extract_at_timestamps(), then merges them with detail-selected frames via merge_frames() to create a chronological visual summary.

The claude-video repository provides AI-powered video analysis tools that synchronize visual frames with transcript cues. Understanding how transcript-cue timestamps are extracted and merged with detail-selected frames is essential for generating accurate video summaries. This pipeline is implemented in the skills/watch/scripts/ directory, specifically within frames.py and orchestrated by watch.py.

Overview of the Frame Selection Pipeline

The visual summary pipeline operates in three distinct stages:

1. Transcript-cue extraction – Processing user-provided timestamps from the --timestamps flag.
2. Automatic detail selection – Running scene-change, key-frame, or uniform sampling engines based on the --detail option.
3. Chronological merging – Combining both frame sets into a final ordered list.

All merging and extraction logic resides in skills/watch/scripts/frames.py, while the high-level coordination occurs in skills/watch/scripts/watch.py.

Parsing Transcript-Cue Timestamps

User-provided timestamps undergo normalization through the parse_timestamps() function in skills/watch/scripts/frames.py (lines 295–310).

The parser handles multiple time formats including SS, MM:SS, and HH:MM:SS. The parse_time() helper converts each entry to floating-point seconds, removes duplicates, and returns a sorted list.

# From frames.py - conceptual usage

cue_timestamps = parse_timestamps("1:30,45,2:15")

# Returns: [45.0, 90.0, 135.0] (sorted)

This ensures that transcript-cue timestamps are standardized before frame extraction begins.

Extracting Frames at Cue Timestamps

The extract_at_timestamps() function (lines 324–390 in skills/watch/scripts/frames.py) handles the actual frame extraction for each parsed cue.

This function performs several critical operations:

• Window filtering – Discards timestamps outside the active [start, end] segment window, tracking them as dropped_out_of_window in metadata.
• Budget enforcement – When the cue count exceeds max_frames, it applies even sampling via _even_indices() while preserving the first and last cues.
• Frame generation – Executes single-frame ffmpeg extractions for each valid timestamp.

Each extracted frame receives a reason field set to "transcript-cue", distinguishing it from automatically selected frames.

cue_frames, cue_meta = extract_at_timestamps(
    video_path="input.mp4",
    out_dir=Path("/tmp/frames"),
    timestamps=[30.0, 60.0, 90.0],
    resolution=512,
    max_frames=10  # Optional cap with even sampling

)

Selecting Detail Frames

Simultaneously, the system selects primary detail frames using one of three engines based on the --detail CLI argument:

• extract_scene_or_uniform() – Balanced scene-change detection with uniform fallback.
• extract_keyframes() – Extracts video keyframes only.
• extract() – Uniform time sampling.

These functions return frame dictionaries with reason fields such as "scene-change" or "uniform", depending on the extraction method.

Merging Cue and Detail Frames

The merge_frames() function (lines 312–321 in skills/watch/scripts/frames.py) combines the primary detail frames with the pinned transcript-cue frames.

The merging process follows this sequence:

1. Concatenates the detail frame list with the cue frame list.
2. Sorts the combined list by timestamp_seconds to ensure chronological order.
3. Re-indexes entries from 0 to n-1.

Because transcript-cue frames are treated as "pinned," they are never dropped during budget allocation—reserved slots are maintained for the automatic detail selection.

final_frames = merge_frames(detail_frames, cue_frames)

# Produces chronologically ordered list with mixed reasons

Driver Orchestration in watch.py

The skills/watch/scripts/watch.py file coordinates the entire pipeline. Around lines 178–190, the driver:

1. Parses the raw timestamp string using parse_timestamps(args.timestamps).
2. Calls extract_at_timestamps() to generate cue_frames and metadata.
3. Executes the selected detail engine to produce primary_frames.
4. Invokes merge_frames(primary_frames, cue_frames) to create the final ordered set.

# Conceptual flow from watch.py

if args.timestamps:
    cue_timestamps = parse_timestamps(args.timestamps)
    cue_frames, cue_meta = extract_at_timestamps(
        video_path, out_dir, cue_timestamps, ...
    )

primary_frames, _ = extract_scene_or_uniform(...)  # or other engine

merged = merge_frames(primary_frames, cue_frames)

Practical Implementation Example

This complete example demonstrates the transcript-cue workflow:

from pathlib import Path
from frames import (
    parse_timestamps, 
    extract_at_timestamps, 
    extract_scene_or_uniform,
    merge_frames
)

# Step 1: Parse user-provided transcript cues

cue_timestamps = parse_timestamps("1:00,3:30,5:45")

# Step 2: Extract frames at cue timestamps

cue_frames, cue_meta = extract_at_timestamps(
    video_path="lecture.mp4",
    out_dir=Path("/tmp/cues"),
    timestamps=cue_timestamps,
    resolution=512,
    max_frames=None  # Keep all cues

)

# Step 3: Generate detail frames (scene-change detection)

detail_frames, _ = extract_scene_or_uniform(
    video_path="lecture.mp4",
    out_dir=Path("/tmp/detail"),
    fps=2.0,
    target_frames=20,
    resolution=512
)

# Step 4: Merge into chronological sequence

final_frames = merge_frames(detail_frames, cue_frames)

print(f"Total frames: {len(final_frames)}")
print(f"First frame reason: {final_frames[0]['reason']}")

Summary

• Transcript-cue timestamps are parsed via parse_timestamps() in skills/watch/scripts/frames.py, supporting SS, MM:SS, and HH:MM:SS formats.
• Frame extraction occurs through extract_at_timestamps(), which filters by time windows, applies even sampling when budget-constrained, and tags frames with reason: "transcript-cue".
• Detail frames are selected by separate engines (extract_scene_or_uniform, extract_keyframes, or extract) based on the --detail CLI option.
• Chronological merging is handled by merge_frames(), which concatenates both lists, sorts by timestamp_seconds, and re-indexes the results.
• Pipeline orchestration in skills/watch/scripts/watch.py coordinates parsing, extraction, and merging before presenting the final visual summary.

Frequently Asked Questions

What timestamp formats does claude-video accept for transcript cues?

The system accepts absolute timestamps in three formats: seconds only (SS), minutes and seconds (MM:SS), or hours, minutes, and seconds (HH:MM:SS). The parse_time() function in skills/watch/scripts/frames.py normalizes all formats to floating-point seconds for consistent processing.

How does claude-video handle transcript cues that fall outside the video segment?

Timestamps outside the user-specified [start, end] range are filtered out by extract_at_timestamps() and tracked in the metadata dictionary under dropped_out_of_window. These cues are excluded from frame extraction but reported for debugging purposes.

Can I limit the number of transcript-cue frames while preserving specific timestamps?

Yes. When the number of cues exceeds the max_frames parameter, extract_at_timestamps() applies _even_indices() to distribute selections evenly while always preserving the first and last timestamps in the list. This maintains coverage across the video duration without exceeding the frame budget.

How does the merge_frames function ensure chronological order?

The merge_frames() function concatenates the detail frames and cue frames, then sorts the combined list by the timestamp_seconds field. After sorting, it re-indexes the entries from 0 to n-1, guaranteeing that the downstream UI receives a monotonic timeline for rendering the visual story.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}