# How Transcript-Cue Timestamps Are Extracted and Merged with Detail Frames in claude-video

> Discover how claude-video extracts transcript-cue timestamps using parse_timestamps and extract_at_timestamps, then merges them with detail-selected frames for a chronological visual summary.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: internals
- Published: 2026-07-14

---

**The claude-video skill extracts transcript-cue timestamps using `parse_timestamps()` and `extract_at_timestamps()`, then merges them with detail-selected frames via `merge_frames()` to create a chronological visual summary.**

The claude-video repository provides AI-powered video analysis tools that synchronize visual frames with transcript cues. Understanding how transcript-cue timestamps are extracted and merged with detail-selected frames is essential for generating accurate video summaries. This pipeline is implemented in the `skills/watch/scripts/` directory, specifically within [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py) and orchestrated by [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py).

## Overview of the Frame Selection Pipeline

The visual summary pipeline operates in three distinct stages:

1. **Transcript-cue extraction** – Processing user-provided timestamps from the `--timestamps` flag.
2. **Automatic detail selection** – Running scene-change, key-frame, or uniform sampling engines based on the `--detail` option.
3. **Chronological merging** – Combining both frame sets into a final ordered list.

All merging and extraction logic resides in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), while the high-level coordination occurs in [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py).

## Parsing Transcript-Cue Timestamps

User-provided timestamps undergo normalization through the `parse_timestamps()` function in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) (lines 295–310).

The parser handles multiple time formats including `SS`, `MM:SS`, and `HH:MM:SS`. The `parse_time()` helper converts each entry to floating-point seconds, removes duplicates, and returns a sorted list.

```python

# From frames.py - conceptual usage

cue_timestamps = parse_timestamps("1:30,45,2:15")

# Returns: [45.0, 90.0, 135.0] (sorted)

```

This ensures that transcript-cue timestamps are standardized before frame extraction begins.

## Extracting Frames at Cue Timestamps

The `extract_at_timestamps()` function (lines 324–390 in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py)) handles the actual frame extraction for each parsed cue.

This function performs several critical operations:

- **Window filtering** – Discards timestamps outside the active `[start, end]` segment window, tracking them as `dropped_out_of_window` in metadata.
- **Budget enforcement** – When the cue count exceeds `max_frames`, it applies even sampling via `_even_indices()` while preserving the first and last cues.
- **Frame generation** – Executes single-frame `ffmpeg` extractions for each valid timestamp.

Each extracted frame receives a `reason` field set to `"transcript-cue"`, distinguishing it from automatically selected frames.

```python
cue_frames, cue_meta = extract_at_timestamps(
    video_path="input.mp4",
    out_dir=Path("/tmp/frames"),
    timestamps=[30.0, 60.0, 90.0],
    resolution=512,
    max_frames=10  # Optional cap with even sampling

)

```

## Selecting Detail Frames

Simultaneously, the system selects primary detail frames using one of three engines based on the `--detail` CLI argument:

- **`extract_scene_or_uniform()`** – Balanced scene-change detection with uniform fallback.
- **`extract_keyframes()`** – Extracts video keyframes only.
- **`extract()`** – Uniform time sampling.

These functions return frame dictionaries with `reason` fields such as `"scene-change"` or `"uniform"`, depending on the extraction method.

## Merging Cue and Detail Frames

The `merge_frames()` function (lines 312–321 in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py)) combines the primary detail frames with the pinned transcript-cue frames.

The merging process follows this sequence:

1. Concatenates the detail frame list with the cue frame list.
2. Sorts the combined list by `timestamp_seconds` to ensure chronological order.
3. Re-indexes entries from `0` to `n-1`.

Because transcript-cue frames are treated as "pinned," they are never dropped during budget allocation—reserved slots are maintained for the automatic detail selection.

```python
final_frames = merge_frames(detail_frames, cue_frames)

# Produces chronologically ordered list with mixed reasons

```

## Driver Orchestration in watch.py

The [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py) file coordinates the entire pipeline. Around lines 178–190, the driver:

1. Parses the raw timestamp string using `parse_timestamps(args.timestamps)`.
2. Calls `extract_at_timestamps()` to generate `cue_frames` and metadata.
3. Executes the selected detail engine to produce `primary_frames`.
4. Invokes `merge_frames(primary_frames, cue_frames)` to create the final ordered set.

```python

# Conceptual flow from watch.py

if args.timestamps:
    cue_timestamps = parse_timestamps(args.timestamps)
    cue_frames, cue_meta = extract_at_timestamps(
        video_path, out_dir, cue_timestamps, ...
    )

primary_frames, _ = extract_scene_or_uniform(...)  # or other engine

merged = merge_frames(primary_frames, cue_frames)

```

## Practical Implementation Example

This complete example demonstrates the transcript-cue workflow:

```python
from pathlib import Path
from frames import (
    parse_timestamps, 
    extract_at_timestamps, 
    extract_scene_or_uniform,
    merge_frames
)

# Step 1: Parse user-provided transcript cues

cue_timestamps = parse_timestamps("1:00,3:30,5:45")

# Step 2: Extract frames at cue timestamps

cue_frames, cue_meta = extract_at_timestamps(
    video_path="lecture.mp4",
    out_dir=Path("/tmp/cues"),
    timestamps=cue_timestamps,
    resolution=512,
    max_frames=None  # Keep all cues

)

# Step 3: Generate detail frames (scene-change detection)

detail_frames, _ = extract_scene_or_uniform(
    video_path="lecture.mp4",
    out_dir=Path("/tmp/detail"),
    fps=2.0,
    target_frames=20,
    resolution=512
)

# Step 4: Merge into chronological sequence

final_frames = merge_frames(detail_frames, cue_frames)

print(f"Total frames: {len(final_frames)}")
print(f"First frame reason: {final_frames[0]['reason']}")

```

## Summary

- **Transcript-cue timestamps** are parsed via `parse_timestamps()` in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), supporting `SS`, `MM:SS`, and `HH:MM:SS` formats.
- **Frame extraction** occurs through `extract_at_timestamps()`, which filters by time windows, applies even sampling when budget-constrained, and tags frames with `reason: "transcript-cue"`.
- **Detail frames** are selected by separate engines (`extract_scene_or_uniform`, `extract_keyframes`, or `extract`) based on the `--detail` CLI option.
- **Chronological merging** is handled by `merge_frames()`, which concatenates both lists, sorts by `timestamp_seconds`, and re-indexes the results.
- **Pipeline orchestration** in [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py) coordinates parsing, extraction, and merging before presenting the final visual summary.

## Frequently Asked Questions

### What timestamp formats does claude-video accept for transcript cues?

The system accepts absolute timestamps in three formats: seconds only (`SS`), minutes and seconds (`MM:SS`), or hours, minutes, and seconds (`HH:MM:SS`). The `parse_time()` function in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) normalizes all formats to floating-point seconds for consistent processing.

### How does claude-video handle transcript cues that fall outside the video segment?

Timestamps outside the user-specified `[start, end]` range are filtered out by `extract_at_timestamps()` and tracked in the metadata dictionary under `dropped_out_of_window`. These cues are excluded from frame extraction but reported for debugging purposes.

### Can I limit the number of transcript-cue frames while preserving specific timestamps?

Yes. When the number of cues exceeds the `max_frames` parameter, `extract_at_timestamps()` applies `_even_indices()` to distribute selections evenly while always preserving the first and last timestamps in the list. This maintains coverage across the video duration without exceeding the frame budget.

### How does the merge_frames function ensure chronological order?

The `merge_frames()` function concatenates the detail frames and cue frames, then sorts the combined list by the `timestamp_seconds` field. After sorting, it re-indexes the entries from `0` to `n-1`, guaranteeing that the downstream UI receives a monotonic timeline for rendering the visual story.