How Claude Video Achieves Chronological Order with Timestamps in Frame Extraction

Claude Video extracts frames using ffmpeg and preserves chronological order by computing precise timestamps for each frame, sorting files by physical sequence, and re-indexing mixed sources using timestamp-based merging.

Claude Video is an open-source video processing framework that maintains temporal accuracy across frame extraction pipelines. The system ensures chronological integrity by assigning precise timestamps during capture, enforcing physical file ordering, and implementing robust merging algorithms. This timestamp-centric approach guarantees that downstream transcription and analysis components receive frames in true chronological sequence regardless of extraction method.

Timestamp Capture During Frame Extraction

Claude Video employs different timestamp strategies depending on the extraction mode, but both methods yield temporally accurate frame metadata that preserves the original video sequence.

Calculating Timestamps for Uniform Extraction

For uniform frame extraction via the extract() function, timestamps are computed mathematically from the start offset and frame index. In skills/watch/scripts/frames.py, the system calculates each frame's temporal position using:

"timestamp_seconds": round(offset + (i / fps if fps > 0 else 0.0), 2)

This computation ensures that every extracted frame receives a precise timestamp relative to the video start, with sub-second accuracy rounded to two decimal places. The fps parameter determines the temporal spacing between consecutive frames, while offset accounts for any starting position adjustments.

Parsing Timestamps from Scene-Change Detection

When using scene-change extraction via extract_scene_candidates(), Claude Video leverages ffmpeg's showinfo filter to capture exact presentation timestamps. The system parses ffmpeg's output using the regular expression:

SHOWINFO_TS_RE = re.compile(r"pts_time:([0-9.]+)")

This regex extracts pts_time values from ffmpeg's filter output, creating timestamps that align exactly with the video's internal clock rather than calculated approximations. The extracted values correspond precisely to the saved JPEG files, ensuring temporal accuracy for scene-based frames.

Sorting Frame Files by Physical Sequence

Before timestamp assignment, Claude Video enforces chronological order by sorting physical files on disk. Both extraction methods sort the output directory contents using:

sorted(out_dir.glob("frame_*.jpg"))

This step guarantees that the physical order of JPEG files matches the chronological order in which ffmpeg wrote them. By sorting on disk before processing timestamps, the system creates a deterministic sequence that corresponds to the source video's temporal flow, preventing filesystem ordering issues from disrupting the frame sequence.

Merging Frame Sources with Timestamp-Based Re-indexing

When combining frames from multiple sources—such as primary extraction frames and transcript cue frames—Claude Video uses the merge_frames() function to maintain chronological integrity. The merging process in skills/watch/scripts/frames.py re-indexes all frames by sorting on the timestamp_seconds key:

merged = sorted([*primary, *pinned], key=lambda f: f["timestamp_seconds"])

This consolidation step guarantees a single, chronological sequence even when mixing different extraction engines or frame sources. The timestamp-based sort ensures that scene-change frames, uniform extraction frames, and transcript cue frames interleave correctly according to their actual temporal positions in the source video.

Practical Implementation Examples

Uniform Extraction with Calculated Timestamps

from pathlib import Path
from skills.watch.scripts.frames import extract, get_metadata, auto_fps

video = "sample.mp4"
out_dir = Path("frames")
meta = get_metadata(video)
fps, _ = auto_fps(meta["duration_seconds"])
frames = extract(video, out_dir, fps=fps)

# `frames` is a list of dicts sorted by timestamp_seconds

print(frames[0]["timestamp_seconds"])  # → 0.00 (first frame)

print(frames[-1]["timestamp_seconds"]) # → video duration (last frame)

Scene-Change Extraction with Parsed Timestamps

from pathlib import Path
from skills.watch.scripts.frames import extract_scene_candidates

video = "sample.mp4"
out_dir = Path("scene_frames")
candidates = extract_scene_candidates(video, out_dir)

# Each candidate already contains an accurate timestamp

for f in candidates[:3]:
    print(f["timestamp_seconds"], f["reason"])

# → 0.00 first-frame

# → 12.34 scene-change

# → 24.78 scene-change

Merging Primary and Transcript Cue Frames

from skills.watch.scripts.frames import merge_frames

primary = [...]          # frames from uniform or scene extraction

pinned = [...]           # frames generated from transcript timestamps

merged = merge_frames(primary, pinned)

# The merged list is chronological:

assert merged[0]["timestamp_seconds"] <= merged[1]["timestamp_seconds"]

Summary

• Timestamp calculation in uniform extraction uses frame index and FPS to determine temporal position with round(offset + (i / fps), 2) precision.
• Scene-change extraction parses exact pts_time values from ffmpeg's showinfo filter using regex matching for sub-second accuracy.
• Physical file sorting via sorted(out_dir.glob("frame_*.jpg")) ensures filesystem order matches chronological sequence before processing.
• Timestamp-based merging via merge_frames() re-indexes mixed frame sources using key=lambda f: f["timestamp_seconds"] to maintain temporal integrity across extraction methods.

Frequently Asked Questions

How does Claude Video handle timestamp accuracy when extracting frames at different FPS rates?

Claude Video calculates timestamps dynamically using the formula round(offset + (i / fps if fps > 0 else 0.0), 2), where i represents the frame index and fps the extraction rate. This ensures that regardless of the chosen frame rate, each timestamp accurately reflects its temporal position relative to the video start, rounded to two decimal places for consistency.

What is the difference between uniform extraction and scene-change extraction timestamps?

Uniform extraction computes timestamps mathematically from frame indices and FPS values, while scene-change extraction parses actual presentation timestamps (pts_time) from ffmpeg's internal clock. The latter method captures exact temporal positions where scene changes occur, providing higher precision for keyframe identification compared to calculated approximations.

Why does Claude Video sort frames both by filename and by timestamp?

The system first sorts physical files using sorted(out_dir.glob("frame_*.jpg")) to ensure filesystem order matches ffmpeg's output sequence, then uses timestamp values for merging operations. This dual approach prevents filesystem metadata inconsistencies from disrupting chronological order while enabling accurate interleaving of frames from different extraction sources.

How does the merge_frames function ensure chronological order when combining different frame sources?

The merge_frames() function combines primary and pinned frame lists, then re-sorts the unified collection using key=lambda f: f["timestamp_seconds"]. This timestamp-based re-indexing ensures that frames from uniform extraction, scene-change detection, and transcript cues interleave correctly according to their actual temporal positions in the source video, regardless of extraction method or original collection order.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}