How Native Caption Extraction Prioritizes Manual Over Auto-Generated Subtitles in Claude Video

The watch skill prioritizes manual subtitles by downloading them first with --write-subs, then using alphabetical sorting in _pick_subtitle to select video.en.vtt over video.en.auto.vtt before falling back to Whisper transcription.

The bradautomates/claude-video repository implements a deterministic two-stage pipeline to ensure high-quality manual captions are preferred over auto-generated alternatives. This native caption extraction strategy leverages specific yt-dlp flags and filename-based sorting logic in skills/watch/scripts/download.py to guarantee the best available subtitle source is always selected.

Two-Stage Download Strategy

The prioritization happens through a coordinated download and selection process that guarantees manual subtitles take precedence whenever they exist.

Stage 1: yt-dlp Download Flags

The fetch_captions function constructs a command list that downloads manual subtitles before auto-generated ones. According to the source code on lines 71-80 of skills/watch/scripts/download.py, the function runs yt-dlp with:

• --write-subs to download any manual subtitles first
• --write-auto-subs to download auto-generated subtitles second

This ordering ensures that if both types exist for a video, both files are written to the output directory. The function builds the command programmatically, appending the manual flag before the auto flag to establish the priority at the download level.

Stage 2: Subtitle Selection Logic

After the download completes, the helper function _pick_subtile (lines 44-53 of the same file) handles the selection. It implements a deterministic sorting algorithm:

1. Scans the output folder for all VTT files
2. Sorts filenames alphabetically
3. Builds a preferred list containing English language markers (e.g., .en., .en-US., .en-GB., .en-orig.)

Because manual caption filenames typically follow the pattern video.en.vtt while auto-generated files use video.en.auto.vtt, the alphabetical sort places the manual file before the auto file. The function returns the first entry of the preferred list, ensuring the manual subtitle is selected whenever present.

Implementation Details in download.py

The prioritization logic is implemented through specific function calls and filename patterns. Here is how to use the watch skill programmatically:

# Example: Using the watch skill to fetch captions from a YouTube URL

from skills.watch.scripts.download import fetch_captions
from pathlib import Path

url = "https://www.youtube.com/watch?v=abc123"
out_dir = Path("/tmp/video_meta")
result = fetch_captions(url, out_dir)

print("Subtitle chosen:", result["subtitle_path"])

# → Points to the manual VTT file if it existed, otherwise the auto‑generated one.

Under the hood, the Python code executes this yt-dlp command:

# Direct command-line equivalent (what the Python code runs under the hood)

yt-dlp \
  --skip-download \
  --write-info-json \
  --write-subs \
  --write-auto-subs \
  --sub-langs "en.*" \
  --sub-format vtt \
  --convert-subs vtt \
  -o "/tmp/video_meta/video.%(ext)s" \
  "https://www.youtube.com/watch?v=abc123"

The --write-subs flag precedes --write-auto-subs in the argument list, ensuring manual captions are requested first in the yt-dlp execution order.

Fallback Chain and Whisper Integration

If no manual subtitle is available, the auto-generated file becomes the selected subtitle automatically. This guarantees that fetch_captions always returns a VTT caption source when any subtitles exist. Only when both manual and auto-generated subtitles are absent does the system fall back to Whisper transcription, as orchestrated in skills/watch/scripts/watch.py.

The test suite in tests/test_download.py verifies this prioritization behavior, ensuring that manual subtitles are consistently preferred over auto-generated alternatives across different video sources and language configurations.

Summary

• Download order matters: The fetch_captions function passes --write-subs before --write-auto-subs to yt-dlp, ensuring both files are retrieved but manual sources are written first.
• Alphabetical sorting decides: The _pick_subtitle function sorts VTT filenames alphabetically, causing video.en.vtt to rank higher than video.en.auto.vtt.
• English language filtering: The selection logic specifically targets English markers (.en., .en-US., .en-GB., .en-orig.) to identify relevant subtitle files.
• Deterministic fallback: If manual subtitles are unavailable, auto-generated captions are used automatically; only if neither exists does the system fall back to Whisper.

Frequently Asked Questions

What happens if only auto-generated subtitles exist?

If no manual subtitles are available for a video, the _pick_subtitle function will still find and return the auto-generated VTT file (e.g., video.en.auto.vtt) because it matches the English language filters and is the first (and only) valid option in the sorted list. The watch skill then proceeds with transcription using these auto-generated captions rather than invoking Whisper.

How does the alphabetical sorting work exactly?

The function collects all VTT files in the output directory and sorts them using standard string comparison. Since "video.en.vtt" comes before "video.en.auto.vtt" alphabetically (the dot after en precedes the letter a in auto), the manual subtitle file naturally appears first in the list. The function then filters for English language markers and returns the first match, guaranteeing manual precedence.

Can I modify the language preference from English?

The current implementation in skills/watch/scripts/download.py hardcodes English language markers (.en., .en-US., .en-GB., .en-orig.) in the _pick_subtitle function. To support other languages, you would need to modify the regular expression or string matching logic on lines 44-53 to include your target language codes (e.g., .es. for Spanish or .fr. for French).

Where is the manual-over-auto logic tested?

The prioritization logic is verified in tests/test_download.py, which contains test cases ensuring that when both manual and auto-generated subtitle files exist in the output directory, the function consistently returns the manual subtitle path. These tests validate the alphabetical sorting and English-filtering behavior used by _pick_subtitle.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}