# How Native Caption Extraction Prioritizes Manual Over Auto-Generated Subtitles in Claude Video

> Learn how Claude Video's native caption extraction prioritizes manual subtitles over auto-generated ones. Understand the download and sorting logic for optimal subtitle selection.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: deep-dive
- Published: 2026-08-06

---

**The watch skill prioritizes manual subtitles by downloading them first with `--write-subs`, then using alphabetical sorting in `_pick_subtitle` to select `video.en.vtt` over `video.en.auto.vtt` before falling back to Whisper transcription.**

The `bradautomates/claude-video` repository implements a deterministic two-stage pipeline to ensure high-quality manual captions are preferred over auto-generated alternatives. This native caption extraction strategy leverages specific yt-dlp flags and filename-based sorting logic in [`skills/watch/scripts/download.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/download.py) to guarantee the best available subtitle source is always selected.

## Two-Stage Download Strategy

The prioritization happens through a coordinated download and selection process that guarantees manual subtitles take precedence whenever they exist.

### Stage 1: yt-dlp Download Flags

The `fetch_captions` function constructs a command list that downloads manual subtitles **before** auto-generated ones. According to the source code on lines 71-80 of [`skills/watch/scripts/download.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/download.py), the function runs yt-dlp with:

- `--write-subs` to download any **manual** subtitles first
- `--write-auto-subs` to download **auto-generated** subtitles second

This ordering ensures that if both types exist for a video, both files are written to the output directory. The function builds the command programmatically, appending the manual flag before the auto flag to establish the priority at the download level.

### Stage 2: Subtitle Selection Logic

After the download completes, the helper function `_pick_subtile` (lines 44-53 of the same file) handles the selection. It implements a deterministic sorting algorithm:

1. **Scans** the output folder for all VTT files
2. **Sorts** filenames alphabetically
3. **Builds** a preferred list containing English language markers (e.g., `.en.`, `.en-US.`, `.en-GB.`, `.en-orig.`)

Because manual caption filenames typically follow the pattern `video.en.vtt` while auto-generated files use `video.en.auto.vtt`, the alphabetical sort places the manual file **before** the auto file. The function returns the first entry of the preferred list, ensuring the manual subtitle is selected whenever present.

## Implementation Details in download.py

The prioritization logic is implemented through specific function calls and filename patterns. Here is how to use the watch skill programmatically:

```python

# Example: Using the watch skill to fetch captions from a YouTube URL

from skills.watch.scripts.download import fetch_captions
from pathlib import Path

url = "https://www.youtube.com/watch?v=abc123"
out_dir = Path("/tmp/video_meta")
result = fetch_captions(url, out_dir)

print("Subtitle chosen:", result["subtitle_path"])

# → Points to the manual VTT file if it existed, otherwise the auto‑generated one.

```

Under the hood, the Python code executes this yt-dlp command:

```bash

# Direct command-line equivalent (what the Python code runs under the hood)

yt-dlp \
  --skip-download \
  --write-info-json \
  --write-subs \
  --write-auto-subs \
  --sub-langs "en.*" \
  --sub-format vtt \
  --convert-subs vtt \
  -o "/tmp/video_meta/video.%(ext)s" \
  "https://www.youtube.com/watch?v=abc123"

```

The `--write-subs` flag precedes `--write-auto-subs` in the argument list, ensuring manual captions are requested first in the yt-dlp execution order.

## Fallback Chain and Whisper Integration

If no manual subtitle is available, the auto-generated file becomes the selected subtitle automatically. This guarantees that `fetch_captions` always returns a VTT caption source when any subtitles exist. Only when both manual and auto-generated subtitles are absent does the system fall back to Whisper transcription, as orchestrated in [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py).

The test suite in [`tests/test_download.py`](https://github.com/bradautomates/claude-video/blob/main/tests/test_download.py) verifies this prioritization behavior, ensuring that manual subtitles are consistently preferred over auto-generated alternatives across different video sources and language configurations.

## Summary

- **Download order matters**: The `fetch_captions` function passes `--write-subs` before `--write-auto-subs` to yt-dlp, ensuring both files are retrieved but manual sources are written first.
- **Alphabetical sorting decides**: The `_pick_subtitle` function sorts VTT filenames alphabetically, causing `video.en.vtt` to rank higher than `video.en.auto.vtt`.
- **English language filtering**: The selection logic specifically targets English markers (`.en.`, `.en-US.`, `.en-GB.`, `.en-orig.`) to identify relevant subtitle files.
- **Deterministic fallback**: If manual subtitles are unavailable, auto-generated captions are used automatically; only if neither exists does the system fall back to Whisper.

## Frequently Asked Questions

### What happens if only auto-generated subtitles exist?

If no manual subtitles are available for a video, the `_pick_subtitle` function will still find and return the auto-generated VTT file (e.g., `video.en.auto.vtt`) because it matches the English language filters and is the first (and only) valid option in the sorted list. The watch skill then proceeds with transcription using these auto-generated captions rather than invoking Whisper.

### How does the alphabetical sorting work exactly?

The function collects all VTT files in the output directory and sorts them using standard string comparison. Since `"video.en.vtt"` comes before `"video.en.auto.vtt"` alphabetically (the dot after `en` precedes the letter `a` in `auto`), the manual subtitle file naturally appears first in the list. The function then filters for English language markers and returns the first match, guaranteeing manual precedence.

### Can I modify the language preference from English?

The current implementation in [`skills/watch/scripts/download.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/download.py) hardcodes English language markers (`.en.`, `.en-US.`, `.en-GB.`, `.en-orig.`) in the `_pick_subtitle` function. To support other languages, you would need to modify the regular expression or string matching logic on lines 44-53 to include your target language codes (e.g., `.es.` for Spanish or `.fr.` for French).

### Where is the manual-over-auto logic tested?

The prioritization logic is verified in [`tests/test_download.py`](https://github.com/bradautomates/claude-video/blob/main/tests/test_download.py), which contains test cases ensuring that when both manual and auto-generated subtitle files exist in the output directory, the function consistently returns the manual subtitle path. These tests validate the alphabetical sorting and English-filtering behavior used by `_pick_subtitle`.