How Claude-video's /watch Endpoint Handles transcript, efficient, balanced, and token-burner Detail Modes

Claude-video's /watch endpoint routes video processing through four distinct detail modes—transcript, efficient, balanced, and token-burner—that determine frame extraction budgets, algorithm selection, and token usage based on a hierarchical configuration system.

Claude-video is an open-source video analysis tool that processes video content through a flexible detail mode system. The /watch endpoint in the bradautomates/claude-video repository allows users to control frame extraction and transcription behavior through four distinct modes. Understanding how these modes resolve and their specific frame budgets helps optimize token usage and processing time for different video analysis tasks.

Detail Mode Configuration Hierarchy

The detail mode follows a strict resolution order defined in skills/watch/scripts/config.py【/cache/repos/github.com/bradautomates/claude-video/main/skills/watch/scripts/config.py#L12-L60】. The system evaluates potential sources in priority order, falling back to subsequent options only when higher-priority values are absent.

Resolution Order

The mode selection follows this hierarchy:

1. CLI flag --detail (highest priority)
2. Environment variable WATCH_DETAIL (or values in ~/.config/watch/.env)
3. Default value "balanced" (fallback)

The get_config() function implements this logic:

DEFAULT_DETAIL = "balanced"
DETAILS = {"transcript", "efficient", "balanced", "token-burner"}

def get_config() -> dict:
    detail = (
        os.environ.get("WATCH_DETAIL")
        or file_values.get("WATCH_DETAIL")
        or DEFAULT_DETAIL
    )
    if detail not in DETAILS:
        detail = DEFAULT_DETAIL
    return {"detail": detail, "config_file": str(CONFIG_FILE)}

Configuration Validation

If the resolved value does not exist in the DETAILS set, the system safely defaults to "balanced". This validation prevents invalid modes from crashing the pipeline while ensuring predictable behavior.

Frame Budget and Extraction Algorithms

The frame_cap() function in skills/watch/scripts/config.py【/cache/repos/github.com/bradautomates/claude-video/main/skills/watch/scripts/config.py#L65-L74】 maps each mode to a numeric cap or None for uncapped extraction:

• transcript: None (no frames unless timestamps provided)
• efficient: 50 (fast keyframe extraction)
• balanced: 100 (scene-aware extraction)
• token-burner: None (uncapped scene-aware extraction)

Transcript Mode

When detail="transcript", the pipeline skips frame extraction entirely unless the user supplies explicit timestamps via --timestamps. The system sets audio_only = True to avoid downloading video streams unnecessarily. This mode prioritizes minimal token usage by processing only the spoken content through either existing subtitles or Whisper API transcription.

Efficient Mode

The efficient mode caps frame extraction at 50 frames and invokes the extract_keyframes() function. This algorithm uses fast keyframe detection to provide quick visual overviews with minimal processing time. This mode suits rapid content scanning where scene-level detail is unnecessary.

Balanced Mode

As the default configuration, balanced mode allocates a 100-frame budget and utilizes extract_scene_or_uniform(). This scene-aware algorithm attempts to capture representative frames across video segments while respecting the hard cap. The system intelligently distributes frames across the timeline to maximize coverage within the 100-frame limit.

Token-burner Mode

The token-burner mode removes the frame cap entirely (None), allowing extract_scene_or_uniform() to generate as many frames as the scene-change detector identifies. This can produce hundreds of frames for long videos, significantly increasing token consumption. The system emits a warning when processing generates more than 250 frames【/cache/repos/github.com/bradautomates/claude-video/main/skills/watch/scripts/watch.py#L19-L24】 to alert users about potential cost implications.

Execution Flow in watch.py

The main orchestration logic in skills/watch/scripts/watch.py【/cache/repos/github.com/bradautomates/claude-video/main/skills/watch/scripts/watch.py#L71-L73】 implements the mode-specific processing pipeline.

Frame Budget Calculation

After resolving the detail mode, the script calculates the effective detail_budget by accounting for any cue frames requested via --timestamps【/cache/repos/github.com/bradautomates/claude-video/main/skills/watch/scripts/watch.py#L95-L99】:

# Subtract cue frames from the budget

if detail == "transcript":
    detail_budget = 0  # Unless timestamps provided

else:
    detail_budget = frame_cap(detail)

Engine Selection Logic

The script selects extraction engines based on the resolved mode【/cache/repos/github.com/bradautomates/claude-video/main/skills/watch/scripts/watch.py#L95-L105】【/cache/repos/github.com/bradautomates/claude-video/main/skills/watch/scripts/watch.py#L190-L206】:

• detail == "efficient" → Calls extract_keyframes() for fast keyframe extraction
• detail in {"balanced", "token-burner"} → Calls extract_scene_or_uniform() for scene-aware processing
• detail == "transcript" → Skips frame extraction unless cue timestamps exist

When timestamps are provided alongside transcript mode, those specific frames are extracted as cue frames and appear in the final report despite the transcript setting【/cache/repos/github.com/bradautomates/claude-video/main/skills/watch/scripts/watch.py#L76-L84】.

Transcript Processing Pipeline

The transcript handling operates independently of frame extraction, utilizing two data sources:

1. Existing subtitles: If dl["subtitle_path"] exists, the system parses VTT files using parse_vtt()
2. Whisper API: When no subtitles exist and --no-whisper is not set, the script calls transcribe_video() to generate transcripts

In transcript-only mode, the final report displays "Frames: skipped (transcript detail)" while showing the transcript segment count and source (captions or Whisper)【/cache/repos/github.com/bradautomates/claude-video/main/skills/watch/scripts/watch.py#L88-L99】.

Practical Usage Examples

The following commands demonstrate each detail mode:

# Fast keyframe extraction (max 50 frames)

watch https://youtu.be/xyz --detail efficient

# Scene-aware extraction with 100-frame cap

watch https://youtu.be/xyz --detail balanced

# Uncapped scene extraction (high token usage)

watch https://youtu.be/xyz --detail token-burner

# Transcript only (no frames)

watch https://youtu.be/xyz --detail transcript

# Transcript with specific cue frames

watch https://youtu.be/xyz --detail transcript --timestamps "0:30,1:45"

Summary

• Configuration hierarchy: CLI flag --detail takes precedence over WATCH_DETAIL environment variable, with "balanced" as the ultimate fallback.
• Frame budgets: transcript (0), efficient (50), balanced (100), token-burner (uncapped).
• Algorithm selection: efficient uses extract_keyframes(); balanced and token-burner use extract_scene_or_uniform(); transcript skips frames unless timestamps are provided.
• Token warnings: The system warns when token-burner mode generates more than 250 frames or when capped modes produce sparse coverage on long videos.
• Source files: Mode resolution occurs in skills/watch/scripts/config.py while execution logic resides in skills/watch/scripts/watch.py.

Frequently Asked Questions

How does Claude-video choose which detail mode to use when I don't specify one?

The system follows a strict hierarchy: it first checks for the --detail CLI flag, then falls back to the WATCH_DETAIL environment variable (or the value in ~/.config/watch/.env), and finally defaults to "balanced" if neither is set. This ensures consistent behavior across different execution environments while allowing easy overrides.

What's the difference between efficient and balanced modes in terms of processing?

Efficient mode extracts up to 50 frames using fast keyframe detection, prioritizing speed over scene coverage. Balanced mode uses scene-aware extraction (extract_scene_or_uniform()) with a 100-frame cap, attempting to distribute frames evenly across the video timeline. The balanced algorithm provides better visual representation but requires more processing time to analyze scene changes.

Can I extract specific frames while still using transcript mode?

Yes. When you provide --timestamps alongside --detail transcript, the system extracts those specific timestamps as cue frames while still skipping the general frame extraction pipeline. This allows you to reference specific visual moments in a video while primarily consuming the transcript content, optimizing for both precision and token efficiency.

Why does token-burner mode warn about frame counts?

The token-burner mode removes the frame cap entirely, allowing the scene-detection algorithm to generate as many frames as it finds relevant. For long videos with frequent scene changes, this can produce hundreds of frames, significantly increasing API token consumption and costs. The warning at 250 frames serves as a safeguard to alert users before they incur unexpected expenses from uncapped frame extraction.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}