# How OpenMontage Analyzes YouTube Reference Videos to Create Production Plans: A Technical Deep Dive

> Discover how OpenMontage analyzes YouTube reference videos, extracting data from platform detection to motion classification, to generate structured production blueprints.

- Repository: [Calesthio/OpenMontage](https://github.com/calesthio/OpenMontage)
- Tags: deep-dive
- Published: 2026-08-30

---

**OpenMontage converts YouTube videos into structured production blueprints by orchestrating nine analytical stages—from platform detection and transcript extraction to scene segmentation and motion classification—all encapsulated in a machine-readable `VideoAnalysisBrief`.**

The OpenMontage project provides an automated pipeline for reverse-engineering video content into actionable production specifications. By analyzing reference videos from YouTube, the system generates detailed `VideoAnalysisBrief` objects that inform downstream creative agents about content structure, visual style, and technical requirements.

## Platform Detection and Media Ingestion

The analysis pipeline begins in [`tools/analysis/video_analyzer.py`](https://github.com/calesthio/OpenMontage/blob/main/tools/analysis/video_analyzer.py) with the **`VideoAnalyzer`** orchestrator class. The system first validates the input source through **`_is_url`** and classifies the platform via **`_detect_platform`** (lines 31‑44). This method recognizes standard YouTube URLs, Shorts links, and local file paths, returning identifiers such as `youtube`, `shorts`, or `local`.

Once classified as a remote URL, the **`VideoDownloader`** tool (invoked at lines 98‑115) retrieves the media using **yt‑dlp**. The downloader operates in two modes: for "transcript‑only" analysis, it fetches only metadata (title, duration, resolution, uploader info), while deeper analysis modes download the video at 720p maximum resolution and extract a clean audio track. All metadata populates the initial `VideoAnalysisBrief` structure.

## Transcript Acquisition via Dual-Path Strategy

OpenMontage employs a cascading transcription strategy to maximize accuracy while minimizing processing overhead. The primary path leverages **`TranscriptFetcher`** (lines 60‑90) to query YouTube's caption infrastructure via the **youtube‑transcript‑api** library. The tool automatically discovers available language tracks and prefers human‑generated captions, falling back to auto‑generated alternatives when necessary.

If the YouTube API fails or the platform lacks captions, the system executes a fallback to **`Transcriber`** (lines 124‑142). This component utilizes **faster‑whisper** to perform on‑device speech‑to‑text conversion on the previously extracted audio track. This dual‑path approach ensures that every reference video yields a complete `narration_transcript` field regardless of platform caption availability.

## Structural Analysis: Scene Detection and Keyframe Extraction

With media files secured locally, the pipeline proceeds to structural decomposition. The **`SceneDetect`** tool (lines 181‑190) utilizes **PySceneDetect** with content‑based detection to identify logical scene boundaries. The analyzer splits the video into discrete segments, recording start/end timestamps and scene indices in `brief["structure_analysis"]["scenes"]`.

Following scene detection, **`FrameSampler`** (lines 244‑259) extracts representative **keyframes** at scene boundaries, mid‑points, and—during "deep" analysis—additional intra‑scene timestamps. These JPEG outputs, stored with their corresponding scene indices, provide visual references for downstream vision models tasked with generating shot descriptions.

## Motion Classification and Visual Profiling

Each detected scene undergoes motion analysis via **`_classify_scene_motion`** (lines 690‑780). The method samples frame pairs within the scene and computes dense optical flow using the Farneback algorithm. The resulting flow patterns categorize scenes into three production‑relevant types:

- **static_image**: Low optical flow indicates still imagery or slides
- **animated_still**: Uniform flow suggests Ken Burns‑style motion or slow pans
- **motion_clip**: Heterogeneous flow signals dynamic camera movement or subject motion

This classification directly impacts pipeline recommendations, determining whether the production plan requires motion‑heavy rendering capabilities.

## Audio Profiling and Style Synthesis

Parallel to visual analysis, the **`AudioEnergy`** tool (lines 501‑510) analyzes the extracted audio waveform to build an `audio_energy_profile`. The tool identifies loudness peaks, dynamic range characteristics, and recommended timing offsets for cut points.

The orchestrator then synthesizes these discrete analyses into a cohesive **`style_profile`**. Helper methods **`_suggest_pipeline`** and **`_estimate_complexity`** (lines 556‑566) analyze transcript word count (calculating words‑per‑minute), scene duration averages, motion ratios, and audio characteristics to recommend specific production pipelines such as *animation*, *cinematic*, or *animated‑explainer*, along with estimated complexity tiers.

## Output Generation: The VideoAnalysisBrief

The final stage persists the enriched data structure via **`_save_brief`** (lines 792‑799). The completed `VideoAnalysisBrief` contains:

- Full transcript text and metadata
- Scene structure with timestamps and motion classifications
- Extracted keyframe file references
- Audio energy profiles
- Pipeline suggestions and complexity estimates

This JSON artifact and its associated keyframe directory provide downstream agents in the calesthio/OpenMontage ecosystem with a structured, machine‑readable representation of the reference video's content, structure, and production requirements.

## Practical Implementation

The following examples demonstrate the `VideoAnalyzer` interface for different analysis depths:

```python
from tools.analysis.video_analyzer import VideoAnalyzer

# Example 1: Fast transcription-only analysis (no video download)

result = VideoAnalyzer().execute({
    "source": "https://www.youtube.com/watch?v=abcdef12345",
    "analysis_depth": "transcript_only",
})
print(result.data["narration_transcript"]["full_text"])

```

```python

# Example 2: Deep analysis for short-form content

result = VideoAnalyzer().execute({
    "source": "https://youtu.be/xyz987",
    "analysis_depth": "deep",
    "max_keyframes": 30,
})
brief = result.data
print("Suggested pipeline:", brief["replication_guidance"]["suggested_pipeline"])
print("Scene count:", brief["structure_analysis"]["total_scenes"])
print("Keyframes saved:", len(brief["keyframes"]))

```

## Summary

- **OpenMontage's `VideoAnalyzer`** orchestrates a nine-stage pipeline to deconstruct YouTube videos into production-ready specifications.
- The system uses **platform detection** (`_detect_platform`) and **yt‑dlp**-based downloading to handle diverse YouTube formats including Shorts.
- **Dual-path transcription** leverages `youtube‑transcript‑api` for speed, falling back to **faster‑whisper** for audio-based transcription when captions are unavailable.
- **PySceneDetect** and **optical flow analysis** (`_classify_scene_motion`) segment videos and classify motion types to inform pipeline selection.
- The **`VideoAnalysisBrief`** JSON output encapsulates transcripts, scenes, keyframes, and style recommendations for downstream automation.

## Frequently Asked Questions

### What analysis depths does OpenMontage support for YouTube videos?

OpenMontage supports `"transcript_only"` for rapid text extraction without downloading video files, and `"deep"` for comprehensive analysis including scene detection, keyframe extraction, and motion classification. The depth parameter is passed directly to `VideoAnalyzer.execute()` and controls whether the `VideoDownloader` fetches full 720p media or just metadata.

### How does OpenMontage handle YouTube videos without captions?

When the `TranscriptFetcher` fails to retrieve YouTube captions via the youtube‑transcript‑api (lines 60‑90), the system automatically falls back to the `Transcriber` class (lines 124‑142). This component uses faster‑whisper to perform local speech‑to‑text conversion on the audio track extracted by `VideoDownloader`, ensuring transcript availability regardless of platform caption status.

### What motion types can OpenMontage detect in reference videos?

The `_classify_scene_motion` method (lines 690‑780) categorizes each scene into **static_image** (minimal movement), **animated_still** (uniform motion such as pans or zooms), or **motion_clip** (complex heterogeneous movement). The classification relies on Farneback optical flow calculations sampled across scene frames and directly influences whether the suggested production pipeline requires motion graphics or video editing capabilities.

### Is OpenMontage's video analysis pipeline completely local?

Yes. According to the calesthio/OpenMontage source code, all analytical stages—including whisper transcription via faster‑whisper, scene detection via PySceneDetect, and optical flow processing—execute locally without requiring external API keys. The only network dependencies are yt‑dlp for YouTube media retrieval and the youtube‑transcript‑api for caption fetching, both of which interact directly with YouTube's public infrastructure.