# How the video_analysis_brief Artifact Grounds Reference-Video Requests in OpenMontage Production Plans

> Learn how the video_analysis_brief artifact grounds reference-video requests in OpenMontage production plans. Discover how structured metadata ensures deterministic video selections for improved production.

- Repository: [Calesthio/OpenMontage](https://github.com/calesthio/OpenMontage)
- Tags: deep-dive
- Published: 2026-08-30

---

**TLDR:** The `video_analysis_brief` artifact stores structured metadata—including exact video IDs, timestamps, and provider sources—generated by video understanding tools, which stage-director skills consume to build deterministic requests for video selectors, ensuring production plans reference concrete, pre-analyzed assets rather than ambiguous queries.

OpenMontage is an open-source framework for automated video production that orchestrates complex pipelines through discrete stages of analysis and generation. The **video_analysis_brief** artifact acts as the critical interface between these phases, grounding every reference-video request in concrete data extracted during the initial understanding pass.

## What Is the video_analysis_brief Artifact?

The `video_analysis_brief` artifact is a JSON document defined by the schema in [`schemas/artifacts/video_analysis_brief.schema.json`](https://github.com/calesthio/OpenMontage/blob/main/schemas/artifacts/video_analysis_brief.schema.json). It encapsulates the results of automated video analysis, including scene summaries, detected keyframes, and the **reference_video** object that enables deterministic video retrieval in downstream stages.

### Schema Structure and the reference_video Field

According to the schema definition, the brief contains a `reference_video` object with specific fields that identify an exact video segment:

- **source**: The video provider or catalog (e.g., "pexels")
- **id**: The unique identifier for the specific video asset
- **clip_start**: The start timestamp in seconds for the relevant segment
- **clip_end**: The end timestamp in seconds for the relevant segment
- **description**: A human-readable summary of the clip content

## How the Pipeline Grounds Requests Using the Brief

The grounding mechanism operates through a three-stage flow that transforms analyzed video metadata into executable production commands.

### Stage 1: Generation by Video Understanding Tools

Tools such as [`tools/video/video_understand.py`](https://github.com/calesthio/OpenMontage/blob/main/tools/video/video_understand.py) or provider-specific implementations like [`grok_video.py`](https://github.com/calesthio/OpenMontage/blob/main/grok_video.py) analyze raw footage and emit the `video_analysis_brief` artifact. During this phase, the tool populates the `reference_video` field with precise metadata extracted from the source material, ensuring the production plan captures exactly what was analyzed.

### Stage 2: Consumption by Stage-Director Skills

Stage-director skills, implemented as Markdown files under `skills/pipelines/`, retrieve the brief from the persistent store via [`lib/checkpoint.py`](https://github.com/calesthio/OpenMontage/blob/main/lib/checkpoint.py). The skill parses the JSON structure, extracts the reference video metadata, and constructs a tool-call payload that includes exact IDs and timestamps rather than search terms.

### Stage 3: Execution by Video Selector Tools

The [`tools/video/video_selector.py`](https://github.com/calesthio/OpenMontage/blob/main/tools/video/video_selector.py) module receives the grounded request containing the specific `reference_video_id`. It validates the reference against the provider catalog and routes the request to the appropriate backend—such as `pexels_video`—ensuring the production plan retrieves the exact clip specified in the analysis phase.

## Code Implementation Examples

The following Python snippet demonstrates how a stage-director skill reads the brief from the checkpoint system and prepares a selector request:

```python
import json
from pathlib import Path
from lib.checkpoint import Checkpoint

# Load the latest checkpoint for the current pipeline run

ckpt = Checkpoint.load_latest()

# The brief is stored under the "artifacts" key

video_brief = json.loads(ckpt.artifacts["video_analysis_brief"])

ref = video_brief["reference_video"]
payload = {
    "reference_video_id": ref["id"],
    "clip_start": ref["clip_start"],
    "clip_end": ref["clip_end"],
    "quality": "high",
}

# Ask the tool registry to invoke the selector

from lib.tool_registry import ToolRegistry
result = ToolRegistry.get_by_capability("video").execute(payload)

```

A typical `video_analysis_brief` JSON structure appears as follows:

```json
{
  "title": "video_analysis_brief",
  "reference_video": {
    "source": "pexels",
    "id": "9876543",
    "clip_start": 5.0,
    "clip_end": 18.2,
    "description": "Aerial view of a city at sunset"
  },
  "scene_summary": [
    {"scene": 1, "tags": ["city", "sunset"], "confidence": 0.94}
  ],
  "keyframes": ["frame_001.jpg", "frame_045.jpg"]
}

```

In skill definitions, the reference data is injected into tool calls using template syntax:

```markdown
{% tool video_selector %}
{
  "reference_video_id": "{{ video_brief.reference_video.id }}",
  "clip_start": {{ video_brief.reference_video.clip_start }},
  "clip_end": {{ video_brief.reference_video.clip_end }},
  "quality": "high"
}
{% endtool %}

```

## Key Source Files in the Architecture

The grounding mechanism relies on specific components across the OpenMontage codebase:

- [`schemas/artifacts/video_analysis_brief.schema.json`](https://github.com/calesthio/OpenMontage/blob/main/schemas/artifacts/video_analysis_brief.schema.json): Defines the JSON Schema for the artifact, including the `reference_video` object structure.
- `pipeline_defs/*.yaml`: Pipeline manifests that declare which stages produce or require the `video_analysis_brief` artifact.
- `skills/pipelines/**/video-selection-director.md`: Stage-director skill that reads the brief and formulates grounded selector requests.
- [`tools/video/video_selector.py`](https://github.com/calesthio/OpenMontage/blob/main/tools/video/video_selector.py): Implements routing logic for reference-video requests to appropriate providers.
- [`tools/video/video_understand.py`](https://github.com/calesthio/OpenMontage/blob/main/tools/video/video_understand.py): Generates the brief from raw video analysis.
- [`lib/checkpoint.py`](https://github.com/calesthio/OpenMontage/blob/main/lib/checkpoint.py): Persists artifacts between pipeline stages, enabling cross-stage data sharing.

## Summary

- The **video_analysis_brief** artifact stores deterministic metadata about reference videos extracted during the analysis phase.
- **Video understanding tools** such as [`video_understand.py`](https://github.com/calesthio/OpenMontage/blob/main/video_understand.py) populate the `reference_video` field with exact IDs and timestamps.
- **Stage-director skills** consume the brief via the checkpoint system to construct precise tool requests without ambiguity.
- The **video_selector** tool routes grounded requests to specific providers like Pexels, ensuring retrieval of the exact analyzed segment.
- This architecture guarantees **reproducibility** and **consistency** across complex video production pipelines.

## Frequently Asked Questions

### What fields are required in the reference_video object?

The `reference_video` object requires `source` (provider name), `id` (unique asset identifier), `clip_start`, and `clip_end` to define the temporal boundaries. An optional `description` field provides human-readable context about the video segment for debugging and documentation purposes.

### How does the checkpoint system persist artifacts between stages?

The [`lib/checkpoint.py`](https://github.com/calesthio/OpenMontage/blob/main/lib/checkpoint.py) module provides a persistent store where each pipeline stage writes its output artifacts. Subsequent stages call `Checkpoint.load_latest()` to access the `video_analysis_brief` and other artifacts, ensuring data flows deterministically through the pipeline without loss or mutation.

### Can multiple reference videos be specified in one brief?

While the core schema defines `reference_video` as a singular object, the architecture supports extending the brief to include arrays of references or generating multiple brief artifacts per pipeline run. The current implementation emphasizes single-reference grounding to ensure deterministic clip selection and avoid ambiguity in production plans.

### Which tools generate the video_analysis_brief artifact?

Primary generation occurs in [`tools/video/video_understand.py`](https://github.com/calesthio/OpenMontage/blob/main/tools/video/video_understand.py) and provider-specific tools such as [`grok_video.py`](https://github.com/calesthio/OpenMontage/blob/main/grok_video.py). These tools analyze raw footage, extract metadata including scene summaries and keyframes, and write the structured brief to the checkpoint system via the artifact registry for downstream consumption.