How the video_analysis_brief Artifact Grounds Reference-Video Requests in OpenMontage Production Plans

TLDR: The video_analysis_brief artifact stores structured metadata—including exact video IDs, timestamps, and provider sources—generated by video understanding tools, which stage-director skills consume to build deterministic requests for video selectors, ensuring production plans reference concrete, pre-analyzed assets rather than ambiguous queries.

OpenMontage is an open-source framework for automated video production that orchestrates complex pipelines through discrete stages of analysis and generation. The video_analysis_brief artifact acts as the critical interface between these phases, grounding every reference-video request in concrete data extracted during the initial understanding pass.

What Is the video_analysis_brief Artifact?

The video_analysis_brief artifact is a JSON document defined by the schema in schemas/artifacts/video_analysis_brief.schema.json. It encapsulates the results of automated video analysis, including scene summaries, detected keyframes, and the reference_video object that enables deterministic video retrieval in downstream stages.

Schema Structure and the reference_video Field

According to the schema definition, the brief contains a reference_video object with specific fields that identify an exact video segment:

• source: The video provider or catalog (e.g., "pexels")
• id: The unique identifier for the specific video asset
• clip_start: The start timestamp in seconds for the relevant segment
• clip_end: The end timestamp in seconds for the relevant segment
• description: A human-readable summary of the clip content

How the Pipeline Grounds Requests Using the Brief

The grounding mechanism operates through a three-stage flow that transforms analyzed video metadata into executable production commands.

Stage 1: Generation by Video Understanding Tools

Tools such as tools/video/video_understand.py or provider-specific implementations like grok_video.py analyze raw footage and emit the video_analysis_brief artifact. During this phase, the tool populates the reference_video field with precise metadata extracted from the source material, ensuring the production plan captures exactly what was analyzed.

Stage 2: Consumption by Stage-Director Skills

Stage-director skills, implemented as Markdown files under skills/pipelines/, retrieve the brief from the persistent store via lib/checkpoint.py. The skill parses the JSON structure, extracts the reference video metadata, and constructs a tool-call payload that includes exact IDs and timestamps rather than search terms.

Stage 3: Execution by Video Selector Tools

The tools/video/video_selector.py module receives the grounded request containing the specific reference_video_id. It validates the reference against the provider catalog and routes the request to the appropriate backend—such as pexels_video—ensuring the production plan retrieves the exact clip specified in the analysis phase.

Code Implementation Examples

The following Python snippet demonstrates how a stage-director skill reads the brief from the checkpoint system and prepares a selector request:

import json
from pathlib import Path
from lib.checkpoint import Checkpoint

# Load the latest checkpoint for the current pipeline run

ckpt = Checkpoint.load_latest()

# The brief is stored under the "artifacts" key

video_brief = json.loads(ckpt.artifacts["video_analysis_brief"])

ref = video_brief["reference_video"]
payload = {
    "reference_video_id": ref["id"],
    "clip_start": ref["clip_start"],
    "clip_end": ref["clip_end"],
    "quality": "high",
}

# Ask the tool registry to invoke the selector

from lib.tool_registry import ToolRegistry
result = ToolRegistry.get_by_capability("video").execute(payload)

A typical video_analysis_brief JSON structure appears as follows:

{
  "title": "video_analysis_brief",
  "reference_video": {
    "source": "pexels",
    "id": "9876543",
    "clip_start": 5.0,
    "clip_end": 18.2,
    "description": "Aerial view of a city at sunset"
  },
  "scene_summary": [
    {"scene": 1, "tags": ["city", "sunset"], "confidence": 0.94}
  ],
  "keyframes": ["frame_001.jpg", "frame_045.jpg"]
}

In skill definitions, the reference data is injected into tool calls using template syntax:

{% tool video_selector %}
{
  "reference_video_id": "{{ video_brief.reference_video.id }}",
  "clip_start": {{ video_brief.reference_video.clip_start }},
  "clip_end": {{ video_brief.reference_video.clip_end }},
  "quality": "high"
}
{% endtool %}

Key Source Files in the Architecture

The grounding mechanism relies on specific components across the OpenMontage codebase:

• schemas/artifacts/video_analysis_brief.schema.json: Defines the JSON Schema for the artifact, including the reference_video object structure.
• pipeline_defs/*.yaml: Pipeline manifests that declare which stages produce or require the video_analysis_brief artifact.
• skills/pipelines/**/video-selection-director.md: Stage-director skill that reads the brief and formulates grounded selector requests.
• tools/video/video_selector.py: Implements routing logic for reference-video requests to appropriate providers.
• tools/video/video_understand.py: Generates the brief from raw video analysis.
• lib/checkpoint.py: Persists artifacts between pipeline stages, enabling cross-stage data sharing.

Summary

• The video_analysis_brief artifact stores deterministic metadata about reference videos extracted during the analysis phase.
• Video understanding tools such as video_understand.py populate the reference_video field with exact IDs and timestamps.
• Stage-director skills consume the brief via the checkpoint system to construct precise tool requests without ambiguity.
• The video_selector tool routes grounded requests to specific providers like Pexels, ensuring retrieval of the exact analyzed segment.
• This architecture guarantees reproducibility and consistency across complex video production pipelines.

Frequently Asked Questions

What fields are required in the reference_video object?

The reference_video object requires source (provider name), id (unique asset identifier), clip_start, and clip_end to define the temporal boundaries. An optional description field provides human-readable context about the video segment for debugging and documentation purposes.

How does the checkpoint system persist artifacts between stages?

The lib/checkpoint.py module provides a persistent store where each pipeline stage writes its output artifacts. Subsequent stages call Checkpoint.load_latest() to access the video_analysis_brief and other artifacts, ensuring data flows deterministically through the pipeline without loss or mutation.

Can multiple reference videos be specified in one brief?

While the core schema defines reference_video as a singular object, the architecture supports extending the brief to include arrays of references or generating multiple brief artifacts per pipeline run. The current implementation emphasizes single-reference grounding to ensure deterministic clip selection and avoid ambiguity in production plans.

Which tools generate the video_analysis_brief artifact?

Primary generation occurs in tools/video/video_understand.py and provider-specific tools such as grok_video.py. These tools analyze raw footage, extract metadata including scene summaries and keyframes, and write the structured brief to the checkpoint system via the artifact registry for downstream consumption.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}