How the OpenMontage Documentary Montage Pipeline Builds a CLIP-Indexed Corpus from Free Stock Sources

The OpenMontage documentary montage pipeline constructs a searchable CLIP-indexed corpus through three tightly-coupled stages: adapter-based source discovery across free stock APIs, offline CLIP embedding of downloaded media with intelligent thumbnail extraction, and atomic persistence to JSONL and NumPy formats that enable fast cosine-similarity retrieval.

The OpenMontage repository provides an open-source toolkit for AI-assisted documentary filmmaking. Its documentary montage pipeline eliminates expensive licensing costs by targeting free-tier stock sources while maintaining professional search capabilities through vector indexing. The system transforms raw stock footage into a locally hosted, CLIP-indexed corpus that supports natural language video retrieval without repeated API calls.

Stage 1: Source Discovery and Search

The pipeline begins with the CorpusBuilder class, defined in tools/video/corpus_builder.py, which extends BaseTool to orchestrate the ingestion workflow. When CorpusBuilder.execute (lines 39‑48) is invoked, it enumerates available adapters from the tools.video.stock_sources module—supporting providers like Pexels, Unsplash, Wikimedia, Archive.org, and NASA that require no API key or accept free-tier access.

For each user-provided query, the builder calls source.search(query, filters) across selected adapters. This returns a normalized list of Candidate objects that abstract away API-specific response formats. The CorpusBuilder.get_status method (lines 103‑114) tracks progress across these distributed search operations, ensuring the pipeline can resume interrupted builds.

Stage 2: Download, Thumbnail Extraction, and CLIP Embedding

The _process_candidate method (lines 73‑122 in corpus_builder.py) handles the heavy lifting for each discovered asset. This stage operates entirely offline after the initial download, making the pipeline inexpensive and repeatable.

Cache-Aware Media Download

Before fetching new content, the pipeline checks clip_cache.get_default_cache (implemented in tools/video/clip_cache.py) for existing clip bytes. If the asset is absent, it falls back to src.download from the respective stock source adapter. This shared on-disk cache at ~/.openmontage/clips_cache/ prevents redundant downloads across multiple projects.

Thumbnail Generation

For video assets, _extract_video_thumbs (lines 42‑73) uses OpenCV to split videos into N evenly-spaced frames based on the thumbs_per_video parameter. Static images are processed through _save_as_jpeg to ensure consistent formatting. These JPEG thumbnails serve as the visual input for CLIP encoding.

Vector Encoding with CLIP

The lib/clip_embedder.py module lazy-loads the OpenAI CLIP model (openai/clip-vit-base-patch32) to generate embeddings. The pipeline calls embed_images (lines 78‑89) on the extracted thumbnails, producing L2-normalized 512-dimensional vectors. For text metadata, embed_texts (lines 94‑100) encodes source tags or the original search query.

For video candidates, the pool_frames function aggregates per-frame vectors into a single representative visual embedding. These pooled vectors, combined with tag embeddings, create a rich multimodal representation of each clip.

Stage 3: Corpus Persistence and Indexing

Once processed, each candidate becomes a ClipRecord containing provenance metadata—ID, source, dimensions, motion score, and local file path. The Corpus.add method (lines 90‑100 in lib/corpus.py) appends this record along with its 512-dim visual vector and tag vector to an in-memory index.

When ingestion completes or reaches max_new_clips, Corpus.save (lines 71‑81) performs an atomic write to the corpus_dir. The persistence layer generates three files:

• index.jsonl – Human-readable metadata records
• embeddings.npy – NumPy matrix of visual vectors
• tag_embeddings.npy – NumPy matrix of text vectors

The JSONL rows maintain one-to-one correspondence with the embedding matrix rows, enabling downstream tools to perform fast cosine similarity via simple matrix multiplication (embeddings @ query_vec). The Corpus class exposes retrieval methods including rank_by_text, knn, and find_similar_set for querying the compiled dataset.

Practical Implementation Examples

The following examples demonstrate building and querying a corpus using the OpenMontage pipeline:

# Build a corpus of 20 video clips about "ocean waves"

from tools.video.corpus_builder import CorpusBuilder

builder = CorpusBuilder()
result = builder.execute({
    "corpus_dir": "projects/ocean/corpus",
    "queries": [{"query": "ocean waves", "kind": "video", "per_source": 5}],
    "sources": ["pexels", "wikimedia", "archive_org"],  # free sources

    "max_new_clips": 20,
    "thumbs_per_video": 5,
})
print(result.success, result.data["clips_added"])

# Rank clips by textual prompt after corpus construction

from lib.corpus import Corpus
from lib.clip_embedder import embed_texts
from pathlib import Path

corp = Corpus(Path("projects/ocean/corpus"))
corp.load()
query_vec = embed_texts(["stormy seas at night"])[0]
top_hits = corp.rank_by_text(query_vec, k=10)

for rec, score in top_hits:
    print(rec.clip_id, score, rec.local_path)

Summary

• The CorpusBuilder tool in tools/video/corpus_builder.py orchestrates the entire ingestion workflow by coordinating multiple free stock source adapters.
• Offline processing occurs through _process_candidate, which handles download caching, OpenCV-based thumbnail extraction, and CLIP vector generation via lib/clip_embedder.py.
• The pipeline uses the openai/clip-vit-base-patch32 model to produce 512-dimensional L2-normalized embeddings for both visual frames and text metadata.
• Persistence uses an atomic write system producing JSONL metadata and NumPy matrix files (embeddings.npy, tag_embeddings.npy) that enable fast vector similarity search.
• All supported stock sources (Pexels, Wikimedia, Archive.org, NASA, etc.) offer free-tier access, making the pipeline cost-effective for documentary production.

Frequently Asked Questions

What free stock sources does the OpenMontage pipeline support?

The pipeline supports multiple adapters in tools/video/stock_sources/ including Pexels, Unsplash, Wikimedia Commons, Archive.org, and NASA imagery. Each adapter either requires no API key or accepts free-tier authentication, allowing users to build large corpora without licensing fees. The modular adapter architecture makes it straightforward to add new free sources by implementing the standard search and download interface.

How does the pipeline prevent redundant downloads across projects?

The clip_cache.py module maintains a shared on-disk cache at ~/.openmontage/clips_cache/. Before downloading, _process_candidate checks this cache via clip_cache.get_default_cache(); if the clip bytes exist locally, the pipeline skips the network request. This caching layer ensures that identical assets referenced in multiple documentary projects are downloaded only once, significantly reducing bandwidth and API quota consumption.

Which CLIP model version does the embedding system use?

According to lib/clip_embedder.py, the pipeline uses OpenAI's CLIP-ViT-B/32 model (openai/clip-vit-base-patch32). This model generates 512-dimensional L2-normalized vectors for both image and text inputs. The implementation lazy-loads the model weights to minimize memory overhead, and the pool_frames function aggregates multiple video frames into a single representative vector for efficient storage and retrieval.

How is the corpus structured to enable fast similarity search?

The Corpus.save method in lib/corpus.py (lines 71‑81) writes two NumPy .npy files—embeddings.npy and tag_embeddings.npy—alongside a JSONL metadata file. The row indices align across all three files, allowing downstream tools to compute cosine similarities using efficient matrix multiplication (embeddings @ query_vec) without parsing JSON. This structure supports millisecond-scale retrieval via methods like rank_by_text and knn defined in the Corpus class.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}