How OpenMontage's 7-Dimension Provider Scoring Engine Ranks AI Video Tools

OpenMontage uses a deterministic, multi-dimensional scoring system defined in lib/scoring.py that evaluates AI providers across seven weighted dimensions—task fit, output quality, control, reliability, cost efficiency, latency, and continuity—to rank tools like video_selector and select the optimal provider for each generation task.

The OpenMontage framework employs a sophisticated 7-dimension provider scoring engine to intelligently route generation tasks to the most suitable AI provider. This deterministic system, implemented in lib/scoring.py, evaluates candidate tools across multiple quality and operational metrics, ensuring optimal provider selection for capabilities like video generation.

The Seven Dimensions of Provider Evaluation

The scoring engine evaluates every candidate provider across seven distinct dimensions, each returning a normalized float between 0 and 1. These dimensions capture both technical capability and operational fitness for the specific task context.

Task Fit and Output Quality

Task fit measures how closely a provider’s advertised best_for attributes match the user’s intent and style requirements. The _compute_task_fit function calculates this using keyword overlap and synonym expansion against the task description.

Output quality represents the expected fidelity of the generated asset. This dimension derives its score from either historic quality_score data or the provider’s stability tier, where production tiers map to 0.9, beta to 0.7, and similar gradations.

Control and Reliability

Control quantifies the amount of creative control the provider exposes through features like controlnet, reference_image support, and other fine-tuning parameters. The _compute_control function weights these feature flags to produce the final score.

Reliability indicates the probability that the tool will successfully complete the request. This score draws from historic success rates or current operational status, calculated within the score_provider function (lines 100-110).

Cost, Latency, and Continuity

Cost efficiency evaluates value-for-money by comparing estimated execution costs against remaining budget constraints. The _compute_cost_efficiency function applies heuristics that favor free tiers when appropriate while maintaining quality standards.

Latency predicts turnaround time based on the latency_p50_seconds metric. The scoring maps sub-second responses to 1.0, while requests exceeding 60 seconds receive 0.2, with linear interpolation between these bounds.

Continuity ensures stylistic consistency by preferring providers already locked into the current production path. The _compute_continuity function boosts scores for providers used in recent operations, preventing jarring style shifts across sequences.

How the Weighted Score Is Calculated

After computing individual dimension scores, the engine aggregates them using fixed weights defined in ProviderScore.weighted_score (lines 35-45). The weight distribution prioritizes task relevance and quality while balancing operational concerns:

Dimension	Weight
task_fit	0.30
output_quality	0.20
control	0.15
reliability	0.15
cost_efficiency	0.10
latency	0.05
continuity	0.05

The score_provider function (lines 73-131) builds a ProviderScore instance for each candidate by invoking the individual dimension calculators. Subsequently, rank_providers (lines 33-41) sorts these scores in descending order, producing a ranked list that downstream selectors consume.

How video_selector Uses the Scoring Engine

The video_selector tool in tools/video/video_selector.py implements a five-stage pipeline that leverages the scoring engine to route video generation requests intelligently.

Candidate Discovery and Context Preparation

First, _providers() gathers all registered tools with capability="video_generation". Then _prepare_task_context calls normalize_task_context (lines 42-50) to transform loose user payloads into structured dictionaries required by the scorer, standardizing prompts, operations, and style parameters.

Scoring and Provider Selection

Inside the execute method (lines 13-19), when the operation mode is not "rank", the selector invokes:

rankings = rank_providers(candidates, task_context)

If the user specifies a preferred_provider, the selector checks whether that provider’s score lies within preferred_provider_gap (default 0.15) of the top score (lines 28-30). When the preferred provider meets this threshold, it wins selection; otherwise, the highest-scored provider prevails (lines 21-33).

Result Augmentation and Transparency

The selected tool’s ProviderScore.explain() method attaches human-readable reasoning to the final ToolResult (lines 55-57). For debugging or explicit ranking requests, _serialize_rankings (lines 68-82) exports the full provider ranking, enabling audit trails and optimization analysis.

Practical Implementation Examples

The following example demonstrates manual ranking of video providers for a cinematic brief:

from lib.scoring import rank_providers, normalize_task_context
from tools.video.video_selector import VideoSelector

selector = VideoSelector()
candidates = selector._providers()                      # all video tools

task_context = normalize_task_context(
    {},
    prompt="Create a cinematic trailer for a futuristic city",
    capability="video_generation",
    operation="text_to_video",
)

rankings = rank_providers(candidates, task_context)
for r in rankings[:3]:
    print(r.explain())

To use the selector in diagnostic "rank" mode without generating assets:

result = selector.execute({
    "prompt": "A short explainer about AI safety",
    "operation": "rank",
    "target_operation": "text_to_video",
})
print(result.data["explanation"])   # human-readable reasons for top 5 providers

Summary

• The 7-dimension provider scoring engine in lib/scoring.py provides deterministic, explainable rankings across task fit, quality, control, reliability, cost, latency, and continuity dimensions.
• Fixed weights prioritize task fit (30%) and output quality (20%), ensuring relevance and fidelity remain primary selection criteria.
• video_selector automates candidate discovery, context normalization, and preference-aware selection through the rank_providers pipeline.
• Transparency features like ProviderScore.explain() and _serialize_rankings provide audit trails for every routing decision.
• The preferred provider gap (default 0.15) allows user preferences to override the top-ranked provider when quality degradation remains within acceptable bounds.

Frequently Asked Questions

What are the seven dimensions used in OpenMontage's provider scoring engine?

The seven dimensions are task fit (keyword relevance), output quality (historic fidelity or stability tier), control (creative parameter availability), reliability (success probability), cost efficiency (value-for-money), latency (expected turnaround time), and continuity (consistency with current production providers). Each returns a normalized 0-1 score, with weights aggregating them into a final ranking.

How does video_selector handle user-preferred providers?

When a user specifies preferred_provider in the execution payload, video_selector compares that provider’s score against the top-ranked provider. If the gap falls within preferred_provider_gap (default 0.15), the preferred provider is selected despite not having the absolute highest score. If the gap exceeds this threshold, the engine defaults to the highest-scored provider to prevent significant quality degradation.

Can the dimension weights be customized per project?

According to the source code in lib/scoring.py, the weights are hardcoded in ProviderScore.weighted_score (lines 35-45) with fixed values for task fit (0.30), output quality (0.20), and the remaining five dimensions. The current implementation does not expose runtime weight customization, though individual dimension scores respond dynamically to task context and provider capabilities.

How does the scoring engine ensure transparency in its decisions?

Every ProviderScore instance includes an explain() method that returns human-readable justifications for the computed scores. The video_selector attaches these explanations to ToolResult objects and can serialize full rankings via _serialize_rankings, allowing developers to inspect exactly why specific providers were chosen or rejected for any given task.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}