ViMax Variation Types Explained: Small, Medium, and Large Shot Descriptions

ViMax uses three variation types—small, medium, and large—to control frame generation complexity, determining whether a shot requires one frame, two frames, or a full transition sequence.

The ViMax repository implements an intelligent shot-level variation system that optimizes video generation pipelines by scaling computational effort according to visual change magnitude. This mechanism, defined in the ShotDescription model, directly impacts how the Script2VideoPipeline allocates resources when constructing cinematic sequences. Understanding these variation types is essential for optimizing generation costs and ensuring smooth narrative transitions in AI-generated video content.

Understanding ViMax Variation Types

ViMax classifies shot-level variation through the ShotDescription dataclass located in interfaces/shot_description.py (lines 95-124). The critical field variation_type accepts a Literal of three string values that gate the entire generation workflow.

The ShotDescription Model Structure

In interfaces/shot_description.py, the variation type is defined as:

variation_type: Literal["large", "medium", "small"] = Field(
    description="Indicates the degree of change in the shot's content."
)

This field works alongside ff_desc (first frame description) and lf_desc (last frame description) to determine pipeline execution branches.

Small Variation: Single-Frame Generation

Small variation indicates minimal visual change between the beginning and end of a shot—such as subtle facial expressions, slight pose shifts, or modest camera pans.

According to the source code in pipelines/script2video_pipeline.py, small variation shots trigger the most efficient generation path:

• First frame only: The pipeline generates exclusively the initial frame defined in ff_desc
• No last-frame generation: The conditional check if shot_description.variation_type in ["medium","large"] evaluates to false, skipping the generate_frame_for_single_shot call for the last frame
• Direct video assembly: generate_video_for_single_shot receives only the first-frame path, creating video from a single image without transition overhead

Medium Variation: Dual-Frame with Transition

Medium variation signifies noticeable but non-dramatic changes—such as new characters entering the frame, a subject turning toward the camera, or clear action shifts.

The pipeline responds to medium variation by:

1. Generating both frames: Invoking generate_frame_for_single_shot for the last frame (see lines 67-74 and 93-102 in script2video_pipeline.py)
2. Creating transition videos: If the shot is the first of a camera sequence, the pipeline generates a transition video from the parent shot
3. Stitching dual frames: The frame_paths list receives both first and last frame paths (lines 19-26), enabling the video generator to interpolate between two distinct images

Large Variation: Complex Transitions

Large variation represents dramatic compositional changes—such as wide-to-close shots, entirely new environments, or major camera movements like drone pullbacks.

While frame generation mirrors the medium variation path (generating both first and last frames), large variations receive special treatment in the storyboarding phase. As implemented in agents/storyboard_artist.py (lines 104-106), large variations often trigger "exaggerated transitions" that require more sophisticated blending between parent and child shots. The pipeline still executes the dual-frame branch in script2video_pipeline.py, but the transition video complexity increases significantly.

Technical Implementation in the Generation Pipeline

The variation type gates critical logic branches in Script2VideoPipeline. The conditional structure determines computational expenditure:

from vi_max.interfaces.shot_description import ShotDescription
from vi_max.pipelines.script2video_pipeline import Script2VideoPipeline

# Small variation: Only first frame generated

small_shot = ShotDescription(
    idx=0,
    is_last=False,
    cam_idx=0,
    visual_desc="Alice smiles at the camera.",
    variation_type="small",
    variation_reason="Only facial expression changes.",
    ff_desc="Close-up of Alice, smiling.",
    ff_vis_char_idxs=[0],
    lf_desc="",  # Ignored for small variation

    lf_vis_char_idxs=[],
    motion_desc="A gentle zoom in.",
    audio_desc="[Speaker] Alice (Happy): Hello!"
)

# Medium variation: Both frames required

medium_shot = ShotDescription(
    idx=1,
    is_last=False,
    cam_idx=0,
    visual_desc="Bob walks into the room.",
    variation_type="medium",
    variation_reason="New character appears.",
    ff_desc="Wide shot of an empty room.",
    ff_vis_char_idxs=[],
    lf_desc="Medium shot of Bob entering.",
    lf_vis_char_idxs=[1],
    motion_desc="Bob walks from left to right.",
    audio_desc="[Sound Effect] Door creak."
)

# Execute generation

pipeline = Script2VideoPipeline(...)
await pipeline.generate_frames_for_single_camera(
    camera=camera_tree_entry,
    shot_descriptions=[small_shot, medium_shot],
    characters=character_list,
    character_portraits_registry=registry,
    priority_shot_idxs=[]
)

In script2video_pipeline.py, the variation check appears as:

# Lines 67-74 and 93-102

if shot_description.variation_type in ["medium", "large"]:
    # Generate last frame

    last_frame_path = await self.generate_frame_for_single_shot(
        shot_description=shot_description,
        frame_type="last",
        # ... other params

    )

The video assembly logic (lines 19-26) then conditionally includes the last frame:

frame_paths = [first_frame_path]
if shot_description.variation_type in ["medium", "large"]:
    frame_paths.append(last_frame_path)

Impact on Video Generation Performance

The variation type directly scales computational effort across three dimensions:

• Memory usage: Small variations require single-image memory allocation, while medium and large variations double the frame buffer requirements
• API calls: Each frame generation invokes the underlying image generation model; small variations halve API consumption compared to medium/large shots
• Processing time: Transition video generation for medium and large variations adds significant overhead, with large variations potentially requiring additional processing for complex transitions as determined by the storyboard artist agent

Summary

• Small variation generates only the first frame, optimizing for shots with minimal visual change like facial expressions or slight camera movements
• Medium variation triggers dual-frame generation and optional transition videos for noticeable changes such as new character appearances
• Large variation produces the same dual-frame structure as medium but enables complex, exaggerated transitions for dramatic compositional shifts
• Source locations: The ShotDescription model resides in interfaces/shot_description.py, while gating logic appears in pipelines/script2video_pipeline.py (lines 67-74, 93-102) and transition logic appears in agents/storyboard_artist.py (lines 104-106)
• Performance scaling: Computational cost scales linearly from small (1x) to medium/large (2x+ transition overhead)

Frequently Asked Questions

What defines a "small" variation shot in ViMax?

A small variation shot contains only minor changes between the first and last frames, such as subtle facial expressions, slight pose adjustments, or modest camera pans. According to the ViMax source code in interfaces/shot_description.py, these shots require only single-frame generation because the visual content remains essentially consistent throughout the shot duration.

How does the ViMax pipeline handle medium versus large variation types?

Both medium and large variation types trigger dual-frame generation (first and last frames) and transition video creation in script2video_pipeline.py. However, large variations typically involve more complex transition requirements as described in agents/storyboard_artist.py, often resulting in exaggerated transitions that require additional computational resources compared to the standard transitions used for medium variations.

Where in the codebase does ViMax check the variation type?

The primary variation type checks occur in pipelines/script2video_pipeline.py at lines 67-74 and 93-102, where the pipeline conditionally calls generate_frame_for_single_shot for last-frame generation. Additional logic appears at lines 19-26 for video assembly, and semantic definitions reside in agents/storyboard_artist.py (lines 104-106).

Can I force a specific variation type for optimization purposes?

While the variation type is typically determined by the storyboard artist agent based on scene descriptions, you can manually set the variation_type field in the ShotDescription model when constructing shots programmatically. Setting low-change scenes to "small" reduces API calls and generation time by half compared to medium or large variations, though this may sacrifice visual quality for shots that actually require frame interpolation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}