# ViMax Variation Types Explained: Small, Medium, and Large Shot Descriptions

> Discover ViMax variation types small, medium, and large for shot descriptions. Control frame generation complexity and optimize transitions for HKUDS ViMax.

- Repository: [✨Data Intelligence Lab@HKU✨/ViMax](https://github.com/HKUDS/ViMax)
- Tags: deep-dive
- Published: 2026-05-20

---

**ViMax uses three variation types—small, medium, and large—to control frame generation complexity, determining whether a shot requires one frame, two frames, or a full transition sequence.**

The **ViMax** repository implements an intelligent shot-level variation system that optimizes video generation pipelines by scaling computational effort according to visual change magnitude. This mechanism, defined in the `ShotDescription` model, directly impacts how the `Script2VideoPipeline` allocates resources when constructing cinematic sequences. Understanding these variation types is essential for optimizing generation costs and ensuring smooth narrative transitions in AI-generated video content.

## Understanding ViMax Variation Types

ViMax classifies shot-level variation through the `ShotDescription` dataclass located in [`interfaces/shot_description.py`](https://github.com/HKUDS/ViMax/blob/main/interfaces/shot_description.py) (lines 95-124). The critical field `variation_type` accepts a **Literal** of three string values that gate the entire generation workflow.

### The ShotDescription Model Structure

In [`interfaces/shot_description.py`](https://github.com/HKUDS/ViMax/blob/main/interfaces/shot_description.py), the variation type is defined as:

```python
variation_type: Literal["large", "medium", "small"] = Field(
    description="Indicates the degree of change in the shot's content."
)

```

This field works alongside `ff_desc` (first frame description) and `lf_desc` (last frame description) to determine pipeline execution branches.

### Small Variation: Single-Frame Generation

**Small** variation indicates minimal visual change between the beginning and end of a shot—such as subtle facial expressions, slight pose shifts, or modest camera pans.

According to the source code in [`pipelines/script2video_pipeline.py`](https://github.com/HKUDS/ViMax/blob/main/pipelines/script2video_pipeline.py), small variation shots trigger the most efficient generation path:

- **First frame only**: The pipeline generates exclusively the initial frame defined in `ff_desc`
- **No last-frame generation**: The conditional check `if shot_description.variation_type in ["medium","large"]` evaluates to false, skipping the `generate_frame_for_single_shot` call for the last frame
- **Direct video assembly**: `generate_video_for_single_shot` receives only the first-frame path, creating video from a single image without transition overhead

### Medium Variation: Dual-Frame with Transition

**Medium** variation signifies noticeable but non-dramatic changes—such as new characters entering the frame, a subject turning toward the camera, or clear action shifts.

The pipeline responds to medium variation by:

1. **Generating both frames**: Invoking `generate_frame_for_single_shot` for the last frame (see lines 67-74 and 93-102 in [`script2video_pipeline.py`](https://github.com/HKUDS/ViMax/blob/main/script2video_pipeline.py))
2. **Creating transition videos**: If the shot is the first of a camera sequence, the pipeline generates a transition video from the parent shot
3. **Stitching dual frames**: The `frame_paths` list receives both first and last frame paths (lines 19-26), enabling the video generator to interpolate between two distinct images

### Large Variation: Complex Transitions

**Large** variation represents dramatic compositional changes—such as wide-to-close shots, entirely new environments, or major camera movements like drone pullbacks.

While frame generation mirrors the medium variation path (generating both first and last frames), large variations receive special treatment in the storyboarding phase. As implemented in [`agents/storyboard_artist.py`](https://github.com/HKUDS/ViMax/blob/main/agents/storyboard_artist.py) (lines 104-106), large variations often trigger **"exaggerated transitions"** that require more sophisticated blending between parent and child shots. The pipeline still executes the dual-frame branch in [`script2video_pipeline.py`](https://github.com/HKUDS/ViMax/blob/main/script2video_pipeline.py), but the transition video complexity increases significantly.

## Technical Implementation in the Generation Pipeline

The variation type gates critical logic branches in `Script2VideoPipeline`. The conditional structure determines computational expenditure:

```python
from vi_max.interfaces.shot_description import ShotDescription
from vi_max.pipelines.script2video_pipeline import Script2VideoPipeline

# Small variation: Only first frame generated

small_shot = ShotDescription(
    idx=0,
    is_last=False,
    cam_idx=0,
    visual_desc="Alice smiles at the camera.",
    variation_type="small",
    variation_reason="Only facial expression changes.",
    ff_desc="Close-up of Alice, smiling.",
    ff_vis_char_idxs=[0],
    lf_desc="",  # Ignored for small variation

    lf_vis_char_idxs=[],
    motion_desc="A gentle zoom in.",
    audio_desc="[Speaker] Alice (Happy): Hello!"
)

# Medium variation: Both frames required

medium_shot = ShotDescription(
    idx=1,
    is_last=False,
    cam_idx=0,
    visual_desc="Bob walks into the room.",
    variation_type="medium",
    variation_reason="New character appears.",
    ff_desc="Wide shot of an empty room.",
    ff_vis_char_idxs=[],
    lf_desc="Medium shot of Bob entering.",
    lf_vis_char_idxs=[1],
    motion_desc="Bob walks from left to right.",
    audio_desc="[Sound Effect] Door creak."
)

# Execute generation

pipeline = Script2VideoPipeline(...)
await pipeline.generate_frames_for_single_camera(
    camera=camera_tree_entry,
    shot_descriptions=[small_shot, medium_shot],
    characters=character_list,
    character_portraits_registry=registry,
    priority_shot_idxs=[]
)

```

In [`script2video_pipeline.py`](https://github.com/HKUDS/ViMax/blob/main/script2video_pipeline.py), the variation check appears as:

```python

# Lines 67-74 and 93-102

if shot_description.variation_type in ["medium", "large"]:
    # Generate last frame

    last_frame_path = await self.generate_frame_for_single_shot(
        shot_description=shot_description,
        frame_type="last",
        # ... other params

    )

```

The video assembly logic (lines 19-26) then conditionally includes the last frame:

```python
frame_paths = [first_frame_path]
if shot_description.variation_type in ["medium", "large"]:
    frame_paths.append(last_frame_path)

```

## Impact on Video Generation Performance

The variation type directly scales computational effort across three dimensions:

- **Memory usage**: Small variations require single-image memory allocation, while medium and large variations double the frame buffer requirements
- **API calls**: Each frame generation invokes the underlying image generation model; small variations halve API consumption compared to medium/large shots
- **Processing time**: Transition video generation for medium and large variations adds significant overhead, with large variations potentially requiring additional processing for complex transitions as determined by the storyboard artist agent

## Summary

- **Small variation** generates only the first frame, optimizing for shots with minimal visual change like facial expressions or slight camera movements
- **Medium variation** triggers dual-frame generation and optional transition videos for noticeable changes such as new character appearances
- **Large variation** produces the same dual-frame structure as medium but enables complex, exaggerated transitions for dramatic compositional shifts
- **Source locations**: The `ShotDescription` model resides in [`interfaces/shot_description.py`](https://github.com/HKUDS/ViMax/blob/main/interfaces/shot_description.py), while gating logic appears in [`pipelines/script2video_pipeline.py`](https://github.com/HKUDS/ViMax/blob/main/pipelines/script2video_pipeline.py) (lines 67-74, 93-102) and transition logic appears in [`agents/storyboard_artist.py`](https://github.com/HKUDS/ViMax/blob/main/agents/storyboard_artist.py) (lines 104-106)
- **Performance scaling**: Computational cost scales linearly from small (1x) to medium/large (2x+ transition overhead)

## Frequently Asked Questions

### What defines a "small" variation shot in ViMax?

A **small** variation shot contains only minor changes between the first and last frames, such as subtle facial expressions, slight pose adjustments, or modest camera pans. According to the ViMax source code in [`interfaces/shot_description.py`](https://github.com/HKUDS/ViMax/blob/main/interfaces/shot_description.py), these shots require only single-frame generation because the visual content remains essentially consistent throughout the shot duration.

### How does the ViMax pipeline handle medium versus large variation types?

Both **medium** and **large** variation types trigger dual-frame generation (first and last frames) and transition video creation in [`script2video_pipeline.py`](https://github.com/HKUDS/ViMax/blob/main/script2video_pipeline.py). However, **large** variations typically involve more complex transition requirements as described in [`agents/storyboard_artist.py`](https://github.com/HKUDS/ViMax/blob/main/agents/storyboard_artist.py), often resulting in exaggerated transitions that require additional computational resources compared to the standard transitions used for medium variations.

### Where in the codebase does ViMax check the variation type?

The primary variation type checks occur in [`pipelines/script2video_pipeline.py`](https://github.com/HKUDS/ViMax/blob/main/pipelines/script2video_pipeline.py) at lines 67-74 and 93-102, where the pipeline conditionally calls `generate_frame_for_single_shot` for last-frame generation. Additional logic appears at lines 19-26 for video assembly, and semantic definitions reside in [`agents/storyboard_artist.py`](https://github.com/HKUDS/ViMax/blob/main/agents/storyboard_artist.py) (lines 104-106).

### Can I force a specific variation type for optimization purposes?

While the variation type is typically determined by the storyboard artist agent based on scene descriptions, you can manually set the `variation_type` field in the `ShotDescription` model when constructing shots programmatically. Setting low-change scenes to `"small"` reduces API calls and generation time by half compared to medium or large variations, though this may sacrifice visual quality for shots that actually require frame interpolation.