ViMax Pydantic Data Interfaces for Character, Scene, and Shot Descriptions

ViMax utilizes strict Pydantic BaseModel classes located in the main/interfaces package to define strongly-typed schemas for narrative elements, ensuring validated data flows between AI agents without fragile dictionary-based contracts.

The HKUDS/ViMax repository implements a comprehensive type system for video generation pipelines using Pydantic data interfaces. These interfaces model characters, scenes, and shots as validated Python objects, enabling reliable communication between specialized agents like scene extractors, storyboard artists, and camera image generators.

Character Interfaces in main/interfaces/character.py

The character module defines three hierarchical Pydantic models that represent characters at different granularities of the narrative structure.

CharacterInScene

The CharacterInScene class captures a single character's appearance within a specific scene. According to the source code in main/interfaces/character.py, this model enforces strict typing on five primary fields:

• idx: Integer index for positional reference
• identifier_in_scene: String name or tag used in scripts (e.g., "Alice" or "Bob the Builder")
• is_visible: Boolean flag indicating whether the character appears on screen
• static_features: String describing immutable characteristics (physical appearance, clothing baseline)
• dynamic_features: String describing scene-specific variations (temporary accessories, posture changes)

CharacterInEvent and CharacterInNovel

For higher-level narrative aggregation, ViMax provides CharacterInEvent and CharacterInNovel models. These track characters across multiple scenes or the entire novel:

• CharacterInEvent: Links characters to events via index, identifier_in_event, active_scenes (List of scene indices), and static_features
• CharacterInNovel: Provides the broadest scope with index, identifier_in_novel, active_events (List of event indices), and static_features

Scene Interface in main/interfaces/scene.py

The Scene model acts as a container aggregating environment data, character lists, and script text. Defined in main/interfaces/scene.py, it consists of:

• idx: Scene identifier (int)
• is_last: Boolean indicating if this is the final scene
• environment: Instance of EnvironmentInScene (location, time_of_day, weather)
• characters: List of CharacterInScene objects
• script: Raw text string containing dialogue and action descriptions

This design allows the scene_extractor agent to output validated objects that downstream agents can consume without manual parsing.

Shot Description Interfaces in main/interfaces/shot_description.py

Shot-level data moves through two distinct Pydantic interfaces representing different pipeline stages.

ShotBriefDescription

Used by planners during initial storyboarding, ShotBriefDescription provides a lightweight placeholder with:

• idx: Shot index (int)
• is_last: Boolean flag for sequence termination
• cam_idx: Camera identifier (int)
• visual_desc: String describing the visual composition
• audio_desc: String describing ambient sound or music

ShotDescription

The full ShotDescription model expands the brief into production-ready specifications with thirteen strictly-typed fields:

• Core identifiers: idx, is_last, cam_idx
• Variation metadata: variation_type (Literal values like "large", "medium", "small"), variation_reason (explanatory string)
• Frame descriptions: ff_desc (first frame), lf_desc (last frame)
• Character indexing: ff_vis_char_idxs and lf_vis_char_idxs (List[int] referencing CharacterInScene.idx values)
• Motion and audio: motion_desc (action details), audio_desc (sound design)

The character index lists (ff_vis_char_idxs, lf_vis_char_idxs) maintain referential integrity to the parent scene's CharacterInScene objects via their idx fields.

Working with ViMax Pydantic Interfaces

Instantiating these models provides runtime validation and IDE autocomplete support. Below are practical implementations demonstrating the hierarchical relationships between characters, scenes, and shots.

Creating Characters for a Scene

from ViMax.main.interfaces.character import CharacterInScene
from ViMax.main.interfaces.scene import Scene
from ViMax.main.interfaces.environment import EnvironmentInScene

alice = CharacterInScene(
    idx=0,
    identifier_in_scene="Alice",
    is_visible=True,
    static_features="Alice has long blonde hair and blue eyes, and is of slender build.",
    dynamic_features="Wearing a red scarf and a black leather jacket",
)

bob = CharacterInScene(
    idx=1,
    identifier_in_scene="Bob the Builder",
    is_visible=False,
    static_features="Bob the Builder is a middle‑aged man with a sturdy build.",
    dynamic_features="",
)

scene = Scene(
    idx=0,
    is_last=False,
    environment=EnvironmentInScene(
        location="Supermarket",
        time_of_day="Afternoon",
        weather="Indoor",
    ),
    characters=[alice, bob],
    script="""
<Alice> looks around the aisle, searching for the missing ingredient.
<Bob the Builder> (off‑screen) mutters: "I think I saw it near the dairy section."
""",
)

Generating Shot Descriptions

Planners first create brief descriptions before expanding to full specifications:

from ViMax.main.interfaces.shot_description import ShotBriefDescription, ShotDescription

# Initial brief from planner

brief = ShotBriefDescription(
    idx=0,
    is_last=False,
    cam_idx=0,
    visual_desc="An over‑the‑shoulder shot behind <Alice>, showing her hand reaching for a can.",
    audio_desc="[Sound Effect] Soft hum of the refrigerator.",
)

# Expanded production description

full = ShotDescription(
    idx=0,
    is_last=False,
    cam_idx=0,
    visual_desc="A close‑up of <Alice>'s hand as she grabs a can of tomatoes.",
    variation_type="small",
    variation_reason="Only the hand changes between frames.",
    ff_desc="Close‑up of an empty shelf.",
    ff_vis_char_idxs=[],
    lf_desc="Close‑up of the can in Alice's hand.",
    lf_vis_char_idxs=[0],  # References alice.idx

    motion_desc="Alice's fingers close around the can.",
    audio_desc="[Sound Effect] Shelf creak followed by a soft click.",
)

Summary

• ViMax Pydantic data interfaces reside in main/interfaces/character.py, main/interfaces/scene.py, and main/interfaces/shot_description.py
• Character models exist at three scopes: CharacterInScene (single scene), CharacterInEvent (event level), and CharacterInNovel (global)
• Scene objects aggregate environments, character lists, and scripts using strict List and BaseModel typing
• Shot descriptions progress from ShotBriefDescription (planning) to ShotDescription (production), with character visibility tracked via integer index references
• All interfaces use Pydantic validation to ensure type safety across agent boundaries

Frequently Asked Questions

What fields are required for CharacterInScene validation?

According to main/interfaces/character.py, CharacterInScene requires idx (int), identifier_in_scene (str), is_visible (bool), static_features (str), and dynamic_features (str). The model validates that these fields conform to their specified types at instantiation time.

How does ShotDescription reference characters from the parent scene?

ShotDescription uses integer lists ff_vis_char_idxs and lf_vis_char_idxs to store indices corresponding to the idx field of CharacterInScene objects in the parent scene. This maintains lightweight references without duplicating character data.

Why does ViMax use Pydantic instead of dictionaries for data interfaces?

The ViMax codebase uses Pydantic BaseModel classes to provide runtime type validation, IDE autocomplete support, and self-documenting schemas. This prevents runtime errors from missing keys or type mismatches when passing data between agents like scene_extractor and storyboard_artist.

Where are the environment models defined in ViMax?

The EnvironmentInScene model referenced by Scene.characters is defined in main/interfaces/environment.py (companion file to the character and scene modules), providing typed fields for location, time_of_day, and weather.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}