How the Storyboard Artist in ViMax Breaks Down Scripts into Shot-Level Descriptions

The StoryboardArtist agent converts narrative scripts into granular shot descriptions through a two-phase LLM pipeline: first generating high-level shot summaries via design_storyboard, then decomposing each into frame-by-frame specifications using decompose_visual_description.

The ViMax video generation framework employs a dedicated StoryboardArtist agent to transform raw screenplays into structured, production-ready shot lists. Located in agents/storyboard_artist.py, this component bridges creative writing and technical rendering by implementing a sophisticated decomposition strategy that leverages LangChain chat models and strict Pydantic validation.

The Two-Phase Script Breakdown Process

The storyboard artist operates through distinct macro and micro processing stages, ensuring both narrative coherence and technical precision.

Phase 1: Designing the Brief Storyboard

The initial decomposition happens in StoryboardArtist.design_storyboard (lines 70‑86 of agents/storyboard_artist.py). This method accepts a scene script, a list of CharacterInScene objects, and optional user requirements, then orchestrates the following workflow:

• System Prompting: The agent loads system_prompt_template_design_storyboard, instructing the LLM to act as a professional storyboard artist and output structured data.
• Structured Output: A PydanticOutputParser validates the response against the StoryboardResponse model, which contains a storyboard field holding List[ShotBriefDescription].
• Brief Content: Each ShotBriefDescription captures visual and audio descriptions alongside camera index information, creating a high-level roadmap of the scene.

Phase 2: Decomposing Visual Descriptions

Once the brief storyboard exists, StoryboardArtist.decompose_visual_description (lines 124‑158) processes each ShotBriefDescription individually to extract frame-level detail:

The method prompts the LLM to split the visual_desc into three concrete components stored in a ShotDescription object:

• First-frame description (ff_desc): A static snapshot of the shot's opening composition.
• Last-frame description (lf_desc): A static snapshot of the shot's ending composition.
• Motion description (motion_desc): Camera movements and on-screen element motions bridging the two frames.

Additionally, the model returns a variation_type classification (large, medium, or small) and a human-readable variation_reason explaining the shot's dynamism relative to preceding shots.

Key Implementation Details

Data Models and Parsing

Both phases rely on strict schema validation defined in interfaces/shot_description.py. The ShotBriefDescription model handles initial LLM output with fields for camera positioning and high-level descriptions, while ShotDescription extends this with explicit frame boundaries and motion vectors required by downstream rendering pipelines.

Error Handling and Async Execution

The agent wraps both decomposition methods with Tenacity's @retry decorator (utilizing after_func from utils/retry.py for back-off logic), automatically retrying transient API failures up to three times. The async design (await asyncio.wait_for) enables parallel processing of multiple shots when integrated into the larger video-generation workflow orchestrated by pipelines/script2video_pipeline.py.

Practical Code Examples

Generating a Brief Storyboard

from agents.storyboard_artist import StoryboardArtist
from langchain.chat_models import ChatOpenAI
from interfaces import CharacterInScene

# Initialise the LLM (any LangChain chat model works)

chat = ChatOpenAI(model_name="gpt-4o", temperature=0.2)

artist = StoryboardArtist(chat_model=chat)

script = """
INT. COFFEE SHOP – DAY
Alice sits at a corner table, scrolling on her phone. Bob enters, spots Alice, and waves.
ALICE
Hey, Bob! Over here.
BOB
(cheerful)
Hey, Alice! Got a minute?
"""

characters = [
    CharacterInScene(
        identifier_in_scene="Alice",
        static_features="short hair, green sweater",
        dynamic_features="often looks at phone",
    ),
    CharacterInScene(
        identifier_in_scene="Bob",
        static_features="tall, wearing a blue jacket",
        dynamic_features="energetic entrance",
    ),
]

brief_storyboard = await artist.design_storyboard(
    script=script,
    characters=characters,
    user_requirement="no more than 6 shots",
)
print(brief_storyboard)  # → List[ShotBriefDescription]

Expanding a Brief Shot into a Full Description

# Assume we already have a ShotBriefDescription from the previous step

shot_brief = brief_storyboard[0]

full_shot = await artist.decompose_visual_description(
    shot_brief_desc=shot_brief,
    characters=characters,
)

print(full_shot.ff_desc)   # First‑frame text

print(full_shot.lf_desc)   # Last‑frame text

print(full_shot.motion_desc)  # Motion description

print(full_shot.variation_type)  # large | medium | small

These snippets illustrate the complete data flow: high-level storyboard generation followed by granular frame extraction that downstream tools consume.

Integration in the Video Pipeline

As implemented in pipelines/script2video_pipeline.py, the storyboard artist sits between script ingestion and visual rendering. It transforms textual narratives into the structured ShotDescription objects that render_backend.py requires for actual video synthesis, effectively translating creative intent into executable generation commands.

Summary

• Two-phase architecture: design_storyboard creates high-level summaries, while decompose_visual_description extracts frame-level detail.
• Structured output: Pydantic models (ShotBriefDescription, ShotDescription) enforce consistent LLM responses in agents/storyboard_artist.py.
• Granular decomposition: Each shot splits into first-frame, last-frame, and motion descriptions with variation classification.
• Production robustness: Tenacity retry logic and async processing support reliable pipeline execution.

Frequently Asked Questions

What is the difference between ShotBriefDescription and ShotDescription?

ShotBriefDescription (defined in interfaces/shot_description.py) contains high-level visual and audio descriptions plus camera indices generated during the initial storyboard design phase. ShotDescription extends this with granular frame data—specifically ff_desc, lf_desc, and motion_desc—created during the decomposition phase to guide final video rendering.

How does the StoryboardArtist handle API failures during script processing?

Both design_storyboard and decompose_visual_description methods use Tenacity's @retry decorator configured with after_func from utils/retry.py. This automatically retries failed LLM calls up to three times with back-off, ensuring transient network errors don't break the video generation pipeline.

What are the three components of a decomposed visual description?

According to the decompose_visual_description implementation in lines 124‑158, the system extracts: first-frame description (ff_desc) representing the opening static composition, last-frame description (lf_desc) capturing the closing composition, and motion description (motion_desc) detailing camera movements and object animations between the two frames.

How does the storyboard artist integrate with the broader ViMax pipeline?

The StoryboardArtist agent functions as a preprocessing stage within pipelines/script2video_pipeline.py. It accepts raw scripts and character definitions from upstream components, produces validated ShotDescription objects, and passes these structured data to downstream rendering modules that generate actual video frames based on the frame-by-frame specifications.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}