# How the Storyboard Artist in ViMax Breaks Down Scripts into Shot-Level Descriptions

> Discover how ViMax's StoryboardArtist agent transforms scripts into detailed shot descriptions using a two-phase LLM pipeline for efficient visual breakdown.

- Repository: [✨Data Intelligence Lab@HKU✨/ViMax](https://github.com/HKUDS/ViMax)
- Tags: how-to-guide
- Published: 2026-05-20

---

**The StoryboardArtist agent converts narrative scripts into granular shot descriptions through a two-phase LLM pipeline: first generating high-level shot summaries via `design_storyboard`, then decomposing each into frame-by-frame specifications using `decompose_visual_description`.**

The ViMax video generation framework employs a dedicated StoryboardArtist agent to transform raw screenplays into structured, production-ready shot lists. Located in [`agents/storyboard_artist.py`](https://github.com/HKUDS/ViMax/blob/main/agents/storyboard_artist.py), this component bridges creative writing and technical rendering by implementing a sophisticated decomposition strategy that leverages LangChain chat models and strict Pydantic validation.

## The Two-Phase Script Breakdown Process

The storyboard artist operates through distinct macro and micro processing stages, ensuring both narrative coherence and technical precision.

### Phase 1: Designing the Brief Storyboard

The initial decomposition happens in `StoryboardArtist.design_storyboard` (lines 70‑86 of [`agents/storyboard_artist.py`](https://github.com/HKUDS/ViMax/blob/main/agents/storyboard_artist.py)). This method accepts a scene script, a list of `CharacterInScene` objects, and optional user requirements, then orchestrates the following workflow:

- **System Prompting**: The agent loads `system_prompt_template_design_storyboard`, instructing the LLM to act as a professional storyboard artist and output structured data.
- **Structured Output**: A `PydanticOutputParser` validates the response against the `StoryboardResponse` model, which contains a `storyboard` field holding `List[ShotBriefDescription]`.
- **Brief Content**: Each `ShotBriefDescription` captures visual and audio descriptions alongside camera index information, creating a high-level roadmap of the scene.

### Phase 2: Decomposing Visual Descriptions

Once the brief storyboard exists, `StoryboardArtist.decompose_visual_description` (lines 124‑158) processes each `ShotBriefDescription` individually to extract frame-level detail:

The method prompts the LLM to split the `visual_desc` into three concrete components stored in a `ShotDescription` object:

- **First-frame description** (`ff_desc`): A static snapshot of the shot's opening composition.
- **Last-frame description** (`lf_desc`): A static snapshot of the shot's ending composition.
- **Motion description** (`motion_desc`): Camera movements and on-screen element motions bridging the two frames.

Additionally, the model returns a `variation_type` classification (`large`, `medium`, or `small`) and a human-readable `variation_reason` explaining the shot's dynamism relative to preceding shots.

## Key Implementation Details

### Data Models and Parsing

Both phases rely on strict schema validation defined in [`interfaces/shot_description.py`](https://github.com/HKUDS/ViMax/blob/main/interfaces/shot_description.py). The `ShotBriefDescription` model handles initial LLM output with fields for camera positioning and high-level descriptions, while `ShotDescription` extends this with explicit frame boundaries and motion vectors required by downstream rendering pipelines.

### Error Handling and Async Execution

The agent wraps both decomposition methods with Tenacity's `@retry` decorator (utilizing `after_func` from [`utils/retry.py`](https://github.com/HKUDS/ViMax/blob/main/utils/retry.py) for back-off logic), automatically retrying transient API failures up to three times. The async design (`await asyncio.wait_for`) enables parallel processing of multiple shots when integrated into the larger video-generation workflow orchestrated by [`pipelines/script2video_pipeline.py`](https://github.com/HKUDS/ViMax/blob/main/pipelines/script2video_pipeline.py).

## Practical Code Examples

### Generating a Brief Storyboard

```python
from agents.storyboard_artist import StoryboardArtist
from langchain.chat_models import ChatOpenAI
from interfaces import CharacterInScene

# Initialise the LLM (any LangChain chat model works)

chat = ChatOpenAI(model_name="gpt-4o", temperature=0.2)

artist = StoryboardArtist(chat_model=chat)

script = """
INT. COFFEE SHOP – DAY
Alice sits at a corner table, scrolling on her phone. Bob enters, spots Alice, and waves.
ALICE
Hey, Bob! Over here.
BOB
(cheerful)
Hey, Alice! Got a minute?
"""

characters = [
    CharacterInScene(
        identifier_in_scene="Alice",
        static_features="short hair, green sweater",
        dynamic_features="often looks at phone",
    ),
    CharacterInScene(
        identifier_in_scene="Bob",
        static_features="tall, wearing a blue jacket",
        dynamic_features="energetic entrance",
    ),
]

brief_storyboard = await artist.design_storyboard(
    script=script,
    characters=characters,
    user_requirement="no more than 6 shots",
)
print(brief_storyboard)  # → List[ShotBriefDescription]

```

### Expanding a Brief Shot into a Full Description

```python

# Assume we already have a ShotBriefDescription from the previous step

shot_brief = brief_storyboard[0]

full_shot = await artist.decompose_visual_description(
    shot_brief_desc=shot_brief,
    characters=characters,
)

print(full_shot.ff_desc)   # First‑frame text

print(full_shot.lf_desc)   # Last‑frame text

print(full_shot.motion_desc)  # Motion description

print(full_shot.variation_type)  # large | medium | small

```

These snippets illustrate the complete data flow: high-level storyboard generation followed by granular frame extraction that downstream tools consume.

## Integration in the Video Pipeline

As implemented in [`pipelines/script2video_pipeline.py`](https://github.com/HKUDS/ViMax/blob/main/pipelines/script2video_pipeline.py), the storyboard artist sits between script ingestion and visual rendering. It transforms textual narratives into the structured `ShotDescription` objects that [`render_backend.py`](https://github.com/HKUDS/ViMax/blob/main/render_backend.py) requires for actual video synthesis, effectively translating creative intent into executable generation commands.

## Summary

- **Two-phase architecture**: `design_storyboard` creates high-level summaries, while `decompose_visual_description` extracts frame-level detail.
- **Structured output**: Pydantic models (`ShotBriefDescription`, `ShotDescription`) enforce consistent LLM responses in [`agents/storyboard_artist.py`](https://github.com/HKUDS/ViMax/blob/main/agents/storyboard_artist.py).
- **Granular decomposition**: Each shot splits into first-frame, last-frame, and motion descriptions with variation classification.
- **Production robustness**: Tenacity retry logic and async processing support reliable pipeline execution.

## Frequently Asked Questions

### What is the difference between ShotBriefDescription and ShotDescription?

`ShotBriefDescription` (defined in [`interfaces/shot_description.py`](https://github.com/HKUDS/ViMax/blob/main/interfaces/shot_description.py)) contains high-level visual and audio descriptions plus camera indices generated during the initial storyboard design phase. `ShotDescription` extends this with granular frame data—specifically `ff_desc`, `lf_desc`, and `motion_desc`—created during the decomposition phase to guide final video rendering.

### How does the StoryboardArtist handle API failures during script processing?

Both `design_storyboard` and `decompose_visual_description` methods use Tenacity's `@retry` decorator configured with `after_func` from [`utils/retry.py`](https://github.com/HKUDS/ViMax/blob/main/utils/retry.py). This automatically retries failed LLM calls up to three times with back-off, ensuring transient network errors don't break the video generation pipeline.

### What are the three components of a decomposed visual description?

According to the `decompose_visual_description` implementation in lines 124‑158, the system extracts: **first-frame description** (`ff_desc`) representing the opening static composition, **last-frame description** (`lf_desc`) capturing the closing composition, and **motion description** (`motion_desc`) detailing camera movements and object animations between the two frames.

### How does the storyboard artist integrate with the broader ViMax pipeline?

The `StoryboardArtist` agent functions as a preprocessing stage within [`pipelines/script2video_pipeline.py`](https://github.com/HKUDS/ViMax/blob/main/pipelines/script2video_pipeline.py). It accepts raw scripts and character definitions from upstream components, produces validated `ShotDescription` objects, and passes these structured data to downstream rendering modules that generate actual video frames based on the frame-by-frame specifications.